perf: make perf.run-limits code work with Python 3

Authored by indygreg.

Description

perf: make perf.run-limits code work with Python 3

We need b'' because perf.py isn't run through the source
transformer.

We need to cast the exception to bytes using pycompat.bytestr()
because ValueError can't be %s formatted due to built-in exceptions
lacking bytes.

We need to pycompat.sysstr() before the float() and int() cast
so the ValueError message doesn't have b'' in it.

Even with that, it looks like the error message for the ValueError
for float casts added quotes, so we need to account for that in test
output.

Differential Revision: https://phab.mercurial-scm.org/D6200