This is an archive of the discontinued Mercurial Phabricator instance.

Differential D7129

py3: don't index into bytes in phabricator's _tokenize()
ClosedPublic

Authored by Kwan on Oct 17 2019, 5:48 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG47946f08e463: py3: don't index into bytes in phabricator's _tokenize()
rHGb3a1ba3f46f1: py3: don't index into bytes in phabricator's _tokenize()
Summary

phabreading a stack using hg phabread :D1234 under py3 will currently die
with a KeyError because it will index into b':D1234' and return 58 instead
of b':' as a token.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Kwan created this revision.Oct 17 2019, 5:48 PM
Kwan added a commit: rHGb3a1ba3f46f1: py3: don't index into bytes in phabricator's _tokenize().Oct 18 2019, 7:08 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHGb3a1ba3f46f1: py3: don't index into bytes in phabricator's _tokenize() (authored by Kwan). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mhgext/phabricator.py (4 lines)

Diff 17339

hgext/phabricator.py

itertools.takewhile( itertools.takewhile(
lambda ch: ch not in special, pycompat.iterbytestr(view[pos:]) lambda ch: ch not in special, pycompat.iterbytestr(view[pos:])
) )
) )
if symbol: if symbol:
yield (b'symbol', symbol, pos) yield (b'symbol', symbol, pos)
pos += len(symbol) pos += len(symbol)
else: # special char, ignore space else: # special char, ignore space
if text[pos] != b' ': if text[pos : pos + 1] != b' ':
yield (text[pos], None, pos) yield (text[pos : pos + 1], None, pos)
pos += 1 pos += 1
yield (b'end', None, pos) yield (b'end', None, pos)
def _parse(text): def _parse(text):
tree, pos = parser.parser(_elements).parse(_tokenize(text)) tree, pos = parser.parser(_elements).parse(_tokenize(text))
if pos != len(text): if pos != len(text):
raise error.ParseError(b'invalid token', pos) raise error.ParseError(b'invalid token', pos)