This is an archive of the discontinued Mercurial Phabricator instance.

Differential D5049

hgdemandimport: don't use str.isidentifier()
AbandonedPublic

Authored by indygreg on Oct 13 2018, 3:28 AM.

Details

Reviewers
baymax
Group Reviewers
hg-reviewers
Summary

This method doesn't exist in Python 2.7.

The new code is equivalent to what Python was doing up until
isidentifier() was introduced to Lib/tokenize.py in
33856de84d1115a18b699e0ca93c3b921bc6a1af.

Strictly speaking, a simple ascii check is not sufficient, as a proper
identifier is defined by PEP 3131 and allows non-ascii characters. But
we don't plan to use this tokenizer on arbitrary source code, so I think
we can get away with not conforming to PEP 3131.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Oct 13 2018, 3:28 AM
baymax requested changes to this revision.Jan 24 2020, 12:32 PM

There seems to have been no activities on this Diff for the past 3 Months.

By policy, we are automatically moving it out of the need-review state.

Please, move it back to need-review without hesitation if this diff should still be discussed.

:baymax:need-review-idle:

This revision now requires changes to proceed.Jan 24 2020, 12:32 PM
indygreg abandoned this revision.Jan 24 2020, 11:51 PM

Revision Contents
Changeset List

PathPackages
Mhgdemandimport/py3tokenize.py (6 lines)

Diff 12005

hgdemandimport/py3tokenize.py

# * Removed main() and related functionality. # * Removed main() and related functionality.
# * Removed generate_tokens(). # * Removed generate_tokens().
# * Removed open(). # * Removed open().
# * Removed module docstring. # * Removed module docstring.
# * Adjusted for relative imports. # * Adjusted for relative imports.
# * absolute_import added. # * absolute_import added.
# * Removed re.ASCII. # * Removed re.ASCII.
# * Various backports to work on Python 2.7. # * Various backports to work on Python 2.7.
# * Dropped support for non-ASCII identifiers (PEP 3131 support).
from __future__ import absolute_import from __future__ import absolute_import
__author__ = 'Ka-Ping Yee <ping@lfw.org>' __author__ = 'Ka-Ping Yee <ping@lfw.org>'
__credits__ = ('GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, ' __credits__ = ('GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, '
'Skip Montanaro, Raymond Hettinger, Trent Nelson, ' 'Skip Montanaro, Raymond Hettinger, Trent Nelson, '
'Michael Foord') 'Michael Foord')
from codecs import lookup, BOM_UTF8 from codecs import lookup, BOM_UTF8
import collections import collections
from itertools import chain from itertools import chain
import itertools as _itertools import itertools as _itertools
import re import re
import string
from .py3token import * from .py3token import *
cookie_re = re.compile(r'^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)') cookie_re = re.compile(r'^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)')
blank_re = re.compile(br'^[ \t\f]*(?:[#\r\n]|$)') blank_re = re.compile(br'^[ \t\f]*(?:[#\r\n]|$)')
from . import py3token from . import py3token
__all__ = py3token.__all__ + ["tokenize", "detect_encoding", __all__ = py3token.__all__ + ["tokenize", "detect_encoding",
"untokenize", "TokenInfo"] "untokenize", "TokenInfo"]
yield repeat(b'') yield repeat(b'')
return _tokenize(lines(), encoding) return _tokenize(lines(), encoding)
def _tokenize(readline, encoding): def _tokenize(readline, encoding):
lnum = parenlev = continued = 0 lnum = parenlev = continued = 0
namechars = string.ascii_letters + '_'
numchars = '0123456789' numchars = '0123456789'
contstr, needcont = '', 0 contstr, needcont = '', 0
contline = None contline = None
indents = [0] indents = [0]
if encoding is not None: if encoding is not None:
if encoding == "utf-8-sig": if encoding == "utf-8-sig":
# BOM will already have been stripped. # BOM will already have been stripped.
endpats.get(token[1]) or endpats.get(token[1]) or
endpats.get(token[2])) endpats.get(token[2]))
contstr, needcont = line[start:], 1 contstr, needcont = line[start:], 1
contline = line contline = line
break break
else: # ordinary string else: # ordinary string
yield TokenInfo(STRING, token, spos, epos, line) yield TokenInfo(STRING, token, spos, epos, line)
elif initial.isidentifier(): # ordinary name # This doesn't conform to PEP 3131.
elif initial in namechars: # ordinary name
yield TokenInfo(NAME, token, spos, epos, line) yield TokenInfo(NAME, token, spos, epos, line)
elif initial == '\\': # continued stmt elif initial == '\\': # continued stmt
continued = 1 continued = 1
else: else:
if initial in '([{': if initial in '([{':
parenlev += 1 parenlev += 1
elif initial in ')]}': elif initial in ')]}':
parenlev -= 1 parenlev -= 1