This is an archive of the discontinued Mercurial Phabricator instance.

Differential D5050

tests: add test verifying our Python 3 tokenizer works
AbandonedPublic

Authored by indygreg on Oct 13 2018, 3:28 AM.

Details

Reviewers
baymax
Group Reviewers
hg-reviewers
Summary

Our vendored Python 3.7 tokenizer is now fully working on Python 2.7.
Let's verify that by adding a test that ensures we can tokenize all .py
files in the repo.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Oct 13 2018, 3:28 AM
baymax requested changes to this revision.Jan 24 2020, 12:32 PM

There seems to have been no activities on this Diff for the past 3 Months.

By policy, we are automatically moving it out of the need-review state.

Please, move it back to need-review without hesitation if this diff should still be discussed.

:baymax:need-review-idle:

This revision now requires changes to proceed.Jan 24 2020, 12:32 PM
indygreg abandoned this revision.Jan 24 2020, 11:51 PM

Revision Contents
Changeset List

Diff 12006

contrib/check-py3-tokenize.py

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
#!/usr/bin/env python
#
# check-py3-tokenize - check that our Python 3 tokenizer works
#
# Copyright Gregory Szorc <gregory.szorc@gmail.com>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
from __future__ import absolute_import, print_function
import io
import sys
from hgdemandimport import (
py3tokenize,
)
def checkfile(path):
with open(path, 'rb') as fh:
b = io.BytesIO(fh.read())
b.seek(0)
try:
tokens = list(py3tokenize.tokenize(b.readline))
except Exception as e:
print('%s failed to tokenize: %s' % (path, e))
return False
try:
py3tokenize.untokenize(tokens)
except Exception as e:
print('%s failed to untokenize: %' % (path, e))
return False
return True
if __name__ == '__main__':
argv = sys.argv
if len(argv) < 2 or (argv[1] == '-' and len(argv) > 2):
print('Usage: %s {-|file [file] [file] ...}')
sys.exit(1)
if argv[1] == '-':
argv = argv[:1]
argv.extend(l.rstrip() for l in sys.stdin.readlines())
code = 0
for path in argv[1:]:
if not checkfile(path):
code = 1
sys.exit(code)

contrib/import-checker.py

import testparseutil import testparseutil
# Whitelist of modules that symbols can be directly imported from. # Whitelist of modules that symbols can be directly imported from.
allowsymbolimports = ( allowsymbolimports = (
'__future__', '__future__',
'bzrlib', 'bzrlib',
'hgclient', 'hgclient',
'hgdemandimport',
'mercurial', 'mercurial',
'mercurial.hgweb.common', 'mercurial.hgweb.common',
'mercurial.hgweb.request', 'mercurial.hgweb.request',
'mercurial.i18n', 'mercurial.i18n',
'mercurial.node', 'mercurial.node',
# for revlog to re-export constant to extensions # for revlog to re-export constant to extensions
'mercurial.revlogutils.constants', 'mercurial.revlogutils.constants',
# for cffi modules to re-export pure functions # for cffi modules to re-export pure functions

tests/test-check-py3-tokenize.t

  • This file was added.
#require test-repo
$ . "$TESTDIR/helpers-testrepo.sh"
$ cd "$TESTDIR"/..
$ testrepohg files 'set:**.py' \
> 'tests/**.t' \
> | sed 's-\\-/-g' | "$PYTHON" contrib/check-py3-tokenize.py -