This is an archive of the discontinued Mercurial Phabricator instance.

Differential D7225

import-checker: open all source files as utf-8
ClosedPublic

Authored by indygreg on Nov 4 2019, 11:57 PM.

Details

Reviewers
pulkit
Group Reviewers
hg-reviewers
Commits
rHGa8454e846736: import-checker: open all source files as utf-8
Summary

Before, we opened in text mode and used the default encoding
to interpret the bytes within.

This caused problems interpreting some byte sequences in some
files.

This commit changes things to always open files as UTF-8, which
makes the error go away.

test-check-module-imports.t now passes on Python 3.5 and 3.6
with this change.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Nov 4 2019, 11:57 PM
pulkit accepted this revision.Nov 5 2019, 4:16 AM
This revision is now accepted and ready to land.Nov 5 2019, 4:16 AM
indygreg added a commit: rHGa8454e846736: import-checker: open all source files as utf-8.Nov 5 2019, 11:03 AM
Closed by commit rHGa8454e846736: import-checker: open all source files as utf-8 (authored by indygreg). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mcontrib/import-checker.py (7 lines)

Diff 17556

contrib/import-checker.py

#!/usr/bin/env python #!/usr/bin/env python
from __future__ import absolute_import, print_function from __future__ import absolute_import, print_function
import ast import ast
import collections import collections
import io
import os import os
import sys import sys
# Import a minimal set of stdlib modules needed for list_stdlib_modules() # Import a minimal set of stdlib modules needed for list_stdlib_modules()
# to work when run from a virtualenv. The modules were chosen empirically # to work when run from a virtualenv. The modules were chosen empirically
# so that the return value matches the return value without virtualenv. # so that the return value matches the return value without virtualenv.
if True: # disable lexical sorting checks if True: # disable lexical sorting checks
try: try:
the input file. the input file.
""" """
py = False py = False
if not f.endswith('.t'): if not f.endswith('.t'):
with open(f, 'rb') as src: with open(f, 'rb') as src:
yield src.read(), modname, f, 0 yield src.read(), modname, f, 0
py = True py = True
if py or f.endswith('.t'): if py or f.endswith('.t'):
with open(f, 'r') as src: # Strictly speaking we should sniff for the magic header that denotes
# Python source file encoding. But in reality we don't use anything
# other than ASCII (mainly) and UTF-8 (in a few exceptions), so
# simplicity is fine.
with io.open(f, 'r', encoding='utf-8') as src:
for script, modname, t, line in embedded(f, modname, src): for script, modname, t, line in embedded(f, modname, src):
yield script, modname.encode('utf8'), t, line yield script, modname.encode('utf8'), t, line
def main(argv): def main(argv):
if len(argv) < 2 or (argv[1] == '-' and len(argv) > 2): if len(argv) < 2 or (argv[1] == '-' and len(argv) > 2):
print('Usage: %s {-|file [file] [file] ...}') print('Usage: %s {-|file [file] [file] ...}')
return 1 return 1