This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8337

pycompat: change argv conversion semantics
ClosedPublic

Authored by indygreg on Mar 28 2020, 9:40 PM.

Download Raw Diff

Details

Reviewers

None

Group Reviewers

hg-reviewers

Commits

rHG00e0c5c06ed5: pycompat: change argv conversion semantics

Summary

Use of os.fsencode() to convert Python's sys.argv back to bytes
was not correct because it isn't the logically inverse operation
from what CPython was doing under the hood.

This commit changes the logic for doing the str -> bytes
conversion. This required a separate implementation for
POSIX and Windows.

The Windows behavior is arguably not ideal. The previous
behavior on Windows was leading to failing tests, such as
test-http-branchmap.t, which defines a utf-8 branch name
via a command argument. Previously, Mercurial's argument
parser looked to be receiving wchar_t bytes in some cases.
After this commit, behavior on Windows is compatible with
Python 2, where CPython did not implement int wmain() and
Windows was performing a Unicode to ANSI conversion on the
wchar_t native command line.

Arguably better behavior on Windows would be for Mercurial to
preserve the original Unicode sequence coming from Python and
to wrap this in a bytes-like type so we can round trip safely.
But, this would be new, backwards incompatible behavior. My
goal for this commit was to converge Mercurial behavior on
Python 3 on Windows to fix busted tests. And I believe I was
successful, as this commit fixes 9 tests on my Windows
machine and 14 tests in the AWS CI environment!

Diff Detail

Repository

rHG Mercurial

Branch

default

Lint

No Linters Available

Unit

No Unit Test Coverage

Event Timeline

indygreg created this revision.Mar 28 2020, 9:40 PM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptMar 28 2020, 9:40 PM

Herald added a subscriber: mercurial-devel. · View Herald Transcript

@yuja I'd appreciate your eyes on this since you have a firm grasp on Windows/Unicode matters...

# Since Python 3 converts argv to wchar_t type by Py_DecodeLocale() on Unix,

# we can use os.fsencode() to get back bytes argv.

#

# https://hg.python.org/cpython/file/v3.5.1/Programs/python.c#l55

#

# On Windows, the native argv is unicode and is converted to MBCS bytes

# since we do enable the legacy filesystem encoding. if getattr(sys, 'argv', None) is not None:

sysargv = list(map(os.fsencode, sys.argv))

+ # On POSIX, the char argv array is converted to Python str using
+ # Py_DecodeLocale(). The inverse of this is Py_EncodeLocale(), which isn't
+ # directly callable from Python code. So, we need to emulate it.
+ # Py_DecodeLocale() calls mbstowcs() and falls back to mbrtowc() with
+ # surrogateescape error handling on failure. These functions take the
+ # current system locale into account. So, the inverse operation is to
+ # .encode() using the system locale's encoding and using the
+ # surrogateescape error handler. The only tricky part here is getting
+ # the system encoding correct, since locale.getlocale() can return
+ # None. We fall back to the filesystem encoding if lookups via locale
+ # fail, as this seems like a reasonable thing to do.
+ #
+ # On Windows, the wchar_t argv is passed into the interpreter as-is.
+ # Like POSIX, we need to emulate what Py_EncodeLocale() would do. But
+ # there's an additional wrinkle. What we really want to access is the
+ # ANSI codepage representation of the arguments, as this is what
+ # int main() would receive if Python 3 didn't define int wmain()
+ # (this is how Python 2 worked). To get that, we encode with the mbcs
+ # encoding, which will pass CP_ACP to the underlying Windows API to
+ # produce bytes.
+ if os.name == r'nt':
+ sysargv = [a.encode("mbcs", "ignore") for a in sys.argv]

On Windows, my assumption was os.fsencode() == .encode("mbcs") if
sys._enablelegacywindowsfsencoding(). So this looks good to me.
Perhaps, the "ignore" error mode would match the legacy Windows behavior.

+ else:
+ encoding = (
+ locale.getlocale()[1]
+ or locale.getdefaultlocale()[1]
+ or sys.getfilesystemencoding()
+ )
+ sysargv = [a.encode(encoding, "surrogateescape") for a in sys.argv]

I'm not pretty sure if the locale encoding is the encoding Py_DecodeLocale()
would use. There are many ifdefs for __APPLE__. The doc says use
os.fsencode(), but that's no longer valid (or wrong from the start)?

https://docs.python.org/3/library/sys.html#sys.argv

Something might be changed around 3.7 or 3.8. Since bytes argv handling
has been moved from int main() to preconfig.c, things could become
more dynamic. But I don't know. Just my guess.

Overall, the new code looks good, but I have no idea if that's more correct.

This looks like it also fixes the phabricator test on Windows, which diverged by creating a different hash for the create alpha for phabricator test € commit.

In D8337#124542, @yuja wrote:

On Windows, my assumption was os.fsencode() == .encode("mbcs") if
sys._enablelegacywindowsfsencoding(). So this looks good to me.
Perhaps, the "ignore" error mode would match the legacy Windows behavior.

Does this mean it could be a problem running from source on Windows? For example, hg version (as opposed to hg.exe version) seems to be equivalent to python hg, which obviously doesn't have the proper environment variable or C API option to enable legacy mode. Should there be code early on that detects this and warns/aborts?

> On Windows, my assumption was os.fsencode() == .encode("mbcs") if
> sys._enablelegacywindowsfsencoding(). So this looks good to me.
> Perhaps, the "ignore" error mode would match the legacy Windows behavior.
Does this mean it could be a problem running from source on Windows?  For example, `hg version` (as opposed to `hg.exe version`) seems to be equivalent to `python hg`, which obviously doesn't have the proper environment variable or C API option to enable legacy mode.  Should there be code early on that detects this and warns/aborts?

It's hacked up by pycompat.py so python hg should be fine.

indygreg added a commit: rHG00e0c5c06ed5: pycompat: change argv conversion semantics.Apr 1 2020, 2:55 PM

This revision was not accepted when it landed; it landed in state Needs Review.

Closed by commit rHG00e0c5c06ed5: pycompat: change argv conversion semantics (authored by indygreg). · Explain Why

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

			Path	Packages
M			mercurial/pycompat.py (38 lines)

Commit	Parents	Author	Summary	Date
7cd33a4da668	8befcb36a8ae	Gregory Szorc		Mar 28 2020, 3:18 PM

Status	Author	Revision
Closed	indygreg	D8337 pycompat: change argv conversion semantics
Closed	indygreg	D8336 tests: use `f --hexdump` to print file content
Closed	indygreg	D8335 url: pass str to pathname2url
Closed	indygreg	D8334 tests: pass str to matchoutput()

Diff 20903

mercurial/pycompat.py

	return _rapply(f, xs)			return _rapply(f, xs)


	if ispy3:			if ispy3:
	import builtins			import builtins
	import codecs			import codecs
	import functools			import functools
	import io			import io
				import locale
	import struct			import struct

	if os.name == r'nt' and sys.version_info >= (3, 6):			if os.name == r'nt' and sys.version_info >= (3, 6):
	# MBCS (or ANSI) filesystem encoding must be used as before.			# MBCS (or ANSI) filesystem encoding must be used as before.
	# Otherwise non-ASCII filenames in existing repositories would be			# Otherwise non-ASCII filenames in existing repositories would be
	# corrupted.			# corrupted.
	# This must be set once prior to any fsencode/fsdecode calls.			# This must be set once prior to any fsencode/fsdecode calls.
	sys._enablelegacywindowsfsencoding() # pytype: disable=module-attr			sys._enablelegacywindowsfsencoding() # pytype: disable=module-attr
	long = int			long = int

	# TODO: .buffer might not exist if std streams were replaced; we'll need			# TODO: .buffer might not exist if std streams were replaced; we'll need
	# a silly wrapper to make a bytes stream backed by a unicode one.			# a silly wrapper to make a bytes stream backed by a unicode one.
	stdin = sys.stdin.buffer			stdin = sys.stdin.buffer
	stdout = sys.stdout.buffer			stdout = sys.stdout.buffer
	stderr = sys.stderr.buffer			stderr = sys.stderr.buffer

	# Since Python 3 converts argv to wchar_t type by Py_DecodeLocale() on Unix,
	# we can use os.fsencode() to get back bytes argv.
	#
	# https://hg.python.org/cpython/file/v3.5.1/Programs/python.c#l55
	#
	# On Windows, the native argv is unicode and is converted to MBCS bytes
	# since we do enable the legacy filesystem encoding.
	if getattr(sys, 'argv', None) is not None:			if getattr(sys, 'argv', None) is not None:
	sysargv = list(map(os.fsencode, sys.argv))			# On POSIX, the char** argv array is converted to Python str using
				# Py_DecodeLocale(). The inverse of this is Py_EncodeLocale(), which isn't
				# directly callable from Python code. So, we need to emulate it.
				# Py_DecodeLocale() calls mbstowcs() and falls back to mbrtowc() with
				# surrogateescape error handling on failure. These functions take the
				# current system locale into account. So, the inverse operation is to
				# .encode() using the system locale's encoding and using the
				# surrogateescape error handler. The only tricky part here is getting
				# the system encoding correct, since `locale.getlocale()` can return
				# None. We fall back to the filesystem encoding if lookups via `locale`
				# fail, as this seems like a reasonable thing to do.
				#
				# On Windows, the wchar_t **argv is passed into the interpreter as-is.
				# Like POSIX, we need to emulate what Py_EncodeLocale() would do. But
				# there's an additional wrinkle. What we really want to access is the
				# ANSI codepage representation of the arguments, as this is what
				# `int main()` would receive if Python 3 didn't define `int wmain()`
				# (this is how Python 2 worked). To get that, we encode with the mbcs
				# encoding, which will pass CP_ACP to the underlying Windows API to
				# produce bytes.
				if os.name == r'nt':
				sysargv = [a.encode("mbcs", "ignore") for a in sys.argv]
				else:
				encoding = (
				locale.getlocale()[1]
				or locale.getdefaultlocale()[1]
				or sys.getfilesystemencoding()
				)
				sysargv = [a.encode(encoding, "surrogateescape") for a in sys.argv]

	bytechr = struct.Struct('>B').pack			bytechr = struct.Struct('>B').pack
	byterepr = b'%r'.__mod__			byterepr = b'%r'.__mod__

	class bytestr(bytes):			class bytestr(bytes):
	"""A bytes which mostly acts as a Python 2 str			"""A bytes which mostly acts as a Python 2 str

	>>> bytestr(), bytestr(bytearray(b'foo')), bytestr(u'ascii'), bytestr(1)			>>> bytestr(), bytestr(bytearray(b'foo')), bytestr(u'ascii'), bytestr(1)

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	20903		Mar 28 2020, 9:40 PM	★	★
Diff 2	20948	rHG00e0c5c06ed5e63fa098d077272307bd23e1f174	Mar 28 2020, 3:18 PM	★	★