This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8339

dispatch: force \n for newlines on sys.std* streams (BC)
ClosedPublic

Authored by indygreg on Mar 29 2020, 4:31 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG02fa5392bab6: dispatch: force \n for newlines on sys.std* streams (BC)
Summary

The sys.std* streams behave differently on Python 3. On Python 3,
these streams are an io.TextIOWrapper that wraps a binary buffer
stored on a .buffer attribute. These TextIOWrapper instances
normalize \n to os.linesep by default. On Windows, this means
that \n is normalized to \r\n. So functions like print() which
have an implicit end='\n' will actually emit \r\n for line endings.

While most parts of Mercurial go through the ui.write() layer to
print output, some code - notably in extensions and hooks - can use
print(). If this code was using print() or otherwise writing to
sys.std* on Windows, Mercurial would emit \r\n.

In reality, pretty much everything on Windows reacts to \n just fine.
Mercurial itself doesn't emit \r\n when going through the ui layer.
Changing the sys.std* streams to not normalize line endings sounds
like a scary change. But I think it is safe. It also makes Mercurial
on Python 3 behave similarly to Python 2, which did not perform \r\n
normalization in print() by default.

.. bc:: sys.{stdout, stderr, stdin} now use \n line endings on Python 3

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Mar 29 2020, 4:31 PM
indygreg added a comment.Mar 29 2020, 4:35 PM

As scary as this patch sounds, I'm pretty sure it is safe, as I believe it restores compatibility with Python 2. Changing sys.std* to be binary streams instead of text streams would be a bigger BC break. And that is not a change I want to make, as this would invalidate assumptions in 3rd party code about the behavior of these streams on Python 3!

indygreg added a commit: rHG02fa5392bab6: dispatch: force \n for newlines on sys.std* streams (BC).Apr 1 2020, 3:09 PM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG02fa5392bab6: dispatch: force \n for newlines on sys.std* streams (BC) (authored by indygreg). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/dispatch.py (46 lines)

Diff 20950

mercurial/dispatch.py

# dispatch.py - command dispatching for mercurial # dispatch.py - command dispatching for mercurial
# #
# Copyright 2005-2007 Matt Mackall <mpm@selenic.com> # Copyright 2005-2007 Matt Mackall <mpm@selenic.com>
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
from __future__ import absolute_import, print_function from __future__ import absolute_import, print_function
import difflib import difflib
import errno import errno
import getopt import getopt
import io
import os import os
import pdb import pdb
import re import re
import signal import signal
import sys import sys
import traceback import traceback
_silencestdio() _silencestdio()
sys.exit(status & 255) sys.exit(status & 255)
if pycompat.ispy3: if pycompat.ispy3:
def initstdio(): def initstdio():
pass # stdio streams on Python 3 are io.TextIOWrapper instances proxying another
# buffer. These streams will normalize \n to \r\n by default. Mercurial's
# preferred mechanism for writing output (ui.write()) uses io.BufferedWriter
# instances, which write to the underlying stdio file descriptor in binary
# mode. ui.write() uses \n for line endings and no line ending normalization
# is attempted through this interface. This "just works," even if the system
# preferred line ending is not \n.
#
# But some parts of Mercurial (e.g. hooks) can still send data to sys.stdout
# and sys.stderr. They will inherit the line ending normalization settings,
# potentially causing e.g. \r\n to be emitted. Since emitting \n should
# "just work," here we change the sys.* streams to disable line ending
# normalization, ensuring compatibility with our ui type.
# write_through is new in Python 3.7.
kwargs = {
"newline": "\n",
"line_buffering": sys.stdout.line_buffering,
}
if util.safehasattr(sys.stdout, "write_through"):
kwargs["write_through"] = sys.stdout.write_through
sys.stdout = io.TextIOWrapper(
sys.stdout.buffer, sys.stdout.encoding, sys.stdout.errors, **kwargs
)
kwargs = {
"newline": "\n",
"line_buffering": sys.stderr.line_buffering,
}
if util.safehasattr(sys.stderr, "write_through"):
kwargs["write_through"] = sys.stderr.write_through
sys.stderr = io.TextIOWrapper(
sys.stderr.buffer, sys.stderr.encoding, sys.stderr.errors, **kwargs
)
# No write_through on read-only stream.
sys.stdin = io.TextIOWrapper(
sys.stdin.buffer,
sys.stdin.encoding,
sys.stdin.errors,
# None is universal newlines mode.
newline=None,
line_buffering=sys.stdin.line_buffering,
)
def _silencestdio(): def _silencestdio():
for fp in (sys.stdout, sys.stderr): for fp in (sys.stdout, sys.stderr):
# Check if the file is okay # Check if the file is okay
try: try:
fp.flush() fp.flush()
continue continue
except IOError: except IOError: