This is an archive of the discontinued Mercurial Phabricator instance.

Differential D11081

run-test: rework the redirection script for python on windows
ClosedPublic

Authored by marmoute on Jul 11 2021, 5:37 PM.

Details

Reviewers
Alphare
Group Reviewers
hg-reviewers
Commits
rHGe1130abae834: run-test: rework the redirection script for python on windows
Summary

This should get use something something overall simpler and clearer. Especially,
we now have a python.exe script (even if by default Windows has
C:\Python27\python.exe hardcoded in the register to open .py file)

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

marmoute created this revision.Jul 11 2021, 5:37 PM
mharbison72 added a subscriber: mharbison72.Jul 11 2021, 7:54 PM
mharbison72 added inline comments.
tests/run-tests.py
3609–3616

Can you take a look at this line and the commit comment? I was confused by the commit comment because it mentions the file association stuff on Windows (I'm pretty sure we tried to avoid that), and this line talks about not using a symlink, but I didn't see that before.

marmoute added inline comments.Jul 11 2021, 8:30 PM
tests/run-tests.py
3609–3616

The symlinks we are making a reference to are in the first if block. Nothing to do with the previous state of the code before this commit. This is a reference to other code in the file.

marmoute updated this revision to Diff 29234.Jul 13 2021, 2:29 PM
Alphare added a subscriber: Alphare.Jul 15 2021, 9:30 AM
Alphare added inline comments.
tests/run-tests.py
3630

This probably won't work with the CI now that we don't install the py executable anymore on Windows.

marmoute added inline comments.Jul 15 2021, 9:35 AM
tests/run-tests.py
3630

Which is fine, because we do not run the Windows CI with Python2. We only do Python3 run.

Alphare accepted this revision.Jul 15 2021, 9:36 AM
Alphare added inline comments.
tests/run-tests.py
3630

Yes, I thought it was worth mentioning

This revision is now accepted and ready to land.Jul 15 2021, 9:36 AM

Revision Contents
Changeset List

PathPackages
Mtests/run-tests.py (103 lines)

Diff 29234

tests/run-tests.py

try: try:
os.remove(f) os.remove(f)
except OSError: except OSError:
pass pass
def _usecorrectpython(self): def _usecorrectpython(self):
"""Configure the environment to use the appropriate Python in tests.""" """Configure the environment to use the appropriate Python in tests."""
# Tests must use the same interpreter as us or bad things will happen. # Tests must use the same interpreter as us or bad things will happen.
if sys.platform == 'win32': if WINDOWS and PYTHON3:
pyexe_names = [b'python', b'python3', b'python.exe']
elif WINDOWS:
pyexe_names = [b'python', b'python.exe'] pyexe_names = [b'python', b'python.exe']
elif sys.version_info[0] < 3: elif PYTHON3:
pyexe_names = [b'python', b'python2']
else:
pyexe_names = [b'python', b'python3'] pyexe_names = [b'python', b'python3']
else:
pyexe_names = [b'python', b'python2']
# os.symlink() is a thing with py3 on Windows, but it requires # os.symlink() is a thing with py3 on Windows, but it requires
# Administrator rights. # Administrator rights.
if getattr(os, 'symlink', None) and not WINDOWS: if not WINDOWS and getattr(os, 'symlink', None):
msg = "# Making python executable in test path a symlink to '%s'" msg = "# Making python executable in test path a symlink to '%s'"
msg %= sysexecutable msg %= sysexecutable
vlog(msg) vlog(msg)
for pyexename in pyexe_names: for pyexename in pyexe_names:
mypython = os.path.join(self._custom_bin_dir, pyexename) mypython = os.path.join(self._custom_bin_dir, pyexename)
try: try:
if os.readlink(mypython) == sysexecutable: if os.readlink(mypython) == sysexecutable:
continue continue
os.unlink(mypython) os.unlink(mypython)
except OSError as err: except OSError as err:
if err.errno != errno.ENOENT: if err.errno != errno.ENOENT:
raise raise
if self._findprogram(pyexename) != sysexecutable: if self._findprogram(pyexename) != sysexecutable:
try: try:
os.symlink(sysexecutable, mypython) os.symlink(sysexecutable, mypython)
self._createdfiles.append(mypython) self._createdfiles.append(mypython)
except OSError as err: except OSError as err:
# child processes may race, which is harmless # child processes may race, which is harmless
if err.errno != errno.EEXIST: if err.errno != errno.EEXIST:
raise raise
elif WINDOWS and not os.getenv('MSYSTEM'):
raise AssertionError('cannot run test on Windows without MSYSTEM')
else: else:
# Windows doesn't have `python3.exe`, and MSYS cannot understand the # Generate explicit file instead of symlink
# reparse point with that name provided by Microsoft. Create a #
# simple script on PATH with that name that delegates to the py3 # This is especially important as Windows doesn't have
# launcher so the shebang lines work. # `python3.exe`, and MSYS cannot understand the reparse point with
if os.getenv('MSYSTEM'): # that name provided by Microsoft. Create a simple script on PATH
py3exe_name = os.path.join(self._custom_bin_dir, b'python3') # with that name that delegates to the py3 launcher so the shebang
with open(py3exe_name, 'wb') as f: # lines work.
esc_executable = _sys2bytes(shellquote(sysexecutable))
mharbison72Unsubmitted
Not Done

Can you take a look at this line and the commit comment? I was confused by the commit comment because it mentions the file association stuff on Windows (I'm pretty sure we tried to avoid that), and this line talks about not using a symlink, but I didn't see that before.

mharbison72: Can you take a look at this line and the commit comment? I was confused by the commit comment…
marmouteAuthorUnsubmitted
Done

The symlinks we are making a reference to are in the first if block. Nothing to do with the previous state of the code before this commit. This is a reference to other code in the file.

marmoute: The symlinks we are making a reference to are in the first `if` block. Nothing to do with the…
for pyexename in pyexe_names:
stub_exec_path = os.path.join(self._custom_bin_dir, pyexename)
with open(stub_exec_path, 'wb') as f:
f.write(b'#!/bin/sh\n') f.write(b'#!/bin/sh\n')
f.write(b'py -3.%d "$@"\n' % sys.version_info[1]) f.write(b'%s "$@"\n' % esc_executable)
pyexe_name = os.path.join(self._custom_bin_dir, b'python') if WINDOWS:
with open(pyexe_name, 'wb') as f: if not PYTHON3:
# lets try to build a valid python3 executable for the
# scrip that requires it.
py3exe_name = os.path.join(self._custom_bin_dir, b'python3')
with open(py3exe_name, 'wb') as f:
f.write(b'#!/bin/sh\n') f.write(b'#!/bin/sh\n')
f.write(b'py -%d.%d "$@"\n' % sys.version_info[0:2]) f.write(b'py -3 "$@"\n')
AlphareUnsubmitted
Not Done

This probably won't work with the CI now that we don't install the py executable anymore on Windows.

Alphare: This probably won't work with the CI now that we don't install the `py` executable anymore on…
marmouteAuthorUnsubmitted
Done

Which is fine, because we do not run the Windows CI with Python2. We only do Python3 run.

marmoute: Which is fine, because we do not run the Windows CI with Python2. We only do Python3 run.
AlphareUnsubmitted
Not Done

Yes, I thought it was worth mentioning

Alphare: Yes, I thought it was worth mentioning
exedir, exename = os.path.split(sysexecutable) # adjust the path to make sur the main python finds it own dll
for pyexename in pyexe_names:
msg = "# Modifying search path to find %s as %s in '%s'"
msg %= (exename, pyexename, exedir)
vlog(msg)
path = os.environ['PATH'].split(os.pathsep) path = os.environ['PATH'].split(os.pathsep)
while exedir in path: main_exec_dir = os.path.dirname(sysexecutable)
path.remove(exedir) extra_paths = [_bytes2sys(self._custom_bin_dir), main_exec_dir]
# Binaries installed by pip into the user area like pylint.exe may # Binaries installed by pip into the user area like pylint.exe may
# not be in PATH by default. # not be in PATH by default.
extra_paths = [exedir]
vi = sys.version_info
appdata = os.environ.get('APPDATA') appdata = os.environ.get('APPDATA')
vi = sys.version_info
if appdata is not None: if appdata is not None:
scripts_dir = os.path.join( python_dir = 'Python%d%d' % (vi[0], vi[1])
appdata, scripts_path = [appdata, 'Python', python_dir, 'Scripts']
'Python', if not PYTHON3:
'Python%d%d' % (vi[0], vi[1]), scripts_path = [appdata, 'Python', 'Scripts']
'Scripts', scripts_dir = os.path.join(*scripts_path)
)
if vi.major == 2:
scripts_dir = os.path.join(
appdata,
'Python',
'Scripts',
)
extra_paths.append(scripts_dir) extra_paths.append(scripts_dir)
os.environ['PATH'] = os.pathsep.join(extra_paths + path) os.environ['PATH'] = os.pathsep.join(extra_paths + path)
for pyexename in pyexe_names:
if not self._findprogram(pyexename):
print("WARNING: Cannot find %s in search path" % pyexename)
def _use_correct_mercurial(self): def _use_correct_mercurial(self):
target_exec = os.path.join(self._custom_bin_dir, b'hg') target_exec = os.path.join(self._custom_bin_dir, b'hg')
if self._hgcommand != b'hg': if self._hgcommand != b'hg':
# shutil.which only accept bytes from 3.8 # shutil.which only accept bytes from 3.8
real_exec = which(self._hgcommand) real_exec = which(self._hgcommand)
if real_exec is None: if real_exec is None:
raise ValueError('could not find exec path for "%s"', real_exec) raise ValueError('could not find exec path for "%s"', real_exec)