( )⚙ D10943 run-tests: stop writing a `python3` symlink pointing to python2

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10943

run-tests: stop writing a `python3` symlink pointing to python2
ClosedPublic

Authored by marmoute on Jul 2 2021, 6:37 PM.

Details

Reviewers
Alphare
Group Reviewers
hg-reviewers
Commits
rHG23f5ed6dbcb1: run-tests: stop writing a `python3` symlink pointing to python2
Summary

Having python3 actually pointing to python2 is bad. So we stop doing so.

In addition we need to re-introduce a python executable since some of the
script really need to be able to say "current python" in their shbang. For
example, hghave is one of such script.

The faulty changes where introduced by c102b704edb5.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.Jul 2 2021, 6:37 PM
Alphare accepted this revision.Jul 5 2021, 5:33 AM
This revision is now accepted and ready to land.Jul 5 2021, 5:33 AM
Mathiasdm added a subscriber: Mathiasdm.Jul 5 2021, 6:02 AM

I agree that python3 pointing to python2 is not good. But won't this change break on distributions that don't have a 'python' executable? For example on CentOS 8: python2 and python3 exist, but 'python' does not.

marmoute added a comment.Jul 6 2021, 3:15 AM
In D10943#167821, @Mathiasdm wrote:

I agree that python3 pointing to python2 is not good. But won't this change break on distributions that don't have a 'python' executable? For example on CentOS 8: python2 and python3 exist, but 'python' does not.

no because the test runner is now explicitly creating an appropriate symlink from python → "the python used to run run-tests.py"

Mathiasdm added a comment.Jul 6 2021, 3:39 AM

Then it looks fine to me.

marmoute added a comment.Jul 6 2021, 7:16 AM

I added a test to check that python2/python/python3 all works and return the right version.

baymax updated this revision to Diff 28820.Jul 6 2021, 9:32 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

Alphare added inline comments.Jul 6 2021, 9:37 AM
tests/test-run-tests.t
2052

chmod +x will need to be gated by a #if execbit on Windows, but let's figure it out on a Windows machine and take this as is for now as there might be an issue with the script running at all without an sh prefix.

baymax updated this revision to Diff 28828.Jul 6 2021, 10:14 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

Revision Contents
Changeset List

Diff 28836

tests/dumbhttp.py

#!/usr/bin/env python3 #!/usr/bin/env python
from __future__ import absolute_import from __future__ import absolute_import
""" """
Small and dumb HTTP server for use in tests. Small and dumb HTTP server for use in tests.
""" """
import optparse import optparse

tests/dummysmtpd.py

#!/usr/bin/env python3 #!/usr/bin/env python
"""dummy SMTP server for use in tests""" """dummy SMTP server for use in tests"""
from __future__ import absolute_import from __future__ import absolute_import
import asyncore import asyncore
import optparse import optparse
import smtpd import smtpd

tests/get-with-headers.py

#!/usr/bin/env python3 #!/usr/bin/env python
"""This does HTTP GET requests given a host:port and path and returns """This does HTTP GET requests given a host:port and path and returns
a subset of the headers plus the body of the result.""" a subset of the headers plus the body of the result."""
from __future__ import absolute_import from __future__ import absolute_import
import argparse import argparse
import json import json

tests/hghave

#!/usr/bin/env python3 #!/usr/bin/env python
"""Test the running system for features availability. Exit with zero """Test the running system for features availability. Exit with zero
if all features are there, non-zero otherwise. If a feature name is if all features are there, non-zero otherwise. If a feature name is
prefixed with "no-", the absence of feature is tested. prefixed with "no-", the absence of feature is tested.
""" """
from __future__ import absolute_import, print_function from __future__ import absolute_import, print_function
import hghave import hghave

tests/run-tests.py

os.remove(f) os.remove(f)
except OSError: except OSError:
pass pass
def _usecorrectpython(self): def _usecorrectpython(self):
"""Configure the environment to use the appropriate Python in tests.""" """Configure the environment to use the appropriate Python in tests."""
# Tests must use the same interpreter as us or bad things will happen. # Tests must use the same interpreter as us or bad things will happen.
if sys.platform == 'win32': if sys.platform == 'win32':
pyexename = b'python.exe' pyexe_names = [b'python', b'python.exe']
elif sys.version_info[0] < 3:
pyexe_names = [b'python', b'python2']
else: else:
pyexename = b'python3' # XXX this is wrong with python2... pyexe_names = [b'python', b'python3']
# os.symlink() is a thing with py3 on Windows, but it requires # os.symlink() is a thing with py3 on Windows, but it requires
# Administrator rights. # Administrator rights.
if getattr(os, 'symlink', None) and os.name != 'nt': if getattr(os, 'symlink', None) and os.name != 'nt':
msg = "# Making python executable in test path a symlink to '%s'" msg = "# Making python executable in test path a symlink to '%s'"
msg %= sysexecutable msg %= sysexecutable
vlog(msg) vlog(msg)
for pyexename in [pyexename]: for pyexename in pyexe_names:
mypython = os.path.join(self._tmpbindir, pyexename) mypython = os.path.join(self._tmpbindir, pyexename)
try: try:
if os.readlink(mypython) == sysexecutable: if os.readlink(mypython) == sysexecutable:
continue continue
os.unlink(mypython) os.unlink(mypython)
except OSError as err: except OSError as err:
if err.errno != errno.ENOENT: if err.errno != errno.ENOENT:
raise raise
# Windows doesn't have `python3.exe`, and MSYS cannot understand the # Windows doesn't have `python3.exe`, and MSYS cannot understand the
# reparse point with that name provided by Microsoft. Create a # reparse point with that name provided by Microsoft. Create a
# simple script on PATH with that name that delegates to the py3 # simple script on PATH with that name that delegates to the py3
# launcher so the shebang lines work. # launcher so the shebang lines work.
if os.getenv('MSYSTEM'): if os.getenv('MSYSTEM'):
with open(osenvironb[b'RUNTESTDIR'] + b'/python3', 'wb') as f: with open(osenvironb[b'RUNTESTDIR'] + b'/python3', 'wb') as f:
f.write(b'#!/bin/sh\n') f.write(b'#!/bin/sh\n')
f.write(b'py -3.%d "$@"\n' % sys.version_info[1]) f.write(b'py -3.%d "$@"\n' % sys.version_info[1])
if os.getenv('MSYSTEM'):
with open(osenvironb[b'RUNTESTDIR'] + b'/python2', 'wb') as f:
f.write(b'#!/bin/sh\n')
f.write(b'py -2.%d "$@"\n' % sys.version_info[1])
exedir, exename = os.path.split(sysexecutable) exedir, exename = os.path.split(sysexecutable)
for pyexename in pyexe_names:
msg = "# Modifying search path to find %s as %s in '%s'" msg = "# Modifying search path to find %s as %s in '%s'"
msg %= (exename, pyexename, exedir) msg %= (exename, pyexename, exedir)
vlog(msg) vlog(msg)
path = os.environ['PATH'].split(os.pathsep) path = os.environ['PATH'].split(os.pathsep)
while exedir in path: while exedir in path:
path.remove(exedir) path.remove(exedir)
# Binaries installed by pip into the user area like pylint.exe may # Binaries installed by pip into the user area like pylint.exe may
# not be in PATH by default. # not be in PATH by default.
extra_paths = [exedir] extra_paths = [exedir]
vi = sys.version_info vi = sys.version_info
appdata, appdata,
'Python', 'Python',
'Scripts', 'Scripts',
) )
extra_paths.append(scripts_dir) extra_paths.append(scripts_dir)
os.environ['PATH'] = os.pathsep.join(extra_paths + path) os.environ['PATH'] = os.pathsep.join(extra_paths + path)
for pyexename in pyexe_names:
if not self._findprogram(pyexename): if not self._findprogram(pyexename):
print("WARNING: Cannot find %s in search path" % pyexename) print("WARNING: Cannot find %s in search path" % pyexename)
def _installhg(self): def _installhg(self):
"""Install hg into the test environment. """Install hg into the test environment.
This will also configure hg with the appropriate testing settings. This will also configure hg with the appropriate testing settings.
""" """
vlog("# Performing temporary installation of HG") vlog("# Performing temporary installation of HG")
installerrs = os.path.join(self._hgtmp, b"install.err") installerrs = os.path.join(self._hgtmp, b"install.err")

tests/test-filelog.py

#!/usr/bin/env python3 #!/usr/bin/env python
""" """
Tests the behavior of filelog w.r.t. data starting with '\1\n' Tests the behavior of filelog w.r.t. data starting with '\1\n'
""" """
from __future__ import absolute_import, print_function from __future__ import absolute_import, print_function
from mercurial.node import hex from mercurial.node import hex
from mercurial import ( from mercurial import (
hg, hg,

tests/test-remotefilelog-datapack.py

#!/usr/bin/env python3 #!/usr/bin/env python
from __future__ import absolute_import, print_function from __future__ import absolute_import, print_function
import hashlib import hashlib
import os import os
import random import random
import shutil import shutil
import stat import stat
import struct import struct

tests/test-remotefilelog-histpack.py

#!/usr/bin/env python3 #!/usr/bin/env python
from __future__ import absolute_import from __future__ import absolute_import
import hashlib import hashlib
import os import os
import random import random
import shutil import shutil
import stat import stat
import struct import struct

tests/test-run-tests.t

ERROR: test-conditional-matching.t#foo output changed ERROR: test-conditional-matching.t#foo output changed
! !
Failed test-conditional-matching.t#bar: output changed Failed test-conditional-matching.t#bar: output changed
Failed test-conditional-matching.t#foo: output changed Failed test-conditional-matching.t#foo: output changed
# Ran 2 tests, 0 skipped, 2 failed. # Ran 2 tests, 0 skipped, 2 failed.
python hash seed: * (glob) python hash seed: * (glob)
[1] [1]
Test that a proper "python" has been set up
===========================================
(with a small check-code work around)
$ printf "#!/usr/bi" > test-py3.tmp
$ printf "n/en" >> test-py3.tmp
$ cat << EOF >> test-py3.tmp
> v python3
> import sys
> print('.'.join(str(x) for x in sys.version_info))
> EOF
$ mv test-py3.tmp test-py3.py
$ chmod +x test-py3.py
AlphareUnsubmitted
Not Done

chmod +x will need to be gated by a #if execbit on Windows, but let's figure it out on a Windows machine and take this as is for now as there might be an issue with the script running at all without an sh prefix.

Alphare: `chmod +x` will need to be gated by a `#if execbit` on Windows, but let's figure it out on a…
(with a small check-code work around)
$ printf "#!/usr/bi" > test-py.tmp
$ printf "n/en" >> test-py.tmp
$ cat << EOF >> test-py.tmp
> v python
> import sys
> print('.'.join(str(x) for x in sys.version_info))
> EOF
$ mv test-py.tmp test-py.py
$ chmod +x test-py.py
$ ./test-py3.py
3.* (glob)
$ ./test-py.py
2.* (glob) (no-py3 !)
3.* (glob) (py3 !)

tests/test-status-inprocess.py

#!/usr/bin/env python3 #!/usr/bin/env python
from __future__ import absolute_import, print_function from __future__ import absolute_import, print_function
import sys import sys
from mercurial import ( from mercurial import (
commands, commands,
localrepo, localrepo,
ui as uimod, ui as uimod,

tests/test-stdio.py

#!/usr/bin/env python3 #!/usr/bin/env python
""" """
Tests the buffering behavior of stdio streams in `mercurial.utils.procutil`. Tests the buffering behavior of stdio streams in `mercurial.utils.procutil`.
""" """
from __future__ import absolute_import from __future__ import absolute_import
import contextlib import contextlib
import errno import errno
import os import os

tests/tinyproxy.py

#!/usr/bin/env python3 #!/usr/bin/env python
from __future__ import absolute_import, print_function from __future__ import absolute_import, print_function
__doc__ = """Tiny HTTP Proxy. __doc__ = """Tiny HTTP Proxy.
This module implements GET, HEAD, POST, PUT and DELETE methods This module implements GET, HEAD, POST, PUT and DELETE methods
on BaseHTTPServer, and behaves as an HTTP proxy. The CONNECT on BaseHTTPServer, and behaves as an HTTP proxy. The CONNECT
method is also implemented experimentally, but has not been method is also implemented experimentally, but has not been