( )⚙ D10689 pyoxidizer: use Python 3.9 (BC)

This is an archive of the discontinued Mercurial Phabricator instance.

pyoxidizer: use Python 3.9 (BC)
ClosedPublic

Authored by indygreg on May 6 2021, 8:00 PM.

Details

Summary

Newer versions of PyOxidizer use Python 3.9 by default. We
previously pinned the version to 3.8 to facilitate porting to
a new PyOxidizer version and diffing results.

Now that the porting work is complete, let's bump Python
to Python 3.9.

This will effectively change our Windows Inno and WiX Python 3
installers from Python 3.8 to 3.9.

.. bc::

Windows .msi and .exe installers now use Python 3.9 instead of
Python 3.8.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

indygreg created this revision.May 6 2021, 8:00 PM
baymax updated this revision to Diff 27760.May 7 2021, 11:30 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

I think I've hit a dead end trying to produce installers. I first updated rustup and pyoxidizer on my laptop that I've used in the past to successfully build pyoxidizer binaries, but it prints an error about a zip file with a date before 1980 not being supported. I switched to a clean system, ran .\contrib\install-windows-dependencies.ps1, and that died because it looks like MS removed the py27 compiler. (Maybe that should be cached somewhere for posterity, but I don't think I have a local copy.) Commenting those references out, I was eventually able to get everything installed, but it reproduced the same error:

py -3 contrib/packaging/packaging.py wix --pyoxidizer-target x86_64-pc-windows-msvc
running build_doc
generating doc/common.txt
generating doc/hg-ssh.8.txt
generating doc/hg-ssh.8.gendoc.txt
generating doc/hg-ssh.8.html
normalizing doc/hg-ssh.8.html to LF line endings
generating doc/hg.1.txt
generating doc/hg.1.gendoc.txt
generating doc/hg.1.html
normalizing doc/hg.1.html to LF line endings
generating doc/hgignore.5.txt
generating doc/hgignore.5.gendoc.txt
generating doc/hgignore.5.html
normalizing doc/hgignore.5.html to LF line endings
generating doc/hgrc.5.txt
generating doc/hgrc.5.gendoc.txt
generating doc/hgrc.5.html
normalizing doc/hgrc.5.html to LF line endings
downloading https://versaweb.dl.sourceforge.net/project/gnuwin32/gettext/0.14.4/gettext-0.14.4-bin.zip to C:\Users\mharbison\mercurial\build\wix-x86_64-pc-windows-msvc\gettext-0.14.4-bin.zip
C:\Users\mharbison\mercurial\build\wix-x86_64-pc-windows-msvc\gettext-0.14.4-bin.zip exists and passes integrity checks
downloading https://versaweb.dl.sourceforge.net/project/gnuwin32/gettext/0.14.4/gettext-0.14.4-dep.zip to C:\Users\mharbison\mercurial\build\wix-x86_64-pc-windows-msvc\gettext-0.14.4-dep.zip
C:\Users\mharbison\mercurial\build\wix-x86_64-pc-windows-msvc\gettext-0.14.4-dep.zip exists and passes integrity checks
resolving 1 targets
resolving target msi
resolving target app
resolving target distribution
resolving target exe
resolving Python distribution Url { url: "https://github.com/indygreg/python-build-standalone/releases/download/20210506/cpython-3.9.5-x86_64-pc-windows-msvc-shared-pgo-20210506T0943.tar.zst", sha256: "4956b8c29ab2841f04cd9aa465e3073be2107ced59636e2a82d35b0c8815e217" }
downloading https://github.com/indygreg/python-build-standalone/releases/download/20210506/cpython-3.9.5-x86_64-pc-windows-msvc-shared-pgo-20210506T0943.tar.zst
Python distribution available at C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\cpython-3.9.5-x86_64-pc-windows-msvc-shared-pgo-20210506T0943.tar.zst
reading data from Python distribution...
pip installing to C:\Temp\pyoxidizer-pip-installqd9qZz\install
Using pip 21.1.1 from C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip (python 3.9)
Non-user install due to --prefix or --target option
Created temporary directory: C:\Temp\pip-target-4tvzyhct
Created temporary directory: C:\Temp\pip-ephem-wheel-cache-747t3i1b
Created temporary directory: C:\Temp\pip-req-tracker-0p__z57i
Initialized build tracking at C:\Temp\pip-req-tracker-0p__z57i
Created build tracker: C:\Temp\pip-req-tracker-0p__z57i
Entered build tracker: C:\Temp\pip-req-tracker-0p__z57i
Created temporary directory: C:\Temp\pip-install-7tihpiqe
Processing c:\users\mharbison\mercurial
  Created temporary directory: C:\Temp\pip-req-build-_6p7lhtt
  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.
   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.
  Added file:///C:/Users/mharbison/mercurial to build tracker 'C:\\Temp\\pip-req-tracker-0p__z57i'
  Created temporary directory: C:\Temp\pip-build-env-ibaynzkz
  Created temporary directory: C:\Temp\pip-standalone-pip-4zfitqro
ERROR: Exception:
Traceback (most recent call last):
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip\_internal\cli\base_command.py", line 180, in _main
    status = self.run(options, args)
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip\_internal\cli\req_command.py", line 204, in wrapper
    return func(self, options, args)
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip\_internal\commands\install.py", line 318, in run
    requirement_set = resolver.resolve(
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip\_internal\resolution\resolvelib\resolver.py", line 103, in resolve
    r = self.factory.make_requirement_from_install_req(
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip\_internal\resolution\resolvelib\factory.py", line 418, in make_requirement_from_install_req
    cand = self._make_candidate_from_link(
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip\_internal\resolution\resolvelib\factory.py", line 200, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip\_internal\resolution\resolvelib\candidates.py", line 306, in __init__
    super().__init__(
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip\_internal\resolution\resolvelib\candidates.py", line 151, in __init__
    self.dist = self._prepare()
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip\_internal\resolution\resolvelib\candidates.py", line 234, in _prepare
    dist = self._prepare_distribution()
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip\_internal\resolution\resolvelib\candidates.py", line 317, in _prepare_distribution
    return self._factory.preparer.prepare_linked_requirement(
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip\_internal\operations\prepare.py", line 508, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip\_internal\operations\prepare.py", line 570, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip\_internal\operations\prepare.py", line 60, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip\_internal\distributions\sdist.py", line 34, in prepare_distribution_metadata
    self._setup_isolation(finder)
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip\_internal\distributions\sdist.py", line 62, in _setup_isolation
    self.req.build_env.install_requirements(
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip\_internal\build_env.py", line 198, in install_requirements
    pip_runnable = ctx.enter_context(_create_standalone_pip())
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\contextlib.py", line 429, in enter_context
    result = _cm_type.__enter__(cm)
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\contextlib.py", line 117, in __enter__
    return next(self.gen)
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\site-packages\pip-21.1.1-py3.9.egg\pip\_internal\build_env.py", line 63, in _create_standalone_pip
    zf.write(child, child.relative_to(source.parent).as_posix())
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\zipfile.py", line 1727, in write
    zinfo = ZipInfo.from_file(filename, arcname,
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\zipfile.py", line 517, in from_file
    zinfo = cls(arcname, date_time)
  File "C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\python\install\lib\zipfile.py", line 361, in __init__
    raise ValueError('ZIP does not support timestamps before 1980')
ValueError: ZIP does not support timestamps before 1980
Removed file:///C:/Users/mharbison/mercurial from build tracker 'C:\\Temp\\pip-req-tracker-0p__z57i'
Removed build tracker: 'C:\\Temp\\pip-req-tracker-0p__z57i'
error[PYOXIDIZER_PYTHON_EXECUTABLE]: calling pip install

Caused by:
    command ["C:\\Users\\mharbison\\AppData\\Local\\pyoxidizer\\python_distributions\\python.4956b8c29ab2\\python\\install/python.exe", "-m", "pip", "--disable-pip-version-check", "install", "--target", "C:\\Temp\\pyoxidizer-pip-installqd9qZz\\install", "--verbose", "C:\\Users\\mharbison\\mercurial\\rust\\hgcli/../.."] exited with code 2
  --> C:\Users\mharbison\mercurial\rust\hgcli\pyoxidizer.bzl:99:30
   |
99 |     exe.add_python_resources(exe.pip_install(["--verbose", ROOT]))
   |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PythonExecutable.pip_install()


error: calling pip install

Caused by:
    command ["C:\\Users\\mharbison\\AppData\\Local\\pyoxidizer\\python_distributions\\python.4956b8c29ab2\\python\\install/python.exe", "-m", "pip", "--disable-pip-version-check", "install", "--target", "C:\\Temp\\pyoxidizer-pip-installqd9qZz\\install", "--verbose", "C:\\Users\\mharbison\\mercurial\\rust\\hgcli/../.."] exited with code 2

A couple of other observations- pyoxidizer is on PATH on my laptop in %USERPROFILE%/.cargo/bin- but on this fresh system, the only copy was in %PROGRAMFILES% and not on PATH. (I assume that's due to an MSI install vs what I assume was a cargo install long ago.) That caused the build to fail earlier with a generic "file not found" type message. I'd feel weird about the install script adjusting PATH; OTOH this already requires Administrator rights and installs stuff globally. (Not using an admin prompt was also one of the errors I hit- IDK if there's a way to detect this up front in the install script and exit with a clearer message.) I'm not too concerned about this stuff, but I wanted to write it down before I forget.

Yeah, I was hitting this zip file error too. I'm not sure what's causing it. But I'm actively looking into fixing it!

baymax updated this revision to Diff 27797.May 10 2021, 11:38 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

The refresh on this series was me applying a topic and pushing so that I could pull to the CI system for experimenting. Sorry about that- I forgot about the refresh.

The behavior seems pretty good, but I wanted to highlight a few things to see what you think.

  • After updating pyoxidizer via install-windows-dependencies.ps1, I still got the error about the zip file date until I figured out that I needed to remove C:\Users\mharbison\AppData\Local\pyoxidizer\python_distributions\python.4956b8c29ab2\ and the *.tar.zst file next to it. IDR how I figured out I needed to do that, since that's scrolled off the screen. If it's a one-off, it's probably not a big deal. But if there are scenarios where the content of these things depend on the version of pyoxidizer that built it, maybe some mechanism to detect this and force a rebuild would be a good thing in the long term.
  • These were the last few lines of the installer build output:
running light to produce C:\Users\mharbison\mercurial\build\pyoxidizer\x86_64-pc-windows-msvc\release\msi\mercurial-5.8.msi
C:\Users\mharbison\mercurial\build\pyoxidizer\wxs\mercurial.wxs(55) : warning LGHT1076 : ICE40: REINSTALLMODE is defined in the Property table. This may cause difficulties.
C:\Users\mharbison\mercurial\build\pyoxidizer\wxs\mercurial.wxs(147) : warning LGHT1076 : ICE61: This product should remove only older versions of itself. No Maximum version was detected for the current product. (INSTALLEDMERCURIALPRODUCTS)

IDK if the REINSTALLMODE warning matters, or what it is alluding to.

  • When I ran the installer, I got a message in the system notifications that "An app caused a problem with your default browser setting, so it was reset to Internet Explorer" as it was finishing up. I've never seen that before, am not certain what the default was set to on this system before, and it didn't happen a second time after I set the default to FireFox, uninstalled and then reran the installer.

running light to produce C:\Users\mharbison\mercurial\build\pyoxidizer\x86_64-pc-windows-msvc\release\msi\mercurial-5.8.msi
C:\Users\mharbison\mercurial\build\pyoxidizer\wxs\mercurial.wxs(55) : warning LGHT1076 : ICE40: REINSTALLMODE is defined in the Property table. This may cause difficulties.
C:\Users\mharbison\mercurial\build\pyoxidizer\wxs\mercurial.wxs(147) : warning LGHT1076 : ICE61: This product should remove only older versions of itself. No Maximum version was detected for the current product. (INSTALLEDMERCURIALPRODUCTS)

IDK if the `REINSTALLMODE` warning matters, or what it is alluding to.

This warning has existed for years AFAICT. However, the Python code is suppressing the warning via an argument to light.exe. This argument is not specified in PyOxidizer because PyOxidizer doesn't expose a mechanism to add this type of argument to light.exe invocations.

Should we address the warning? Maybe. But it would be part of a separate series.

  • When I ran the installer, I got a message in the system notifications that "An app caused a problem with your default browser setting, so it was reset to Internet Explorer" as it was finishing up. I've never seen that before, am not certain what the default was set to on this system before, and it didn't happen a second time after I set the default to FireFox, uninstalled and then reran the installer.

That's super strange. I never saw this when testing locally. I can't imagine how Mercurial's MSI installer is triggering that. My guess is it has something to do with triggering the Windows installer service than Mercurial itself. Keep in mind that the WiX XML built through PyOxidizer is very similar to what we were doing before. The main difference is the auto-generated .wxs file holding the fragments for the installed files.

mharbison72 accepted this revision.May 11 2021, 12:18 AM

Thanks for grinding through this packaging stuff!

This revision is now accepted and ready to land.May 11 2021, 12:18 AM
This revision was automatically updated to reflect the committed changes.