Page MenuHomePhabricator

packaging: support building WiX installers with PyOxidizer
ClosedPublic

Authored by indygreg on Apr 21 2020, 10:39 PM.

Details

Summary

We initially implemented PyOxidizer support for Inno installers.
That did most of the heavy work of integrating PyOxidizer into
the packaging system. Implementing WiX installer support was
pretty straightforward.

Aspects of this patch look very similar to Inno's.

The main difference is the handling of the Visual C++
Redistributable Runtime files.

The WiX installer was formerly using merge modules to
install the VC++ 9.0 runtime because this feature is
supported by the WiX installer (it isn't easily available
to Inno installers).

Our strategy for the runtime files is to install the
vcruntime140.dll file next to hg.exe just like any other
file. While we could leverage WiX's functionality for invoking
a VCRedist installer, I don't want to deal with the complexity
at this juncture. So, we let run_pyoxidizer() copy vcruntime140.dll
into the staging directory (like it does for Inno) and our
dynamic WiX XML generator picks it up as a regular file and
installs it.

We did, however, have to teach mercurial.wxs how to conditionally
use the merge modules. But this was rather straightforward.

Comparing the file layout of the WiX installers before and
after:

  • Various lib/*.{pyd, dll} files no longer exist
  • python27.dll was replaced by python37.dll
  • vcruntime140.dll was added

All these changes are expected due to the transition to
Python 3 and to PyOxidizer, which embeded the .pyd and .dll files
in hg.exe.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Apr 21 2020, 10:39 PM

Clarifying: we can still build hg-with-2.x as a traditional install, and can also do PyOxidizer 3.x? or do we lose 2.x support when we land this series?

Clarifying: we can still build hg-with-2.x as a traditional install, and can also do PyOxidizer 3.x? or do we lose 2.x support when we land this series?

We can still do both. Although the filenames of the installers collide. I have some patches queued up that differentiate the installer names to indicate Python 2/3. And I also have some changes to contrib/automation to teach that system to generate Python 3 artifacts.

The goal is to produce Python 2 and 3 variants for the Windows artifacts and distribute those for 5.4. Sometime later (hopefully 5.5!) we can rip out Python 2 support.

indygreg updated this revision to Diff 21220.Apr 24 2020, 6:20 PM

This may not be a big deal, and I'm not sure that this patch is the cause. When I built the py2 installer with py -3 contrib\packaging\packaging.py wix --python C:\Python2717\python.exe, I got a 25K hg.exe in the dist directory with all of the installers that had a mod time 1 minute before the installer. I don't remember seeing this when I was building installers at the end of January. (It also has screwy output, but that's a different issue)

C:\Users\Matt\hg>C:\Users\Matt\hg\dist\hg.exe version
Traceback (most recent call last):
  File "hg", line 43, in <module>
    dispatch.run()
  File "mercurial\dispatch.pyc", line 112, in run

  File "mercurial\dispatch.pyc", line 271, in dispatch

  File "mercurial\ui.pyc", line 308, in load
    >>> u.configbytes(s, b'invalid')
  File "mercurial\rcutil.pyc", line 99, in rccomponents

  File "mercurial\rcutil.pyc", line 69, in default_rc_resources

  File "mercurial\utils\resourceutil.pyc", line 97, in contents

WindowsError: [Error 3] The system cannot find the path specified: 'C:\\Users\\Matt\\hg\\dist\\defaultrc\\*.*'

I'm trying to build this with PyOxidizer and get an error after upgrading to v0.7.0. I wonder if some of the references in rust/hgcli/Cargo.lock need to be updated from 0.7.0-pre. Here's the command output:

C:\Users\Matt\hg>py -3 contrib\packaging\packaging.py wix --pyoxidizer-target x86_64-pc-windows-msvc
downloading https://versaweb.dl.sourceforge.net/project/gnuwin32/gettext/0.14.4/gettext-0.14.4-bin.zip to C:\Users\Matt\hg\build\wix-x86_64-pc-windows-msvc\gettext-0.14.4-bin.zip
C:\Users\Matt\hg\build\wix-x86_64-pc-windows-msvc\gettext-0.14.4-bin.zip exists and passes integrity checks
downloading https://versaweb.dl.sourceforge.net/project/gnuwin32/gettext/0.14.4/gettext-0.14.4-dep.zip to C:\Users\Matt\hg\build\wix-x86_64-pc-windows-msvc\gettext-0.14.4-dep.zip
C:\Users\Matt\hg\build\wix-x86_64-pc-windows-msvc\gettext-0.14.4-dep.zip exists and passes integrity checks
resolving 1 targets
resolving target app_windows
resolving target distribution_windows
resolving target exe_windows
resolving Python distribution Url { url: "https://github.com/indygreg/python-build-standalone/releases/download/20200408/cpython-3.7.7-windows-amd64-shared-pgo-20200409T0115.tar.zst", sha256: "c43c44ebfe9b9f9c59c12481a6233b17bc9a4ad965f8f0dc0063abff4dc59875" }
Python distribution available at C:\Users\Matt\hg\build\pyoxidizer\python_distributions\cpython-3.7.7-windows-amd64-shared-pgo-20200409T0115.tar.zst
reading data from Python distribution...
pip installing to C:\Users\Matt\AppData\Local\Temp\pyoxidizer-pip-install.ILmZHu00Qhwj\install
Non-user install due to --prefix or --target option
Created temporary directory: C:\Users\Matt\AppData\Local\Temp\pip-target-7s2381ds
Created temporary directory: C:\Users\Matt\AppData\Local\Temp\pip-ephem-wheel-cache-wv7fdx_0
Created temporary directory: C:\Users\Matt\AppData\Local\Temp\pip-req-tracker-2zs2cynh
Initialized build tracking at C:\Users\Matt\AppData\Local\Temp\pip-req-tracker-2zs2cynh
Created build tracker: C:\Users\Matt\AppData\Local\Temp\pip-req-tracker-2zs2cynh
Entered build tracker: C:\Users\Matt\AppData\Local\Temp\pip-req-tracker-2zs2cynh
Created temporary directory: C:\Users\Matt\AppData\Local\Temp\pip-install-i2r5lvpe
Processing c:\users\matt\hg
  Created temporary directory: C:\Users\Matt\AppData\Local\Temp\pip-req-build-2l899b53
ERROR: Could not install packages due to an EnvironmentError.
Traceback (most recent call last):
  File "C:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.c43c44ebfe9b\python\install\lib\site-packages\pip-20.0.2-py3.7.egg\pip\_internal\commands\install.py", line 331, in run
    resolver.resolve(requirement_set)
  File "C:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.c43c44ebfe9b\python\install\lib\site-packages\pip-20.0.2-py3.7.egg\pip\_internal\legacy_resolve.py", line 177, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "C:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.c43c44ebfe9b\python\install\lib\site-packages\pip-20.0.2-py3.7.egg\pip\_internal\legacy_resolve.py", line 333, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "C:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.c43c44ebfe9b\python\install\lib\site-packages\pip-20.0.2-py3.7.egg\pip\_internal\legacy_resolve.py", line 282, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "C:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.c43c44ebfe9b\python\install\lib\site-packages\pip-20.0.2-py3.7.egg\pip\_internal\operations\prepare.py", line 482, in prepare_linked_requirement
    hashes=hashes,
  File "C:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.c43c44ebfe9b\python\install\lib\site-packages\pip-20.0.2-py3.7.egg\pip\_internal\operations\prepare.py", line 278, in unpack_url
    return unpack_file_url(link, location, download_dir, hashes=hashes)
  File "C:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.c43c44ebfe9b\python\install\lib\site-packages\pip-20.0.2-py3.7.egg\pip\_internal\operations\prepare.py", line 224, in unpack_file_url
    _copy_source_tree(link_path, location)
  File "C:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.c43c44ebfe9b\python\install\lib\site-packages\pip-20.0.2-py3.7.egg\pip\_internal\operations\prepare.py", line 207, in _copy_source_tree
    shutil.copytree(source, target, **kwargs)
  File "C:\Users\Matt\hg\build\pyoxidizer\python_distributions\python.c43c44ebfe9b\python\install\lib\shutil.py", line 368, in copytree
    raise Error(errors)
shutil.Error: [('C:\\Users\\Matt\\hg\\.hg\\wcache\\checklink', 'C:\\Users\\Matt\\AppData\\Local\\Temp\\pip-req-build-2l899b53\\.hg\\wcache\\checklink', "[Errno 22] Invalid argument: 'C:\\\\Users\\\\Matt\\\\hg\\\\.hg\\\\wcache\\\\checklink'"), ('C:\\Users\\Matt\\hg\\build\\pyoxidizer\\python_distributions\\python.86a3260edabeed314c6f32a931e60dd097fa854b1346561443353e1bc90e3edd\\python\\install\\Lib\\site-packages\\pip-19.3.1-py3.7.egg\\pip\\_vendor\\urllib3\\contrib\\_securetransport', 'C:\\Users\\Matt\\AppData\\Local\\Temp\\pip-req-build-2l899b53\\build\\pyoxidizer\\python_distributions\\python.86a3260edabeed314c6f32a931e60dd097fa854b1346561443353e1bc90e3edd\\python\\install\\Lib\\site-packages\\pip-19.3.1-py3.7.egg\\pip\\_vendor\\urllib3\\contrib\\_securetransport', "[WinError 3] The system cannot find the path specified: 'C:\\\\Users\\\\Matt\\\\AppData\\\\Local\\\\Temp\\\\pip-req-build-2l899b53\\\\build\\\\pyoxidizer\\\\python_distributions\\\\python.86a3260edabeed314c6f32a931e60dd097fa854b1346561443353e1bc90e3edd\\\\python\\\\install\\\\Lib\\\\site-packages\\\\pip-19.3.1-py3.7.egg\\\\pip\\\\_vendor\\\\urllib3\\\\contrib\\\\_securetransport'"), ('C:\\Users\\Matt\\hg\\build\\pyoxidizer\\python_distributions\\python.86a3260edabeed314c6f32a931e60dd097fa854b1346561443353e1bc90e3edd\\python\\install\\Lib\\site-packages\\pip-19.3.1-py3.7.egg\\pip\\_vendor\\urllib3\\packages', 'C:\\Users\\Matt\\AppData\\Local\\Temp\\pip-req-build-2l899b53\\build\\pyoxidizer\\python_distributions\\python.86a3260edabeed314c6f32a931e60dd097fa854b1346561443353e1bc90e3edd\\python\\install\\Lib\\site-packages\\pip-19.3.1-py3.7.egg\\pip\\_vendor\\urllib3\\packages', "[WinError 206] The filename or extension is too long: 'C:\\\\Users\\\\Matt\\\\AppData\\\\Local\\\\Temp\\\\pip-req-build-2l899b53\\\\build\\\\pyoxidizer\\\\python_distributions\\\\python.86a3260edabeed314c6f32a931e60dd097fa854b1346561443353e1bc90e3edd\\\\python\\\\install\\\\Lib\\\\site-packages\\\\pip-19.3.1-py3.7.egg\\\\pip\\\\_vendor\\\\urllib3\\\\packages'")]
Cleaning up...
Removed build tracker: 'C:\\Users\\Matt\\AppData\\Local\\Temp\\pip-req-tracker-2zs2cynh'
error[PIP_INSTALL_ERROR]: error running pip install: error running pip
  --> C:\Users\Matt\hg\rust\hgcli\pyoxidizer.bzl:39:21
   |
39 |     for resource in dist.pip_install(["--verbose", ROOT]):
   |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pip_install()


error: error running pip install: error running pip
contrib/packaging/hgpackaging/cli.py
133

Does this have to be run from a specific Visual Studio command prompt, such that this can be inferred from the environment?

This revision was not accepted when it landed; it landed in state Needs Review.
This revision was automatically updated to reflect the committed changes.