( )⚙ D10688 packaging: use PyOxidizer for producing WiX MSI installer

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10688

packaging: use PyOxidizer for producing WiX MSI installer
ClosedPublic

Authored by indygreg on May 6 2021, 8:00 PM.

Details

Reviewers
mharbison72
Group Reviewers
hg-reviewers
Commits
rHG73f1a10320d1: packaging: use PyOxidizer for producing WiX MSI installer
Summary

We recently taught our in-tree PyOxidizer configuration file to
produce MSI installers with WiX using PyOxidizer's built-in
support for doing so.

This commit changes our WiX + PyOxidizer installer generation
code to use this functionality.

After this change, all the Python packaging code is doing is the
following:

  • Building HTML documentation
  • Making gettext available to the build process.
  • Munging CLI arguments to variables for the pyoxidizer execution.
  • Invoking pyoxidizer build.
  • Copying the produced .msi to the dist/ directory.

Applying this stack on stable and rebuilding the 5.8 MSI installer
produced the following differences from the official 5.8 installer:

  • .exe and .pyd files aren't byte identical (this is expected).
  • Various .dist-info/ directories have different names due to older versions of PyOxidizer being buggy and not properly normalizing package names. (The new behavior is correct.)
  • Various *.dist-info/RECORD files are different due to content divergence of files (this is expected).
  • The python38.dll differs due to newer PyOxidizer shipping a newer version of Python 3.8.
  • We now ship python3.dll because PyOxidizer now includes this file by default.
  • The vcruntime140.dll differs because newer PyOxidizer installs a newer version. We also now ship a vcruntime140_1.dll because newer versions of the redistributable ship 2 files now.

The WiX GUIDs and IDs of installed files have likely changed as a
result of PyOxidizer's different mechanism for generating those
identifiers. This means that an upgrade install of the MSI will
replace files instead of doing an incremental update. This is
likely harmless and we've incurred this kind of breakage before.

As far as I can tell, the new PyOxidizer-built MSI is functionally
equivalent to the old method. Once we drop support for Python 2.7
MSI installers, we can delete the WiX code from the repository.

This commit temporarily drops support for extra .wxs files. We
raise an exception instead of silently not using them, which I think
is appropriate. We should be able to add support back in by injecting
state into pyoxidizer.bzl via --var. I just didn't want to expend
cognitive load to think about the solution as part of this series.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

indygreg created this revision.May 6 2021, 8:00 PM
baymax updated this revision to Diff 27759.May 7 2021, 11:30 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

baymax updated this revision to Diff 27796.May 10 2021, 11:38 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

mharbison72 accepted this revision.May 11 2021, 12:15 AM
This revision is now accepted and ready to land.May 11 2021, 12:15 AM
martinvonz added a subscriber: martinvonz.Jun 17 2021, 7:19 PM

This commit temporarily drops support for extra .wxs files. We
raise an exception instead of silently not using them, which I think
is appropriate. We should be able to add support back in by injecting
state into pyoxidizer.bzl via --var.

Any updates on this? Our internal packaging for Windows depends on the "extra .wxs" feature so it's currently broken. Maybe PyOxidizer has already gained that --var feature?

Revision Contents
Changeset List

Diff 27759

contrib/packaging/hgpackaging/pyoxidizer.py

# pyoxidizer.py - Packaging support for PyOxidizer # pyoxidizer.py - Packaging support for PyOxidizer
# #
# Copyright 2020 Gregory Szorc <gregory.szorc@gmail.com> # Copyright 2020 Gregory Szorc <gregory.szorc@gmail.com>
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
# no-check-code because Python 3 native. # no-check-code because Python 3 native.
import os import os
import pathlib import pathlib
import shutil import shutil
import subprocess import subprocess
import sys import sys
import typing
from .downloads import download_entry from .downloads import download_entry
from .util import ( from .util import (
extract_zip_to_directory, extract_zip_to_directory,
process_install_rules, process_install_rules,
find_vc_runtime_dll, find_vc_runtime_dll,
) )
subprocess.run( subprocess.run(
[sys.executable, str(source_dir / "setup.py"), "build_doc", "--html"], [sys.executable, str(source_dir / "setup.py"), "build_doc", "--html"],
cwd=str(source_dir), cwd=str(source_dir),
check=True, check=True,
) )
def run_pyoxidizer( def run_pyoxidizer(
source_dir: pathlib.Path, build_dir: pathlib.Path, target_triple: str, source_dir: pathlib.Path,
build_dir: pathlib.Path,
target_triple: str,
build_vars: typing.Optional[typing.Dict[str, str]] = None,
target: typing.Optional[str] = None,
) -> pathlib.Path: ) -> pathlib.Path:
"""Run `pyoxidizer` in an environment with access to build dependencies. """Run `pyoxidizer` in an environment with access to build dependencies.
Returns the output directory that pyoxidizer would have used for build Returns the output directory that pyoxidizer would have used for build
artifacts. Actual build artifacts are likely in a sub-directory with the artifacts. Actual build artifacts are likely in a sub-directory with the
name of the pyoxidizer build target that was built. name of the pyoxidizer build target that was built.
""" """
build_vars = build_vars or {}
# We need to make gettext binaries available for compiling i18n files. # We need to make gettext binaries available for compiling i18n files.
gettext_pkg, gettext_entry = download_entry('gettext', build_dir) gettext_pkg, gettext_entry = download_entry('gettext', build_dir)
gettext_dep_pkg = download_entry('gettext-dep', build_dir)[0] gettext_dep_pkg = download_entry('gettext-dep', build_dir)[0]
gettext_root = build_dir / ('gettext-win-%s' % gettext_entry['version']) gettext_root = build_dir / ('gettext-win-%s' % gettext_entry['version'])
if not gettext_root.exists(): if not gettext_root.exists():
extract_zip_to_directory(gettext_pkg, gettext_root) extract_zip_to_directory(gettext_pkg, gettext_root)
"build", "build",
"--path", "--path",
str(source_dir / "rust" / "hgcli"), str(source_dir / "rust" / "hgcli"),
"--release", "--release",
"--target-triple", "--target-triple",
target_triple, target_triple,
] ]
for k, v in sorted(build_vars.items()):
args.extend(["--var", k, v])
if target:
args.append(target)
subprocess.run(args, env=env, check=True) subprocess.run(args, env=env, check=True)
return source_dir / "build" / "pyoxidizer" / target_triple / "release" return source_dir / "build" / "pyoxidizer" / target_triple / "release"
def create_pyoxidizer_install_layout( def create_pyoxidizer_install_layout(
source_dir: pathlib.Path, source_dir: pathlib.Path,
build_dir: pathlib.Path, build_dir: pathlib.Path,

contrib/packaging/hgpackaging/wix.py

import uuid import uuid
import xml.dom.minidom import xml.dom.minidom
from .downloads import download_entry from .downloads import download_entry
from .py2exe import ( from .py2exe import (
build_py2exe, build_py2exe,
stage_install, stage_install,
) )
from .pyoxidizer import create_pyoxidizer_install_layout from .pyoxidizer import (
build_docs_html,
create_pyoxidizer_install_layout,
run_pyoxidizer,
)
from .util import ( from .util import (
extract_zip_to_directory, extract_zip_to_directory,
normalize_windows_version, normalize_windows_version,
process_install_rules, process_install_rules,
sign_with_signtool, sign_with_signtool,
) )
version=None, version=None,
extra_wxs: typing.Optional[typing.Dict[str, str]] = None, extra_wxs: typing.Optional[typing.Dict[str, str]] = None,
extra_features: typing.Optional[typing.List[str]] = None, extra_features: typing.Optional[typing.List[str]] = None,
signing_info: typing.Optional[typing.Dict[str, str]] = None, signing_info: typing.Optional[typing.Dict[str, str]] = None,
): ):
"""Build a WiX MSI installer using PyOxidizer.""" """Build a WiX MSI installer using PyOxidizer."""
hg_build_dir = source_dir / "build" hg_build_dir = source_dir / "build"
build_dir = hg_build_dir / ("wix-%s" % target_triple) build_dir = hg_build_dir / ("wix-%s" % target_triple)
staging_dir = build_dir / "stage"
arch = "x64" if "x86_64" in target_triple else "x86"
build_dir.mkdir(parents=True, exist_ok=True) build_dir.mkdir(parents=True, exist_ok=True)
create_pyoxidizer_install_layout(
source_dir, build_dir, staging_dir, target_triple
)
# We also install some extra files. # Need to ensure docs HTML is built because this isn't done as part of
process_install_rules(EXTRA_INSTALL_RULES, source_dir, staging_dir) # `pip install Mercurial`.
build_docs_html(source_dir)
# And remove some files we don't want. build_vars = {}
for f in STAGING_REMOVE_FILES:
p = staging_dir / f
if p.exists():
print('removing %s' % p)
p.unlink()
return run_wix_packaging( if msi_name:
build_vars["MSI_NAME"] = msi_name
if version:
build_vars["VERSION"] = version
if extra_features:
build_vars["EXTRA_MSI_FEATURES"] = ";".join(extra_features)
if signing_info:
if signing_info["cert_path"]:
build_vars["SIGNING_PFX_PATH"] = signing_info["cert_path"]
if signing_info["cert_password"]:
build_vars["SIGNING_PFX_PASSWORD"] = signing_info["cert_password"]
if signing_info["subject_name"]:
build_vars["SIGNING_SUBJECT_NAME"] = signing_info["subject_name"]
if signing_info["timestamp_url"]:
build_vars["TIME_STAMP_SERVER_URL"] = signing_info["timestamp_url"]
if extra_wxs:
raise Exception(
"support for extra .wxs files has been temporarily dropped"
)
out_dir = run_pyoxidizer(
source_dir, source_dir,
build_dir, build_dir,
staging_dir, target_triple,
arch, build_vars=build_vars,
version, target="msi",
python2=False,
msi_name=msi_name,
extra_wxs=extra_wxs,
extra_features=extra_features,
signing_info=signing_info,
) )
msi_dir = out_dir / "msi"
msi_files = [f for f in os.listdir(msi_dir) if f.endswith(".msi")]
if len(msi_files) != 1:
raise Exception("expected exactly 1 .msi file; got %d" % len(msi_files))
msi_filename = msi_files[0]
msi_path = msi_dir / msi_filename
dist_path = source_dir / "dist" / msi_filename
dist_path.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(msi_path, dist_path)
return {
"msi_path": dist_path,
}
def run_wix_packaging( def run_wix_packaging(
source_dir: pathlib.Path, source_dir: pathlib.Path,
build_dir: pathlib.Path, build_dir: pathlib.Path,
staging_dir: pathlib.Path, staging_dir: pathlib.Path,
arch: str, arch: str,
version: str, version: str,
python2: bool, python2: bool,