( )⚙ D8473 packaging: support building Inno installer with PyOxidizer

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8473

packaging: support building Inno installer with PyOxidizer
ClosedPublic

Authored by indygreg on Apr 21 2020, 10:39 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG94f4f2ec7dee: packaging: support building Inno installer with PyOxidizer
rHG9965d6c32a93: packaging: support building Inno installer with PyOxidizer
Summary

We want to start distributing Mercurial on Python 3 on
Windows. PyOxidizer will be our vehicle for achieving that.

This commit implements basic support for producing Inno
installers using PyOxidizer.

While it is an eventual goal of PyOxidizer to produce
installers, those features aren't yet implemented. So our
strategy for producing Mercurial installers is similar to
what we've been doing with py2exe: invoke a build system to
produce files then stage those files into a directory so they
can be turned into an installer.

We had to make significant alterations to the pyoxidizer.bzl
config file to get it to produce the files that we desire for
a Windows install. This meant differentiating the build targets
so we can target Windows specifically.

We've added a new module to hgpackaging to deal with interacting
with PyOxidizer. It is similar to pyexe: we invoke a build process
then copy files to a staging directory. Ideally these extra
files would be defined in pyoxidizer.bzl. But I don't think it
is worth doing at this time, as PyOxidizer's config files are
lacking some features to make this turnkey.

The rest of the change is introducing a variant of the
Inno installer code that invokes PyOxidizer instead of
py2exe.

Comparing the Python 2.7 based Inno installers with this
one, the following changes were observed:

  • No lib/*.{pyd, dll} files
  • No Microsoft.VC90.CRT.manifest
  • No msvc{m,p,r}90.dll files
  • python27.dll replaced with python37.dll
  • Add vcruntime140.dll file

The disappearance of the .pyd and .dll files is acceptable, as
PyOxidizer has embedded these in hg.exe and loads them from
memory.

The disappearance of the *90* files is acceptable because those
provide the Visual C++ 9 runtime, as required by Python 2.7.
Similarly, the appearance of vcruntime140.dll is a requirement
of Python 3.7.

Diff Detail

Repository
rHG Mercurial
Branch
stable
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

indygreg created this revision.Apr 21 2020, 10:39 PM
indygreg planned changes to this revision.Apr 23 2020, 9:09 PM

This patch is buggy. I'll upload a new version hopefully in the next few hours.

indygreg updated this revision to Diff 21207.Apr 23 2020, 11:28 PM
indygreg updated this revision to Diff 21218.Apr 24 2020, 5:29 PM
indygreg updated this revision to Diff 21219.Apr 24 2020, 6:20 PM
indygreg added a commit: rHG9965d6c32a93: packaging: support building Inno installer with PyOxidizer.Apr 30 2020, 1:51 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG9965d6c32a93: packaging: support building Inno installer with PyOxidizer (authored by indygreg). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 21207

contrib/packaging/hgpackaging/cli.py

inno, inno,
wix, wix,
) )
HERE = pathlib.Path(os.path.abspath(os.path.dirname(__file__))) HERE = pathlib.Path(os.path.abspath(os.path.dirname(__file__)))
SOURCE_DIR = HERE.parent.parent.parent SOURCE_DIR = HERE.parent.parent.parent
def build_inno(python=None, iscc=None, version=None): def build_inno(pyoxidizer_target=None, python=None, iscc=None, version=None):
if not os.path.isabs(python): if not pyoxidizer_target and not python:
raise Exception("--python required unless building with PyOxidizer")
if python and not os.path.isabs(python):
raise Exception("--python arg must be an absolute path") raise Exception("--python arg must be an absolute path")
if iscc: if iscc:
iscc = pathlib.Path(iscc) iscc = pathlib.Path(iscc)
else: else:
iscc = ( iscc = (
pathlib.Path(os.environ["ProgramFiles(x86)"]) pathlib.Path(os.environ["ProgramFiles(x86)"])
/ "Inno Setup 5" / "Inno Setup 5"
/ "ISCC.exe" / "ISCC.exe"
) )
build_dir = SOURCE_DIR / "build" build_dir = SOURCE_DIR / "build"
inno.build( if pyoxidizer_target:
inno.build_with_pyoxidizer(
SOURCE_DIR, build_dir, pyoxidizer_target, iscc, version=version
)
else:
inno.build_with_py2exe(
SOURCE_DIR, build_dir, pathlib.Path(python), iscc, version=version, SOURCE_DIR, build_dir, pathlib.Path(python), iscc, version=version,
) )
def build_wix( def build_wix(
name=None, name=None,
python=None, python=None,
version=None, version=None,
sign_sn=None, sign_sn=None,
sign_cert=None, sign_cert=None,
def get_parser(): def get_parser():
parser = argparse.ArgumentParser() parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers() subparsers = parser.add_subparsers()
sp = subparsers.add_parser("inno", help="Build Inno Setup installer") sp = subparsers.add_parser("inno", help="Build Inno Setup installer")
sp.add_argument("--python", required=True, help="path to python.exe to use") sp.add_argument(
"--pyoxidizer-target",
choices={"i686-pc-windows-msvc", "x86_64-pc-windows-msvc"},
help="Build with PyOxidizer targeting this host triple",
)
sp.add_argument("--python", help="path to python.exe to use")
sp.add_argument("--iscc", help="path to iscc.exe to use") sp.add_argument("--iscc", help="path to iscc.exe to use")
sp.add_argument( sp.add_argument(
"--version", "--version",
help="Mercurial version string to use " help="Mercurial version string to use "
"(detected from __version__.py if not defined", "(detected from __version__.py if not defined",
) )
sp.set_defaults(func=build_inno) sp.set_defaults(func=build_inno)

contrib/packaging/hgpackaging/inno.py

import subprocess import subprocess
import jinja2 import jinja2
from .py2exe import ( from .py2exe import (
build_py2exe, build_py2exe,
stage_install, stage_install,
) )
from .pyoxidizer import run_pyoxidizer
from .util import ( from .util import (
find_vc_runtime_files, find_legacy_vc_runtime_files,
normalize_windows_version, normalize_windows_version,
process_install_rules, process_install_rules,
read_version_py, read_version_py,
) )
EXTRA_PACKAGES = { EXTRA_PACKAGES = {
'dulwich', 'dulwich',
'keyring', 'keyring',
'pygments', 'pygments',
'win32ctypes', 'win32ctypes',
} }
EXTRA_INSTALL_RULES = [ EXTRA_INSTALL_RULES = [
('contrib/win32/mercurial.ini', 'defaultrc/mercurial.rc'), ('contrib/win32/mercurial.ini', 'defaultrc/mercurial.rc'),
] ]
PACKAGE_FILES_METADATA = { PACKAGE_FILES_METADATA = {
'ReadMe.html': 'Flags: isreadme', 'ReadMe.html': 'Flags: isreadme',
} }
def build( def build_with_py2exe(
source_dir: pathlib.Path, source_dir: pathlib.Path,
build_dir: pathlib.Path, build_dir: pathlib.Path,
python_exe: pathlib.Path, python_exe: pathlib.Path,
iscc_exe: pathlib.Path, iscc_exe: pathlib.Path,
version=None, version=None,
): ):
"""Build the Inno installer. """Build the Inno installer using py2exe.
Build files will be placed in ``build_dir``. Build files will be placed in ``build_dir``.
py2exe's setup.py doesn't use setuptools. It doesn't have modern logic py2exe's setup.py doesn't use setuptools. It doesn't have modern logic
for finding the Python 2.7 toolchain. So, we require the environment for finding the Python 2.7 toolchain. So, we require the environment
to already be configured with an active toolchain. to already be configured with an active toolchain.
""" """
if not iscc_exe.exists(): if not iscc_exe.exists():
# Now assemble all the packaged files into the staging directory. # Now assemble all the packaged files into the staging directory.
stage_install(source_dir, staging_dir) stage_install(source_dir, staging_dir)
# We also install some extra files. # We also install some extra files.
process_install_rules(EXTRA_INSTALL_RULES, source_dir, staging_dir) process_install_rules(EXTRA_INSTALL_RULES, source_dir, staging_dir)
# hg.exe depends on VC9 runtime DLLs. Copy those into place. # hg.exe depends on VC9 runtime DLLs. Copy those into place.
for f in find_vc_runtime_files(vc_x64): for f in find_legacy_vc_runtime_files(vc_x64):
if f.name.endswith('.manifest'): if f.name.endswith('.manifest'):
basename = 'Microsoft.VC90.CRT.manifest' basename = 'Microsoft.VC90.CRT.manifest'
else: else:
basename = f.name basename = f.name
dest_path = staging_dir / basename dest_path = staging_dir / basename
print('copying %s to %s' % (f, dest_path)) print('copying %s to %s' % (f, dest_path))
shutil.copyfile(f, dest_path) shutil.copyfile(f, dest_path)
build_installer( build_installer(
source_dir, source_dir,
inno_build_dir, inno_build_dir,
staging_dir, staging_dir,
iscc_exe, iscc_exe,
version, version,
arch="x64" if vc_x64 else None, arch="x64" if vc_x64 else None,
) )
def build_with_pyoxidizer(
source_dir: pathlib.Path,
build_dir: pathlib.Path,
target_triple: str,
iscc_exe: pathlib.Path,
version=None,
):
"""Build the Inno installer using PyOxidizer."""
if not iscc_exe.exists():
raise Exception("%s does not exist" % iscc_exe)
inno_build_dir = build_dir / ("inno-pyoxidizer-%s" % target_triple)
staging_dir = inno_build_dir / "stage"
inno_build_dir.mkdir(parents=True, exist_ok=True)
run_pyoxidizer(source_dir, staging_dir, target_triple)
process_install_rules(EXTRA_INSTALL_RULES, source_dir, staging_dir)
build_installer(
source_dir,
inno_build_dir,
staging_dir,
iscc_exe,
version,
arch="x64" if "x86_64" in target_triple else None,
)
def build_installer( def build_installer(
source_dir: pathlib.Path, source_dir: pathlib.Path,
inno_build_dir: pathlib.Path, inno_build_dir: pathlib.Path,
staging_dir: pathlib.Path, staging_dir: pathlib.Path,
iscc_exe: pathlib.Path, iscc_exe: pathlib.Path,
version, version,
arch=None, arch=None,
): ):

contrib/packaging/hgpackaging/pyoxidizer.py

  • This file was added.
# pyoxidizer.py - Packaging support for PyOxidizer
#
# Copyright 2020 Gregory Szorc <gregory.szorc@gmail.com>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
# no-check-code because Python 3 native.
import pathlib
import shutil
import subprocess
import sys
from .util import (
process_install_rules,
find_vc_runtime_dll,
)
STAGING_RULES_WINDOWS = [
('contrib/bash_completion', 'contrib/'),
('contrib/hgk', 'contrib/hgk.tcl'),
('contrib/hgweb.fcgi', 'contrib/'),
('contrib/hgweb.wsgi', 'contrib/'),
('contrib/logo-droplets.svg', 'contrib/'),
('contrib/mercurial.el', 'contrib/'),
('contrib/mq.el', 'contrib/'),
('contrib/tcsh_completion', 'contrib/'),
('contrib/tcsh_completion_build.sh', 'contrib/'),
('contrib/vim/*', 'contrib/vim/'),
('contrib/win32/postinstall.txt', 'ReleaseNotes.txt'),
('contrib/win32/ReadMe.html', 'ReadMe.html'),
('contrib/xml.rnc', 'contrib/'),
('contrib/zsh_completion', 'contrib/'),
('doc/*.html', 'doc/'),
('doc/style.css', 'doc/'),
('COPYING', 'Copying.txt'),
]
STAGING_RULES_APP = [
('mercurial/helptext/**/*.txt', 'helptext/'),
('mercurial/defaultrc/*.rc', 'defaultrc/'),
('mercurial/locale/**/*', 'locale/'),
('mercurial/templates/**/*', 'templates/'),
]
STAGING_EXCLUDES_WINDOWS = [
"doc/hg-ssh.8.html",
]
def run_pyoxidizer(
source_dir: pathlib.Path, out_dir: pathlib.Path, target_triple: str
):
"""Build Mercurial with PyOxidizer and copy additional files into place.
After successful completion, ``out_dir`` contains files constituting a
Mercurial install.
"""
args = [
"pyoxidizer",
"build",
"--path",
str(source_dir / "rust" / "hgcli"),
"--release",
"--target-triple",
target_triple,
]
subprocess.run(args, check=True)
if "windows" in target_triple:
target = "app_windows"
else:
target = "app_posix"
build_dir = (
source_dir / "build" / "pyoxidizer" / target_triple / "release" / target
)
if out_dir.exists():
print("purging %s" % out_dir)
shutil.rmtree(out_dir)
# Now assemble all the files from PyOxidizer into the staging directory.
shutil.copytree(build_dir, out_dir)
# Move some of those files around.
process_install_rules(STAGING_RULES_APP, build_dir, out_dir)
# Nuke the mercurial/* directory, as we copied resources
# to an appropriate location just above.
shutil.rmtree(out_dir / "mercurial")
# We also need to run setup.py build_doc to produce html files,
# as they aren't built as part of ``pip install``.
# This will fail if docutils isn't installed.
subprocess.run(
[sys.executable, str(source_dir / "setup.py"), "build_doc", "--html"],
cwd=str(source_dir),
check=True,
)
if "windows" in target_triple:
process_install_rules(STAGING_RULES_WINDOWS, source_dir, out_dir)
# Write out a default editor.rc file to configure notepad as the
# default editor.
with (out_dir / "defaultrc" / "editor.rc").open(
"w", encoding="utf-8"
) as fh:
fh.write("[ui]\neditor = notepad\n")
for f in STAGING_EXCLUDES_WINDOWS:
p = out_dir / f
if p.exists():
print("removing %s" % p)
p.unlink()
# Add vcruntimeXXX.dll next to executable.
vc_runtime_dll = find_vc_runtime_dll(x64="x86_64" in target_triple)
shutil.copy(vc_runtime_dll, out_dir / vc_runtime_dll.name)

contrib/packaging/hgpackaging/util.py

tf.extractall(dest) tf.extractall(dest)
def extract_zip_to_directory(source: pathlib.Path, dest: pathlib.Path): def extract_zip_to_directory(source: pathlib.Path, dest: pathlib.Path):
with zipfile.ZipFile(source, 'r') as zf: with zipfile.ZipFile(source, 'r') as zf:
zf.extractall(dest) zf.extractall(dest)
def find_vc_runtime_files(x64=False): def find_vc_runtime_dll(x64=False):
"""Finds Visual C++ Runtime DLL to include in distribution."""
# We invoke vswhere to find the latest Visual Studio install.
vswhere = (
pathlib.Path(os.environ["ProgramFiles(x86)"])
/ "Microsoft Visual Studio"
/ "Installer"
/ "vswhere.exe"
)
if not vswhere.exists():
raise Exception(
"could not find vswhere.exe: %s does not exist" % vswhere
)
args = [
str(vswhere),
"-requires",
"Microsoft.VisualCpp.Redist.14.Latest",
"-latest",
"-property",
"installationPath",
]
vs_install_path = pathlib.Path(
os.fsdecode(subprocess.check_output(args).strip())
)
# This just gets us a path like
# C:\Program Files (x86)\Microsoft Visual Studio\2019\Community
# Actually vcruntime140.dll is under a path like:
# VC\Redist\MSVC\<version>\<arch>\Microsoft.VC14<X>.CRT\vcruntime140.dll.
arch = "x64" if x64 else "x86"
search_glob = (
r"%s\VC\Redist\MSVC\*\%s\Microsoft.VC14*.CRT\vcruntime140.dll"
% (vs_install_path, arch)
)
candidates = glob.glob(search_glob, recursive=True)
for candidate in reversed(candidates):
return pathlib.Path(candidate)
raise Exception("could not find vcruntime140.dll")
def find_legacy_vc_runtime_files(x64=False):
"""Finds Visual C++ Runtime DLLs to include in distribution.""" """Finds Visual C++ Runtime DLLs to include in distribution."""
winsxs = pathlib.Path(os.environ['SYSTEMROOT']) / 'WinSxS' winsxs = pathlib.Path(os.environ['SYSTEMROOT']) / 'WinSxS'
prefix = 'amd64' if x64 else 'x86' prefix = 'amd64' if x64 else 'x86'
candidates = sorted( candidates = sorted(
p p
for p in os.listdir(winsxs) for p in os.listdir(winsxs)

rust/hgcli/pyoxidizer.bzl

ROOT = CWD + "/../.." ROOT = CWD + "/../.."
def make_exe(): # Code to run in Python interpreter.
dist = default_python_distribution() RUN_CODE = "import hgdemandimport; hgdemandimport.enable(); from mercurial import dispatch; dispatch.run()"
code = "import hgdemandimport; hgdemandimport.enable(); from mercurial import dispatch; dispatch.run()"
set_build_path(ROOT + "/build/pyoxidizer")
def make_distribution():
return default_python_distribution()
def make_distribution_windows():
return default_python_distribution(flavor="standalone_dynamic")
def make_exe(dist):
config = PythonInterpreterConfig( config = PythonInterpreterConfig(
raw_allocator = "system", raw_allocator = "system",
run_eval = code, run_eval = RUN_CODE,
# We want to let the user load extensions from the file system # We want to let the user load extensions from the file system
filesystem_importer = True, filesystem_importer = True,
# We need this to make resourceutil happy, since it looks for sys.frozen. # We need this to make resourceutil happy, since it looks for sys.frozen.
sys_frozen = True, sys_frozen = True,
legacy_windows_stdio = True, legacy_windows_stdio = True,
) )
exe = dist.to_python_executable( exe = dist.to_python_executable(
name = "hg", name = "hg",
resources_policy = "prefer-in-memory-fallback-filesystem-relative:lib", resources_policy = "prefer-in-memory-fallback-filesystem-relative:lib",
config = config, config = config,
# Extension may depend on any Python functionality. Include all # Extension may depend on any Python functionality. Include all
# extensions. # extensions.
extension_module_filter = "all", extension_module_filter = "all",
) )
exe.add_python_resources(dist.pip_install([ROOT])) # Add Mercurial to resources.
for resource in dist.pip_install(["--verbose", ROOT]):
# This is a bit wonky and worth explaining.
#
# Various parts of Mercurial don't yet support loading package
# resources via the ResourceReader interface. Or, not having
# file-based resources would be too inconvenient for users.
#
# So, for package resources, we package them both in the
# filesystem as well as in memory. If both are defined,
# PyOxidizer will prefer the in-memory location. So even
# if the filesystem file isn't packaged in the location
# specified here, we should never encounter an errors as the
# resource will always be available in memory.
if type(resource) == "PythonPackageResource":
exe.add_filesystem_relative_python_resource(".", resource)
exe.add_in_memory_python_resource(resource)
else:
exe.add_python_resource(resource)
# On Windows, we install extra packages for convenience.
if "windows" in BUILD_TARGET_TRIPLE:
exe.add_python_resources(
dist.pip_install(["-r", ROOT + "/contrib/packaging/requirements_win32.txt"])
)
return exe return exe
def make_install(exe):
m = FileManifest()
# `hg` goes in root directory. def make_manifest(dist, exe):
m = FileManifest()
m.add_python_resource(".", exe) m.add_python_resource(".", exe)
templates = glob(
include = [ROOT + "/mercurial/templates/**/*"],
strip_prefix = ROOT + "/mercurial/",
)
m.add_manifest(templates)
return m return m
def make_embedded_resources(exe): def make_embedded_resources(exe):
return exe.to_embedded_resources() return exe.to_embedded_resources()
register_target("exe", make_exe)
register_target("app", make_install, depends = ["exe"], default = True) register_target("distribution_posix", make_distribution)
register_target("embedded", make_embedded_resources, depends = ["exe"], default_build_script = True) register_target("distribution_windows", make_distribution_windows)
register_target("exe_posix", make_exe, depends = ["distribution_posix"])
register_target("exe_windows", make_exe, depends = ["distribution_windows"])
register_target(
"app_posix",
make_manifest,
depends = ["distribution_posix", "exe_posix"],
default = "windows" not in BUILD_TARGET_TRIPLE,
)
register_target(
"app_windows",
make_manifest,
depends = ["distribution_windows", "exe_windows"],
default = "windows" in BUILD_TARGET_TRIPLE,
)
resolve_targets() resolve_targets()
# END OF COMMON USER-ADJUSTED SETTINGS. # END OF COMMON USER-ADJUSTED SETTINGS.
# #
# Everything below this is typically managed by PyOxidizer and doesn't need # Everything below this is typically managed by PyOxidizer and doesn't need
# to be updated by people. # to be updated by people.
PYOXIDIZER_VERSION = "0.7.0-pre" PYOXIDIZER_VERSION = "0.7.0"
PYOXIDIZER_COMMIT = "c772a1379c3026314eda1c8ea244b86c0658951d"

tests/test-check-code.t

Skipping contrib/automation/hgautomation/windows.py it has no-che?k-code (glob) Skipping contrib/automation/hgautomation/windows.py it has no-che?k-code (glob)
Skipping contrib/automation/hgautomation/winrm.py it has no-che?k-code (glob) Skipping contrib/automation/hgautomation/winrm.py it has no-che?k-code (glob)
Skipping contrib/fuzz/FuzzedDataProvider.h it has no-che?k-code (glob) Skipping contrib/fuzz/FuzzedDataProvider.h it has no-che?k-code (glob)
Skipping contrib/fuzz/standalone_fuzz_target_runner.cc it has no-che?k-code (glob) Skipping contrib/fuzz/standalone_fuzz_target_runner.cc it has no-che?k-code (glob)
Skipping contrib/packaging/hgpackaging/cli.py it has no-che?k-code (glob) Skipping contrib/packaging/hgpackaging/cli.py it has no-che?k-code (glob)
Skipping contrib/packaging/hgpackaging/downloads.py it has no-che?k-code (glob) Skipping contrib/packaging/hgpackaging/downloads.py it has no-che?k-code (glob)
Skipping contrib/packaging/hgpackaging/inno.py it has no-che?k-code (glob) Skipping contrib/packaging/hgpackaging/inno.py it has no-che?k-code (glob)
Skipping contrib/packaging/hgpackaging/py2exe.py it has no-che?k-code (glob) Skipping contrib/packaging/hgpackaging/py2exe.py it has no-che?k-code (glob)
Skipping contrib/packaging/hgpackaging/pyoxidizer.py it has no-che?k-code (glob)
Skipping contrib/packaging/hgpackaging/util.py it has no-che?k-code (glob) Skipping contrib/packaging/hgpackaging/util.py it has no-che?k-code (glob)
Skipping contrib/packaging/hgpackaging/wix.py it has no-che?k-code (glob) Skipping contrib/packaging/hgpackaging/wix.py it has no-che?k-code (glob)
Skipping i18n/polib.py it has no-che?k-code (glob) Skipping i18n/polib.py it has no-che?k-code (glob)
Skipping mercurial/statprof.py it has no-che?k-code (glob) Skipping mercurial/statprof.py it has no-che?k-code (glob)
Skipping tests/badserverext.py it has no-che?k-code (glob) Skipping tests/badserverext.py it has no-che?k-code (glob)
@commands in debugcommands.py should be in alphabetical order. @commands in debugcommands.py should be in alphabetical order.