This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10683

pyoxidizer: support producing MSI installers
ClosedPublic

Authored by indygreg on May 6 2021, 8:00 PM.

Details

Reviewers
mharbison72
Group Reviewers
hg-reviewers
Commits
rHG603efb3845ba: pyoxidizer: support producing MSI installers
Summary

Newer versions of PyOxidizer have support for building WiX MSI
installers "natively." Essentially, you can script the definition
of your WiX installer via Starlark and PyOxidizer can invoke
WiX tools to produce the installer.

This commit teaches our PyOxidizer config file to produce MSI
installers similarly to how
contrib/packaging/packging.py wix would do it.

We had to make a very minor change to mercurial.wxs to reflect
different paths depending on who builds. This is because when
PyOxidizer builds WiX installers, it does so from an isolated
directory, not Mercurial's source directory. We simply copy the
files into the build environment so they are accessible.

After this change, running pyoxidizer build msi produces a nearly
identical install layout to what the previous method produces.
When I applied this series on top of the 5.8 tag, here is the
list of differences and explanations:

  • docs/*.html files are missing from the new installer because the Python build environment doesn't have docutils.
  • .pyd and .exe files differ, likely because I'm using a different Visual Studio toolchain on my local computer than the official build environment.
  • Various .dist-info/ directories have different names. This is because older versions of PyOxidizer had buggy behavior and weren't properly normalizing package names in .dist-info/ directories. e.g. we went from cached-property-1.5.2.dist-info to cached_property-1.5.2.dist-info.
  • Translations (.mo files) may be missing if gettext isn't in %Path%. This is because the packaging.py code installs gettext and ensures it can be found.
  • Some *.dist-info/RECORD files vary due to SHA-256 content digest divergence due to build environment differences. (This should be harmless.)
  • The new install layout ships a python3.dll because newer versions of PyOxidizer ship this file.
  • The new install layout has a different vcruntime140.dll and also a vcruntime140_1.dll because newer versions of PyOxidizer ship a newer version of the Visual C++ Redistributable Runtime.

The new PyOxidizer functionality is not yet integrated with
packaging.py. This will come in a subsequent commit. So for now, the
new functionality introduced here is unused.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.May 6 2021, 8:00 PM
indygreg added a subscriber: mharbison72.May 6 2021, 9:42 PM

@mharbison72 You may be interested in this series starting at approximately this patch. Version 0.15 of PyOxidizer supports building MSIs "natively." I implemented enough features to port Mercurial's MSI generation to it. I need to have a go at porting TortoiseHg to use PyOxidizer. But I'm optimistic it is doable. Note that you don't need to use the "build a Python binary" pieces of PyOxidizer with MSI generation: they are completely separate.

mharbison72 added a comment.May 7 2021, 12:16 AM

I know very little about how to do WiX properly, but we had some issues with the redistributable runtime in TortoiseHg, and the file overwrite mode. The existing Mercurial installer shouldn't have the redistributable runtime issue (since it uses VS 2008), but does have the same overwrite mode. I'll leave them here in case they're worth considering.

https://foss.heptapod.net/mercurial/tortoisehg/thg/-/issues/5662
https://foss.heptapod.net/mercurial/tortoisehg/thg/-/issues/5630

baymax updated this revision to Diff 27754.May 7 2021, 11:30 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

baymax updated this revision to Diff 27791.May 10 2021, 11:37 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

mharbison72 accepted this revision.May 10 2021, 11:43 PM
This revision is now accepted and ready to land.May 10 2021, 11:43 PM
indygreg added a commit: rHG603efb3845ba: pyoxidizer: support producing MSI installers.May 15 2021, 7:24 PM
Closed by commit rHG603efb3845ba: pyoxidizer: support producing MSI installers (authored by indygreg). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 27942

contrib/packaging/wix/mercurial.wxs

<Feature Id='Misc' Title='Miscellaneous' Description='Contributed scripts' Level='1'> <Feature Id='Misc' Title='Miscellaneous' Description='Contributed scripts' Level='1'>
<ComponentGroupRef Id="hg.group.contrib" /> <ComponentGroupRef Id="hg.group.contrib" />
</Feature> </Feature>
</Feature> </Feature>
<UIRef Id="WixUI_FeatureTree" /> <UIRef Id="WixUI_FeatureTree" />
<UIRef Id="WixUI_ErrorProgressText" /> <UIRef Id="WixUI_ErrorProgressText" />
<?ifdef PyOxidizer?>
<WixVariable Id="WixUILicenseRtf" Value="COPYING.rtf" />
<Icon Id="hgIcon.ico" SourceFile="mercurial.ico" />
<?else?>
<WixVariable Id="WixUILicenseRtf" Value="contrib\packaging\wix\COPYING.rtf" /> <WixVariable Id="WixUILicenseRtf" Value="contrib\packaging\wix\COPYING.rtf" />
<Icon Id="hgIcon.ico" SourceFile="contrib/win32/mercurial.ico" /> <Icon Id="hgIcon.ico" SourceFile="contrib/win32/mercurial.ico" />
<?endif?>
<Upgrade Id='$(var.ProductUpgradeCode)'> <Upgrade Id='$(var.ProductUpgradeCode)'>
<UpgradeVersion <UpgradeVersion
IncludeMinimum='yes' Minimum='0.0.0' IncludeMaximum='no' OnlyDetect='no' IncludeMinimum='yes' Minimum='0.0.0' IncludeMaximum='no' OnlyDetect='no'
Property='INSTALLEDMERCURIALPRODUCTS' /> Property='INSTALLEDMERCURIALPRODUCTS' />
</Upgrade> </Upgrade>
<InstallExecuteSequence> <InstallExecuteSequence>
<RemoveExistingProducts After='InstallInitialize'/> <RemoveExistingProducts After='InstallInitialize'/>
</InstallExecuteSequence> </InstallExecuteSequence>
</Product> </Product>
</Wix> </Wix>

rust/hgcli/pyoxidizer.bzl

# The following variables can be passed in as parameters:
#
# VERSION
# Version string of program being produced.
#
# MSI_NAME
# Root name of MSI installer.
#
# EXTRA_MSI_FEATURES
# ; delimited string of extra features to advertise in the built MSA.
ROOT = CWD + "/../.." ROOT = CWD + "/../.."
VERSION = VARS.get("VERSION", "5.8")
MSI_NAME = VARS.get("MSI_NAME", "mercurial")
EXTRA_MSI_FEATURES = VARS.get("EXTRA_MSI_FEATURES")
IS_WINDOWS = "windows" in BUILD_TARGET_TRIPLE IS_WINDOWS = "windows" in BUILD_TARGET_TRIPLE
# Code to run in Python interpreter. # Code to run in Python interpreter.
RUN_CODE = "import hgdemandimport; hgdemandimport.enable(); from mercurial import dispatch; dispatch.run()" RUN_CODE = "import hgdemandimport; hgdemandimport.enable(); from mercurial import dispatch; dispatch.run()"
set_build_path(ROOT + "/build/pyoxidizer") set_build_path(ROOT + "/build/pyoxidizer")
def make_distribution(): def make_distribution():
return exe return exe
def make_manifest(dist, exe): def make_manifest(dist, exe):
m = FileManifest() m = FileManifest()
m.add_python_resource(".", exe) m.add_python_resource(".", exe)
return m return m
# This adjusts the InstallManifest produced from exe generation to provide
# additional files found in a Windows install layout.
def make_windows_install_layout(manifest):
# Copy various files to new install locations. This can go away once
# we're using the importlib resource reader.
RECURSIVE_COPIES = {
"lib/mercurial/locale/": "locale/",
"lib/mercurial/templates/": "templates/",
}
for (search, replace) in RECURSIVE_COPIES.items():
for path in manifest.paths():
if path.startswith(search):
new_path = path.replace(search, replace)
print("copy %s to %s" % (path, new_path))
file = manifest.get_file(path)
manifest.add_file(file, path = new_path)
# Similar to above, but with filename pattern matching.
# lib/mercurial/helptext/**/*.txt -> helptext/
# lib/mercurial/defaultrc/*.rc -> defaultrc/
for path in manifest.paths():
if path.startswith("lib/mercurial/helptext/") and path.endswith(".txt"):
new_path = path[len("lib/mercurial/"):]
elif path.startswith("lib/mercurial/defaultrc/") and path.endswith(".rc"):
new_path = path[len("lib/mercurial/"):]
else:
continue
print("copying %s to %s" % (path, new_path))
manifest.add_file(manifest.get_file(path), path = new_path)
# We also install a handful of additional files.
EXTRA_CONTRIB_FILES = [
"bash_completion",
"hgweb.fcgi",
"hgweb.wsgi",
"logo-droplets.svg",
"mercurial.el",
"mq.el",
"tcsh_completion",
"tcsh_completion_build.sh",
"xml.rnc",
"zsh_completion",
]
for f in EXTRA_CONTRIB_FILES:
manifest.add_file(FileContent(path = ROOT + "/contrib/" + f), directory = "contrib")
# Individual files with full source to destination path mapping.
EXTRA_FILES = {
"contrib/hgk": "contrib/hgk.tcl",
"contrib/win32/postinstall.txt": "ReleaseNotes.txt",
"contrib/win32/ReadMe.html": "ReadMe.html",
"doc/style.css": "doc/style.css",
"COPYING": "Copying.txt",
}
for source, dest in EXTRA_FILES.items():
print("adding extra file %s" % dest)
manifest.add_file(FileContent(path = ROOT + "/" + source), path = dest)
# And finally some wildcard matches.
manifest.add_manifest(glob(
include = [ROOT + "/contrib/vim/*"],
strip_prefix = ROOT + "/"
))
manifest.add_manifest(glob(
include = [ROOT + "/doc/*.html"],
strip_prefix = ROOT + "/"
))
# But we don't ship hg-ssh on Windows, so exclude its documentation.
manifest.remove("doc/hg-ssh.8.html")
return manifest
def make_msi(manifest):
manifest = make_windows_install_layout(manifest)
if "x86_64" in BUILD_TARGET_TRIPLE:
platform = "x64"
else:
platform = "x86"
manifest.add_file(
FileContent(path = ROOT + "/contrib/packaging/wix/COPYING.rtf"),
path = "COPYING.rtf",
)
manifest.remove("Copying.txt")
manifest.add_file(
FileContent(path = ROOT + "/contrib/win32/mercurial.ini"),
path = "defaultrc/mercurial.rc",
)
manifest.add_file(
FileContent(filename = "editor.rc", content = "[ui]\neditor = notepad\n"),
path = "defaultrc/editor.rc",
)
wix = WiXInstaller("hg", "%s-%s.msi" % (MSI_NAME, VERSION))
# Materialize files in the manifest to the install layout.
wix.add_install_files(manifest)
# From mercurial.wxs.
wix.install_files_root_directory_id = "INSTALLDIR"
# Pull in our custom .wxs files.
defines = {
"PyOxidizer": "1",
"Platform": platform,
"Version": VERSION,
"Comments": "Installs Mercurial version %s" % VERSION,
"PythonVersion": "3",
"MercurialHasLib": "1",
}
if EXTRA_MSI_FEATURES:
defines["MercurialExtraFeatures"] = EXTRA_MSI_FEATURES
wix.add_wxs_file(
ROOT + "/contrib/packaging/wix/mercurial.wxs",
preprocessor_parameters=defines,
)
# Our .wxs references to other files. Pull those into the build environment.
for f in ("defines.wxi", "guids.wxi", "COPYING.rtf"):
wix.add_build_file(f, ROOT + "/contrib/packaging/wix/" + f)
wix.add_build_file("mercurial.ico", ROOT + "/contrib/win32/mercurial.ico")
return wix
register_target("distribution", make_distribution) register_target("distribution", make_distribution)
register_target("exe", make_exe, depends = ["distribution"]) register_target("exe", make_exe, depends = ["distribution"])
register_target("app", make_manifest, depends = ["distribution", "exe"], default = True) register_target("app", make_manifest, depends = ["distribution", "exe"], default = True)
register_target("msi", make_msi, depends = ["app"])
resolve_targets() resolve_targets()