This is an archive of the discontinued Mercurial Phabricator instance.

Differential D7159

packaging: stage installed files for Inno
ClosedPublic

Authored by indygreg on Oct 23 2019, 9:54 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHGd053d3f10b6a: packaging: stage installed files for Inno
rHGd2ea31a2546d: packaging: stage installed files for Inno
Summary

Previously, the Inno installer maintained its own mapping of
source files to install location. (We have to maintain a
similar mapping in the WiX installer.)

Managing the explicit file layout for Windows packages is
cumbersome and redundant. Every time you want to change the
layout you need to change N locations. We frequently forget
to do this and we only find out when people install Mercurial
from our packages at release time.

This commit starts the process of consolidating and simplifying
the logic for managing the install layout on Windows.

We introduce a list of install layout rules. These are simply
source filenames (which can contain wildcards) and destination
paths.

The Inno packaging code has been updated to assemble all
files into a staging directory that mirrors the final install
layout. The list of files to add to the installer is derived
by walking this staging directory and dynamically emitting
the proper entries for the Inno Setup script.

I diffed the file layout before and after this commit and
there is no difference.

Another benefit of this change is that it facilitates easier
testing of the Windows install layout. Before, in order to
test the final install layout, you needed to build an installer
and run it. Now, you can stage files into the final layout
and test from there, without running the installer. This
should cut down on overhead when changing Windows code.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Oct 23 2019, 9:54 PM
indygreg added a commit: rHGd2ea31a2546d: packaging: stage installed files for Inno.Nov 8 2019, 7:04 PM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHGd2ea31a2546d: packaging: stage installed files for Inno (authored by indygreg). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 17811

contrib/packaging/hgpackaging/inno.py

# inno.py - Inno Setup functionality. # inno.py - Inno Setup functionality.
# #
# Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com> # Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
# no-check-code because Python 3 native. # no-check-code because Python 3 native.
import os import os
import pathlib import pathlib
import shutil import shutil
import subprocess import subprocess
import jinja2 import jinja2
from .py2exe import build_py2exe from .py2exe import (
build_py2exe,
stage_install,
)
from .util import find_vc_runtime_files from .util import find_vc_runtime_files
EXTRA_PACKAGES = { EXTRA_PACKAGES = {
'dulwich', 'dulwich',
'keyring', 'keyring',
'pygments', 'pygments',
'win32ctypes', 'win32ctypes',
} }
PACKAGE_FILES_METADATA = {
'ReadMe.html': 'Flags: isreadme',
'hg.exe': "AfterInstall: Touch('{app}\\hg.exe.local')",
}
def build( def build(
source_dir: pathlib.Path, source_dir: pathlib.Path,
build_dir: pathlib.Path, build_dir: pathlib.Path,
python_exe: pathlib.Path, python_exe: pathlib.Path,
iscc_exe: pathlib.Path, iscc_exe: pathlib.Path,
version=None, version=None,
): ):
"""Build the Inno installer. """Build the Inno installer.
Build files will be placed in ``build_dir``. Build files will be placed in ``build_dir``.
py2exe's setup.py doesn't use setuptools. It doesn't have modern logic py2exe's setup.py doesn't use setuptools. It doesn't have modern logic
for finding the Python 2.7 toolchain. So, we require the environment for finding the Python 2.7 toolchain. So, we require the environment
to already be configured with an active toolchain. to already be configured with an active toolchain.
""" """
if not iscc_exe.exists(): if not iscc_exe.exists():
raise Exception('%s does not exist' % iscc_exe) raise Exception('%s does not exist' % iscc_exe)
vc_x64 = r'\x64' in os.environ.get('LIB', '') vc_x64 = r'\x64' in os.environ.get('LIB', '')
arch = 'x64' if vc_x64 else 'x86' arch = 'x64' if vc_x64 else 'x86'
inno_source_dir = source_dir / 'contrib' / 'packaging' / 'inno' inno_source_dir = source_dir / 'contrib' / 'packaging' / 'inno'
inno_build_dir = build_dir / ('inno-%s' % arch) inno_build_dir = build_dir / ('inno-%s' % arch)
staging_dir = inno_build_dir / 'stage'
requirements_txt = ( requirements_txt = (
source_dir / 'contrib' / 'packaging' / 'inno' / 'requirements.txt' source_dir / 'contrib' / 'packaging' / 'inno' / 'requirements.txt'
) )
inno_build_dir.mkdir(parents=True, exist_ok=True) inno_build_dir.mkdir(parents=True, exist_ok=True)
build_py2exe( build_py2exe(
source_dir, source_dir,
build_dir, build_dir,
python_exe, python_exe,
'inno', 'inno',
requirements_txt, requirements_txt,
extra_packages=EXTRA_PACKAGES, extra_packages=EXTRA_PACKAGES,
) )
# Purge the staging directory for every build so packaging is
# pristine.
if staging_dir.exists():
print('purging %s' % staging_dir)
shutil.rmtree(staging_dir)
# Now assemble all the packaged files into the staging directory.
stage_install(source_dir, staging_dir)
# hg.exe depends on VC9 runtime DLLs. Copy those into place. # hg.exe depends on VC9 runtime DLLs. Copy those into place.
for f in find_vc_runtime_files(vc_x64): for f in find_vc_runtime_files(vc_x64):
if f.name.endswith('.manifest'): if f.name.endswith('.manifest'):
basename = 'Microsoft.VC90.CRT.manifest' basename = 'Microsoft.VC90.CRT.manifest'
else: else:
basename = f.name basename = f.name
dest_path = source_dir / 'dist' / basename dest_path = staging_dir / basename
print('copying %s to %s' % (f, dest_path)) print('copying %s to %s' % (f, dest_path))
shutil.copyfile(f, dest_path) shutil.copyfile(f, dest_path)
# The final package layout is simply a mirror of the staging directory.
package_files = []
for root, dirs, files in os.walk(staging_dir):
dirs.sort()
root = pathlib.Path(root)
for f in sorted(files):
full = root / f
rel = full.relative_to(staging_dir)
if str(rel.parent) == '.':
dest_dir = '{app}'
else:
dest_dir = '{app}\\%s' % rel.parent
package_files.append(
{
'source': rel,
'dest_dir': dest_dir,
'metadata': PACKAGE_FILES_METADATA.get(str(rel), None),
}
)
print('creating installer') print('creating installer')
# Install Inno files by rendering a template. # Install Inno files by rendering a template.
jinja_env = jinja2.Environment( jinja_env = jinja2.Environment(
loader=jinja2.FileSystemLoader(str(inno_source_dir)), loader=jinja2.FileSystemLoader(str(inno_source_dir)),
# Need to change these to prevent conflict with Inno Setup. # Need to change these to prevent conflict with Inno Setup.
comment_start_string='{##', comment_start_string='{##',
comment_end_string='##}', comment_end_string='##}',
) )
try: try:
template = jinja_env.get_template('mercurial.iss') template = jinja_env.get_template('mercurial.iss')
except jinja2.TemplateSyntaxError as e: except jinja2.TemplateSyntaxError as e:
raise Exception( raise Exception(
'template syntax error at %s:%d: %s' 'template syntax error at %s:%d: %s'
% (e.name, e.lineno, e.message,) % (e.name, e.lineno, e.message,)
) )
content = template.render() content = template.render(package_files=package_files)
with (inno_build_dir / 'mercurial.iss').open('w', encoding='utf-8') as fh: with (inno_build_dir / 'mercurial.iss').open('w', encoding='utf-8') as fh:
fh.write(content) fh.write(content)
# Copy additional files used by Inno.
for p in ('mercurial.ico', 'postinstall.txt'):
shutil.copyfile(
source_dir / 'contrib' / 'win32' / p, inno_build_dir / p
)
args = [str(iscc_exe)] args = [str(iscc_exe)]
if vc_x64: if vc_x64:
args.append('/dARCH=x64') args.append('/dARCH=x64')
if version: if version:
args.append('/dVERSION=%s' % version) args.append('/dVERSION=%s' % version)
args.append('/Odist') args.append('/Odist')
args.append(str(inno_build_dir / 'mercurial.iss')) args.append(str(inno_build_dir / 'mercurial.iss'))
subprocess.run(args, cwd=str(source_dir), check=True) subprocess.run(args, cwd=str(source_dir), check=True)

contrib/packaging/hgpackaging/py2exe.py

import os import os
import pathlib import pathlib
import subprocess import subprocess
from .downloads import download_entry from .downloads import download_entry
from .util import ( from .util import (
extract_tar_to_directory, extract_tar_to_directory,
extract_zip_to_directory, extract_zip_to_directory,
process_install_rules,
python_exe_info, python_exe_info,
) )
STAGING_RULES = [
('contrib/bash_completion', 'Contrib/'),
('contrib/hgk', 'Contrib/hgk.tcl'),
('contrib/hgweb.fcgi', 'Contrib/'),
('contrib/hgweb.wsgi', 'Contrib/'),
('contrib/mercurial.el', 'Contrib/'),
('contrib/mq.el', 'Contrib/'),
('contrib/tcsh_completion', 'Contrib/'),
('contrib/tcsh_completion_build.sh', 'Contrib/'),
('contrib/vim/*', 'Contrib/Vim/'),
('contrib/win32/postinstall.txt', 'ReleaseNotes.txt'),
('contrib/win32/ReadMe.html', 'ReadMe.html'),
('contrib/xml.rnc', 'Contrib/'),
('contrib/zsh_completion', 'Contrib/'),
('dist/hg.exe', './'),
('dist/lib/*.dll', 'lib/'),
('dist/lib/*.pyd', 'lib/'),
('dist/lib/library.zip', 'lib/'),
('dist/Microsoft.VC*.CRT.manifest', './'),
('dist/msvc*.dll', './'),
('dist/python*.dll', './'),
('doc/*.html', 'Docs/'),
('doc/style.css', 'Docs/'),
('mercurial/help/**/*.txt', 'help/'),
('mercurial/default.d/*.rc', 'default.d/'),
('mercurial/locale/**/*', 'locale/'),
('mercurial/templates/**/*', 'Templates/'),
('CONTRIBUTORS', 'Contributors.txt'),
('COPYING', 'Copying.txt'),
]
def build_py2exe( def build_py2exe(
source_dir: pathlib.Path, source_dir: pathlib.Path,
build_dir: pathlib.Path, build_dir: pathlib.Path,
python_exe: pathlib.Path, python_exe: pathlib.Path,
build_name: str, build_name: str,
venv_requirements_txt: pathlib.Path, venv_requirements_txt: pathlib.Path,
extra_packages=None, extra_packages=None,
extra_excludes=None, extra_excludes=None,
print('building Mercurial') print('building Mercurial')
subprocess.run( subprocess.run(
[str(venv_python), 'setup.py', 'py2exe', 'build_doc', '--html'], [str(venv_python), 'setup.py', 'py2exe', 'build_doc', '--html'],
cwd=str(source_dir), cwd=str(source_dir),
env=env, env=env,
check=True, check=True,
) )
def stage_install(source_dir: pathlib.Path, staging_dir: pathlib.Path):
"""Copy all files to be installed to a directory.
This allows packaging to simply walk a directory tree to find source
files.
"""
process_install_rules(STAGING_RULES, source_dir, staging_dir)

contrib/packaging/hgpackaging/util.py

# util.py - Common packaging utility code. # util.py - Common packaging utility code.
# #
# Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com> # Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
# no-check-code because Python 3 native. # no-check-code because Python 3 native.
import distutils.version import distutils.version
import getpass import getpass
import glob
import os import os
import pathlib import pathlib
import shutil
import subprocess import subprocess
import tarfile import tarfile
import zipfile import zipfile
def extract_tar_to_directory(source: pathlib.Path, dest: pathlib.Path): def extract_tar_to_directory(source: pathlib.Path, dest: pathlib.Path):
with tarfile.open(source, 'r') as tf: with tarfile.open(source, 'r') as tf:
tf.extractall(dest) tf.extractall(dest)
version = distutils.version.LooseVersion(version) version = distutils.version.LooseVersion(version)
return { return {
'arch': arch, 'arch': arch,
'version': version, 'version': version,
'py3': version >= distutils.version.LooseVersion('3'), 'py3': version >= distutils.version.LooseVersion('3'),
} }
def process_install_rules(
rules: list, source_dir: pathlib.Path, dest_dir: pathlib.Path
):
for source, dest in rules:
if '*' in source:
if not dest.endswith('/'):
raise ValueError('destination must end in / when globbing')
# We strip off the source path component before the first glob
# character to construct the relative install path.
prefix_end_index = source[: source.index('*')].rindex('/')
relative_prefix = source_dir / source[0:prefix_end_index]
for res in glob.glob(str(source_dir / source), recursive=True):
source_path = pathlib.Path(res)
if source_path.is_dir():
continue
rel_path = source_path.relative_to(relative_prefix)
dest_path = dest_dir / dest[:-1] / rel_path
dest_path.parent.mkdir(parents=True, exist_ok=True)
print('copying %s to %s' % (source_path, dest_path))
shutil.copy(source_path, dest_path)
# Simple file case.
else:
source_path = pathlib.Path(source)
if dest.endswith('/'):
dest_path = pathlib.Path(dest) / source_path.name
else:
dest_path = pathlib.Path(dest)
full_source_path = source_dir / source_path
full_dest_path = dest_dir / dest_path
full_dest_path.parent.mkdir(parents=True, exist_ok=True)
shutil.copy(full_source_path, full_dest_path)
print('copying %s to %s' % (full_source_path, full_dest_path))

contrib/packaging/inno/mercurial.iss

AppVerName=Mercurial {#VERSION} (64-bit) AppVerName=Mercurial {#VERSION} (64-bit)
OutputBaseFilename=Mercurial-{#VERSION}-x64 OutputBaseFilename=Mercurial-{#VERSION}-x64
ArchitecturesAllowed=x64 ArchitecturesAllowed=x64
ArchitecturesInstallIn64BitMode=x64 ArchitecturesInstallIn64BitMode=x64
#else #else
AppVerName=Mercurial {#VERSION} AppVerName=Mercurial {#VERSION}
OutputBaseFilename=Mercurial-{#VERSION} OutputBaseFilename=Mercurial-{#VERSION}
#endif #endif
InfoAfterFile=contrib/win32/postinstall.txt InfoAfterFile=../postinstall.txt
LicenseFile=COPYING LicenseFile=Copying.txt
ShowLanguageDialog=yes ShowLanguageDialog=yes
AppPublisher=Matt Mackall and others AppPublisher=Matt Mackall and others
AppPublisherURL=https://mercurial-scm.org/ AppPublisherURL=https://mercurial-scm.org/
AppSupportURL=https://mercurial-scm.org/ AppSupportURL=https://mercurial-scm.org/
AppUpdatesURL=https://mercurial-scm.org/ AppUpdatesURL=https://mercurial-scm.org/
{{ 'AppID={{4B95A5F1-EF59-4B08-BED8-C891C46121B3}' }} {{ 'AppID={{4B95A5F1-EF59-4B08-BED8-C891C46121B3}' }}
AppContact=mercurial@mercurial-scm.org AppContact=mercurial@mercurial-scm.org
DefaultDirName={pf}\Mercurial DefaultDirName={pf}\Mercurial
SourceDir=..\.. SourceDir=stage
VersionInfoDescription=Mercurial distributed SCM (version {#VERSION}) VersionInfoDescription=Mercurial distributed SCM (version {#VERSION})
VersionInfoCopyright=Copyright 2005-2019 Matt Mackall and others VersionInfoCopyright=Copyright 2005-2019 Matt Mackall and others
VersionInfoCompany=Matt Mackall and others VersionInfoCompany=Matt Mackall and others
InternalCompressLevel=max InternalCompressLevel=max
SolidCompression=true SolidCompression=true
SetupIconFile=contrib\win32\mercurial.ico SetupIconFile=../mercurial.ico
AllowNoIcons=true AllowNoIcons=true
DefaultGroupName=Mercurial DefaultGroupName=Mercurial
PrivilegesRequired=none PrivilegesRequired=none
ChangesEnvironment=true ChangesEnvironment=true
[Files] [Files]
Source: contrib\mercurial.el; DestDir: {app}/Contrib {% for entry in package_files -%}
Source: contrib\vim\*.*; DestDir: {app}/Contrib/Vim Source: {{ entry.source }}; DestDir: {{ entry.dest_dir }}
Source: contrib\zsh_completion; DestDir: {app}/Contrib {%- if entry.metadata %}; {{ entry.metadata }}{% endif %}
Source: contrib\bash_completion; DestDir: {app}/Contrib {% endfor %}
Source: contrib\tcsh_completion; DestDir: {app}/Contrib
Source: contrib\tcsh_completion_build.sh; DestDir: {app}/Contrib
Source: contrib\hgk; DestDir: {app}/Contrib; DestName: hgk.tcl
Source: contrib\xml.rnc; DestDir: {app}/Contrib
Source: contrib\mercurial.el; DestDir: {app}/Contrib
Source: contrib\mq.el; DestDir: {app}/Contrib
Source: contrib\hgweb.fcgi; DestDir: {app}/Contrib
Source: contrib\hgweb.wsgi; DestDir: {app}/Contrib
Source: contrib\win32\ReadMe.html; DestDir: {app}; Flags: isreadme
Source: contrib\win32\postinstall.txt; DestDir: {app}; DestName: ReleaseNotes.txt
Source: dist\hg.exe; DestDir: {app}; AfterInstall: Touch('{app}\hg.exe.local')
Source: dist\lib\*.dll; Destdir: {app}\lib
Source: dist\lib\*.pyd; Destdir: {app}\lib
Source: dist\python*.dll; Destdir: {app}; Flags: skipifsourcedoesntexist
Source: dist\msvc*.dll; DestDir: {app}; Flags: skipifsourcedoesntexist
Source: dist\Microsoft.VC*.CRT.manifest; DestDir: {app}; Flags: skipifsourcedoesntexist
Source: dist\lib\library.zip; DestDir: {app}\lib
Source: doc\*.html; DestDir: {app}\Docs
Source: doc\style.css; DestDir: {app}\Docs
Source: mercurial\help\*.txt; DestDir: {app}\help
Source: mercurial\help\internals\*.txt; DestDir: {app}\help\internals
Source: mercurial\default.d\*.rc; DestDir: {app}\default.d
Source: mercurial\locale\*.*; DestDir: {app}\locale; Flags: recursesubdirs createallsubdirs skipifsourcedoesntexist
Source: mercurial\templates\*.*; DestDir: {app}\Templates; Flags: recursesubdirs createallsubdirs
Source: CONTRIBUTORS; DestDir: {app}; DestName: Contributors.txt
Source: COPYING; DestDir: {app}; DestName: Copying.txt
[INI] [INI]
Filename: {app}\Mercurial.url; Section: InternetShortcut; Key: URL; String: https://mercurial-scm.org/ Filename: {app}\Mercurial.url; Section: InternetShortcut; Key: URL; String: https://mercurial-scm.org/
Filename: {app}\default.d\editor.rc; Section: ui; Key: editor; String: notepad Filename: {app}\default.d\editor.rc; Section: ui; Key: editor; String: notepad
[UninstallDelete] [UninstallDelete]
Type: files; Name: {app}\Mercurial.url Type: files; Name: {app}\Mercurial.url
Type: filesandordirs; Name: {app}\default.d Type: filesandordirs; Name: {app}\default.d