This is an archive of the discontinued Mercurial Phabricator instance.

Differential D6164

wix: add a hook for a prebuild script to inject extra libraries
ClosedPublic

Authored by durin42 on Mar 21 2019, 8:55 AM.

Details

Reviewers
indygreg
Group Reviewers
hg-reviewers
Commits
rHG715d3220ac4f: wix: add a hook for a prebuild script to inject extra libraries
Summary

I need this to build packages for Google so we can bundle some
extensions in the installed image. My assumption is that this is most
interesting for the .msi images so I only wired it up there. I'm not
thrilled with the interface this provides, but it was an easy way to
retain debug messages on Windows while also having enough structure to
know what lines are actually module names for py2exe.

Still pending on my end: I need to bundle a couple of config files,
and at least one data file. I'm open to advice on how to do those
things, and how to do this better.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

durin42 created this revision.Mar 21 2019, 8:55 AM
indygreg added a comment.Mar 23 2019, 11:04 AM

I like the flexibility. But I'm not super keen about the interface here. Using a script to inject custom options seems like it could be useful. But as it is currently implemented, the script simply prints out \0 delimited package names. So one UI wart is --extra-prebuild-script being a somewhat generic name but that script only emits package names. --extra-packages-script would be a better name.

What do you think about defining an --extra-package CLI argument that takes nargs=* or an --extra-packages that takes a comma-delimited list and then thread that through to the existing extra_packages keyword argument? That seems a bit simpler and easier to extend than scripts.

durin42 added a comment.Mar 25 2019, 12:09 PM
In D6164#89881, @indygreg wrote:

I like the flexibility. But I'm not super keen about the interface here. Using a script to inject custom options seems like it could be useful. But as it is currently implemented, the script simply prints out \0 delimited package names. So one UI wart is --extra-prebuild-script being a somewhat generic name but that script only emits package names. --extra-packages-script would be a better name.

The script also has to install packages into the virtualenv so they're findable.

What do you think about defining an --extra-package CLI argument that takes nargs=* or an --extra-packages that takes a comma-delimited list and then thread that through to the existing extra_packages keyword argument? That seems a bit simpler and easier to extend than scripts.

I'm open to having both an --extra-package *and* a script argument, if that's what you're proposing. Does that sound workable?

indygreg added a comment.Mar 25 2019, 12:25 PM
In D6164#89913, @durin42 wrote:

How about we add an argument to define the path(s) to the pip requirements file(s) to use? By default, it can use the requirements.txt in the repo. Would that solve your use case?

durin42 added a comment.Mar 25 2019, 12:37 PM
In D6164#89914, @indygreg wrote:
In D6164#89913, @durin42 wrote:

How about we add an argument to define the path(s) to the pip requirements file(s) to use? By default, it can use the requirements.txt in the repo. Would that solve your use case?

No, because I have some stuff which (out of tragic necessity at the moment) isn't pip installable, and I have to install it into the virtualenv by banging some rocks together. :(

indygreg added a comment.Mar 25 2019, 1:14 PM
In D6164#89915, @durin42 wrote:
In D6164#89914, @indygreg wrote:
In D6164#89913, @durin42 wrote:

How about we add an argument to define the path(s) to the pip requirements file(s) to use? By default, it can use the requirements.txt in the repo. Would that solve your use case?

No, because I have some stuff which (out of tragic necessity at the moment) isn't pip installable, and I have to install it into the virtualenv by banging some rocks together. :(

Bleh. So you do need the power of a full script here.

I'll queue with the naming change to "virtualenv populate script" or something along those lines.

durin42 added a comment.Mar 25 2019, 2:58 PM
In D6164#89916, @indygreg wrote:
In D6164#89915, @durin42 wrote:
In D6164#89914, @indygreg wrote:
In D6164#89913, @durin42 wrote:

How about we add an argument to define the path(s) to the pip requirements file(s) to use? By default, it can use the requirements.txt in the repo. Would that solve your use case?

No, because I have some stuff which (out of tragic necessity at the moment) isn't pip installable, and I have to install it into the virtualenv by banging some rocks together. :(

Bleh. So you do need the power of a full script here.

Yep. :(

I'll queue with the naming change to "virtualenv populate script" or something along those lines.

Thanks.

indygreg requested changes to this revision.Apr 2 2019, 1:41 PM

Was a newer version of this patch with the requested changes going to be uploaded? It's weird to see additional patches in the series without this one updated...

This revision now requires changes to proceed.Apr 2 2019, 1:41 PM
durin42 added a comment.Apr 2 2019, 1:59 PM
In D6164#90059, @indygreg wrote:

Was a newer version of this patch with the requested changes going to be uploaded? It's weird to see additional patches in the series without this one updated...

You had indicated up-thread you were going to queue it, possibly with some in-flight tweaks. I had to rebase it as I was working on follow-ups, but I was merely waiting for you to land the patch as you'd indicated.

Are you now saying you want me to fix things for you? That's fine, it's just not what you said.

indygreg added a comment.Apr 2 2019, 2:05 PM

I would prefer you fix them :)

durin42 added a comment.Apr 2 2019, 3:14 PM
In D6164#90073, @indygreg wrote:

I would prefer you fix them :)

Okay, then what should we call it? --extra-packages-script since that's what it's for?

indygreg added a comment.Apr 2 2019, 3:25 PM
In D6164#90074, @durin42 wrote:
In D6164#90073, @indygreg wrote:

I would prefer you fix them :)

Okay, then what should we call it? --extra-packages-script since that's what it's for?

Sounds good!

And maybe change it to emit a newline delimited list of packages? I don't see a benefit to using \0 since we don't need binary safety. IMO \0 just makes it harder to implement said scripts.

durin42 updated this revision to Diff 14628.Apr 2 2019, 3:43 PM
Closed by commit rHG715d3220ac4f: wix: add a hook for a prebuild script to inject extra libraries (authored by durin42). · Explain WhyApr 2 2019, 4:28 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 14576

contrib/packaging/hgpackaging/py2exe.py

python_exe_info, python_exe_info,
) )
def build_py2exe(source_dir: pathlib.Path, build_dir: pathlib.Path, def build_py2exe(source_dir: pathlib.Path, build_dir: pathlib.Path,
python_exe: pathlib.Path, build_name: str, python_exe: pathlib.Path, build_name: str,
venv_requirements_txt: pathlib.Path, venv_requirements_txt: pathlib.Path,
extra_packages=None, extra_excludes=None, extra_packages=None, extra_excludes=None,
extra_dll_excludes=None): extra_dll_excludes=None,
extra_prebuild_script=None):
"""Build Mercurial with py2exe. """Build Mercurial with py2exe.
Build files will be placed in ``build_dir``. Build files will be placed in ``build_dir``.
py2exe's setup.py doesn't use setuptools. It doesn't have modern logic py2exe's setup.py doesn't use setuptools. It doesn't have modern logic
for finding the Python 2.7 toolchain. So, we require the environment for finding the Python 2.7 toolchain. So, we require the environment
to already be configured with an active toolchain. to already be configured with an active toolchain.
""" """
check=True) check=True)
# Force distutils to use VC++ settings from environment, which was # Force distutils to use VC++ settings from environment, which was
# validated above. # validated above.
env = dict(os.environ) env = dict(os.environ)
env['DISTUTILS_USE_SDK'] = '1' env['DISTUTILS_USE_SDK'] = '1'
env['MSSdk'] = '1' env['MSSdk'] = '1'
if extra_prebuild_script:
more_packages = set(subprocess.check_output(
extra_prebuild_script,
cwd=build_dir).split(b'\0')[-1].strip().decode('utf-8').splitlines())
if more_packages:
if not extra_packages:
extra_packages = more_packages
else:
extra_packages |= more_packages
if extra_packages: if extra_packages:
env['HG_PY2EXE_EXTRA_PACKAGES'] = ' '.join(sorted(extra_packages)) env['HG_PY2EXE_EXTRA_PACKAGES'] = ' '.join(sorted(extra_packages))
if extra_excludes: if extra_excludes:
env['HG_PY2EXE_EXTRA_EXCLUDES'] = ' '.join(sorted(extra_excludes)) env['HG_PY2EXE_EXTRA_EXCLUDES'] = ' '.join(sorted(extra_excludes))
if extra_dll_excludes: if extra_dll_excludes:
env['HG_PY2EXE_EXTRA_DLL_EXCLUDES'] = ' '.join( env['HG_PY2EXE_EXTRA_DLL_EXCLUDES'] = ' '.join(
sorted(extra_dll_excludes)) sorted(extra_dll_excludes))

contrib/packaging/hgpackaging/wix.py

f = doc.createElement('File') f = doc.createElement('File')
f.setAttribute('Name', p.name) f.setAttribute('Name', p.name)
component.appendChild(f) component.appendChild(f)
return doc.toprettyxml() return doc.toprettyxml()
def build_installer(source_dir: pathlib.Path, python_exe: pathlib.Path, def build_installer(source_dir: pathlib.Path, python_exe: pathlib.Path,
msi_name='mercurial', version=None, post_build_fn=None): msi_name='mercurial', version=None, post_build_fn=None,
extra_prebuild_script=None):
"""Build a WiX MSI installer. """Build a WiX MSI installer.
``source_dir`` is the path to the Mercurial source tree to use. ``source_dir`` is the path to the Mercurial source tree to use.
``arch`` is the target architecture. either ``x86`` or ``x64``. ``arch`` is the target architecture. either ``x86`` or ``x64``.
``python_exe`` is the path to the Python executable to use/bundle. ``python_exe`` is the path to the Python executable to use/bundle.
``version`` is the Mercurial version string. If not defined, ``version`` is the Mercurial version string. If not defined,
``mercurial/__version__.py`` will be consulted. ``mercurial/__version__.py`` will be consulted.
``post_build_fn`` is a callable that will be called after building ``post_build_fn`` is a callable that will be called after building
Mercurial but before invoking WiX. It can be used to e.g. facilitate Mercurial but before invoking WiX. It can be used to e.g. facilitate
signing. It is passed the paths to the Mercurial source, build, and signing. It is passed the paths to the Mercurial source, build, and
dist directories and the resolved Mercurial version. dist directories and the resolved Mercurial version.
""" """
arch = 'x64' if r'\x64' in os.environ.get('LIB', '') else 'x86' arch = 'x64' if r'\x64' in os.environ.get('LIB', '') else 'x86'
hg_build_dir = source_dir / 'build' hg_build_dir = source_dir / 'build'
dist_dir = source_dir / 'dist' dist_dir = source_dir / 'dist'
wix_dir = source_dir / 'contrib' / 'packaging' / 'wix' wix_dir = source_dir / 'contrib' / 'packaging' / 'wix'
requirements_txt = wix_dir / 'requirements.txt' requirements_txt = wix_dir / 'requirements.txt'
build_py2exe(source_dir, hg_build_dir, build_py2exe(source_dir, hg_build_dir,
python_exe, 'wix', requirements_txt, python_exe, 'wix', requirements_txt,
extra_packages=EXTRA_PACKAGES) extra_packages=EXTRA_PACKAGES,
extra_prebuild_script=extra_prebuild_script)
version = version or normalize_version(find_version(source_dir)) version = version or normalize_version(find_version(source_dir))
print('using version string: %s' % version) print('using version string: %s' % version)
if post_build_fn: if post_build_fn:
post_build_fn(source_dir, hg_build_dir, dist_dir, version) post_build_fn(source_dir, hg_build_dir, dist_dir, version)
build_dir = hg_build_dir / ('wix-%s' % arch) build_dir = hg_build_dir / ('wix-%s' % arch)
return { return {
'msi_path': msi_path, 'msi_path': msi_path,
} }
def build_signed_installer(source_dir: pathlib.Path, python_exe: pathlib.Path, def build_signed_installer(source_dir: pathlib.Path, python_exe: pathlib.Path,
name: str, version=None, subject_name=None, name: str, version=None, subject_name=None,
cert_path=None, cert_password=None, cert_path=None, cert_password=None,
timestamp_url=None): timestamp_url=None, extra_prebuild_script=None):
"""Build an installer with signed executables.""" """Build an installer with signed executables."""
post_build_fn = make_post_build_signing_fn( post_build_fn = make_post_build_signing_fn(
name, name,
subject_name=subject_name, subject_name=subject_name,
cert_path=cert_path, cert_path=cert_path,
cert_password=cert_password, cert_password=cert_password,
timestamp_url=timestamp_url) timestamp_url=timestamp_url)
info = build_installer(source_dir, python_exe=python_exe, info = build_installer(source_dir, python_exe=python_exe,
msi_name=name.lower(), version=version, msi_name=name.lower(), version=version,
post_build_fn=post_build_fn) post_build_fn=post_build_fn,
extra_prebuild_script=extra_prebuild_script)
description = '%s %s' % (name, version) description = '%s %s' % (name, version)
sign_with_signtool(info['msi_path'], description, sign_with_signtool(info['msi_path'], description,
subject_name=subject_name, cert_path=cert_path, subject_name=subject_name, cert_path=cert_path,
cert_password=cert_password, timestamp_url=timestamp_url) cert_password=cert_password, timestamp_url=timestamp_url)

contrib/packaging/wix/build.py

parser.add_argument('--sign-cert', parser.add_argument('--sign-cert',
help='Path to certificate to use for signing') help='Path to certificate to use for signing')
parser.add_argument('--sign-password', parser.add_argument('--sign-password',
help='Password for signing certificate') help='Password for signing certificate')
parser.add_argument('--sign-timestamp-url', parser.add_argument('--sign-timestamp-url',
help='URL of timestamp server to use for signing') help='URL of timestamp server to use for signing')
parser.add_argument('--version', parser.add_argument('--version',
help='Version string to use') help='Version string to use')
parser.add_argument('--extra-prebuild-script',
help='Extra script to run after creating virtualenv.')
args = parser.parse_args() args = parser.parse_args()
here = pathlib.Path(os.path.abspath(os.path.dirname(__file__))) here = pathlib.Path(os.path.abspath(os.path.dirname(__file__)))
source_dir = here.parent.parent.parent source_dir = here.parent.parent.parent
sys.path.insert(0, str(source_dir / 'contrib' / 'packaging')) sys.path.insert(0, str(source_dir / 'contrib' / 'packaging'))
from hgpackaging.wix import ( from hgpackaging.wix import (
build_installer, build_installer,
build_signed_installer, build_signed_installer,
) )
fn = build_installer fn = build_installer
kwargs = { kwargs = {
'source_dir': source_dir, 'source_dir': source_dir,
'python_exe': pathlib.Path(args.python), 'python_exe': pathlib.Path(args.python),
'version': args.version, 'version': args.version,
} }
if args.extra_prebuild_script:
kwargs['extra_prebuild_script'] = args.extra_prebuild_script
if args.sign_sn or args.sign_cert: if args.sign_sn or args.sign_cert:
fn = build_signed_installer fn = build_signed_installer
kwargs['name'] = args.name kwargs['name'] = args.name
kwargs['subject_name'] = args.sign_sn kwargs['subject_name'] = args.sign_sn
kwargs['cert_path'] = args.sign_cert kwargs['cert_path'] = args.sign_cert
kwargs['cert_password'] = args.sign_password kwargs['cert_password'] = args.sign_password
kwargs['timestamp_url'] = args.sign_timestamp_url kwargs['timestamp_url'] = args.sign_timestamp_url
fn(**kwargs) fn(**kwargs)