This is an archive of the discontinued Mercurial Phabricator instance.

Differential D6066

inno: script to automate building Inno installer
ClosedPublic

Authored by indygreg on Mar 3 2019, 6:52 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHGd7dc4ac1ff84: inno: script to automate building Inno installer
Summary

The official Inno installer build process is poorly documented.
And attempting to reproduce behavior of the installer uploaded
to www.mercurial-scm.org has revealed a number of unexpected
behaviors.

This commit attempts to improve the state of reproducibility
of the Inno installer by introducing a Python script to
largely automate the building of the installer.

The new script (which must be run from an environment with the
Visual C++ environment configured) takes care of producing an
Inno installer. When run from a fresh Mercurial source checkout
with all the proper system dependencies (the VC++ toolchain,
Windows 10 SDK, and Inno tools) installed, it "just works."
The script takes care of downloading all the Python
dependencies in a secure manner and manages the build
environment for you. You don't need any additional config
files: just launch the script, pointing it at an existing
Python and ISCC binary and it takes care of the rest.

The produced installer creates a Mercurial installation with
a handful of differences from the existing 4.9 installers
(produced by someone else):

  • add_path.exe is missing (this was removed a few changesets ago)
  • The set of api-ms-win-core-* DLLs is different (I suspect this is due to me using a different UCRT / Windows version).
  • kernelbase.dll and msasn1.dll are missing.
  • There are a different set of .pyc files for dulwich, keyring, and pygments due to us using the latest versions of each.
  • We include Tcl/Tk DLLs and .pyc files (I'm not sure why these are missing from the existing installers).
  • We include the urllib3 and win32ctypes packages (which are dependencies of dulwich and pywin32, respectively). I'm not sure why these aren't present in the existing installers.
  • We include a different set of files for the distutils package. I'm not sure why. But it should be harmless.
  • We include the docutils package (it is getting picked up as a dependency somehow). I think this is fine.
  • We include a copy of argparse.pyc. I'm not sure why this was missing from existing installers.
  • We don't have a copy of sqlite3/dump.pyc. I'm not sure why. The SQLite C extension code only imports this module when conn.iterdump() is called. It should be safe to omit.
  • We include files in the email.test and test packages. The set of files is small and their presence should be harmless.

The new script and support code is written in Python 3 because
it is brand new and independent code and I don't believe new
Python projects should be using Python 2 in 2019 if they have
a choice about it.

The readme.txt file has been renamed to readme.rst and overhauled
to reflect the existence of build.py.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Mar 3 2019, 6:52 PM
indygreg updated this revision to Diff 14338.Mar 3 2019, 9:22 PM
Closed by commit rHGd7dc4ac1ff84: inno: script to automate building Inno installer (authored by indygreg). · Explain WhyMar 5 2019, 10:24 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 14368

contrib/packaging/__init__.py

  • This file was added.

contrib/packaging/inno/build.py

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
#!/usr/bin/env python3
# build.py - Inno installer build script.
#
# Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
# This script automates the building of the Inno MSI installer for Mercurial.
# no-check-code because Python 3 native.
import argparse
import os
import pathlib
import shutil
import subprocess
import sys
import tempfile
DOWNLOADS = {
'gettext': {
'url': 'https://versaweb.dl.sourceforge.net/project/gnuwin32/gettext/0.14.4/gettext-0.14.4-bin.zip',
'size': 1606131,
'sha256': '60b9ef26bc5cceef036f0424e542106cf158352b2677f43a01affd6d82a1d641',
'version': '0.14.4',
},
'gettext-dep': {
'url': 'https://versaweb.dl.sourceforge.net/project/gnuwin32/gettext/0.14.4/gettext-0.14.4-dep.zip',
'size': 715086,
'sha256': '411f94974492fd2ecf52590cb05b1023530aec67e64154a88b1e4ebcd9c28588',
},
'py2exe': {
'url': 'https://versaweb.dl.sourceforge.net/project/py2exe/py2exe/0.6.9/py2exe-0.6.9.zip',
'size': 149687,
'sha256': '6bd383312e7d33eef2e43a5f236f9445e4f3e0f6b16333c6f183ed445c44ddbd',
'version': '0.6.9',
},
'virtualenv': {
'url': 'https://files.pythonhosted.org/packages/37/db/89d6b043b22052109da35416abc3c397655e4bd3cff031446ba02b9654fa/virtualenv-16.4.3.tar.gz',
'size': 3713208,
'sha256': '984d7e607b0a5d1329425dd8845bd971b957424b5ba664729fab51ab8c11bc39',
'version': '16.4.3',
},
}
PRINT_PYTHON_INFO = '''
import platform, sys; print("%s:%d" % (platform.architecture()[0], sys.version_info[0]))
'''.strip()
def find_vc_runtime_files(x64=False):
"""Finds Visual C++ Runtime DLLs to include in distribution."""
winsxs = pathlib.Path(os.environ['SYSTEMROOT']) / 'WinSxS'
prefix = 'amd64' if x64 else 'x86'
candidates = sorted(p for p in os.listdir(winsxs)
if p.lower().startswith('%s_microsoft.vc90.crt_' % prefix))
for p in candidates:
print('found candidate VC runtime: %s' % p)
# Take the newest version.
version = candidates[-1]
d = winsxs / version
return [
d / 'msvcm90.dll',
d / 'msvcp90.dll',
d / 'msvcr90.dll',
winsxs / 'Manifests' / ('%s.manifest' % version),
]
def build(source_dir: pathlib.Path, build_dir: pathlib.Path,
python_exe: pathlib.Path, iscc_exe: pathlib.Path,
version=None):
"""Build the Inno installer.
Build files will be placed in ``build_dir``.
py2exe's setup.py doesn't use setuptools. It doesn't have modern logic
for finding the Python 2.7 toolchain. So, we require the environment
to already be configured with an active toolchain.
"""
from packagingutil import (
download_entry,
extract_tar_to_directory,
extract_zip_to_directory,
)
if not iscc.exists():
raise Exception('%s does not exist' % iscc)
if 'VCINSTALLDIR' not in os.environ:
raise Exception('not running from a Visual C++ build environment; '
'execute the "Visual C++ <version> Command Prompt" '
'application shortcut or a vcsvarsall.bat file')
# Identity x86/x64 and validate the environment matches the Python
# architecture.
vc_x64 = r'\x64' in os.environ['LIB']
res = subprocess.run(
[str(python_exe), '-c', PRINT_PYTHON_INFO],
capture_output=True, check=True)
py_arch, py_version = res.stdout.decode('utf-8').split(':')
py_version = int(py_version)
if vc_x64:
if py_arch != '64bit':
raise Exception('architecture mismatch: Visual C++ environment '
'is configured for 64-bit but Python is 32-bit')
else:
if py_arch != '32bit':
raise Exception('architecture mismatch: Visual C++ environment '
'is configured for 32-bit but Python is 64-bit')
if py_version != 2:
raise Exception('Only Python 2 is currently supported')
# Some extensions may require DLLs from the Universal C Runtime (UCRT).
# These are typically not in PATH and py2exe will have trouble finding
# them. We find the Windows 10 SDK and the UCRT files within.
sdk_path = (pathlib.Path(os.environ['ProgramFiles(x86)']) /
'Windows Kits' / '10' / 'Redist' / 'ucrt' / 'DLLs')
if vc_x64:
sdk_path = sdk_path / 'x64'
else:
sdk_path = sdk_path / 'x86'
if not sdk_path.is_dir():
raise Exception('UCRT files could not be found at %s' % sdk_path)
build_dir.mkdir(exist_ok=True)
gettext_pkg = download_entry(DOWNLOADS['gettext'], build_dir)
gettext_dep_pkg = download_entry(DOWNLOADS['gettext-dep'], build_dir)
virtualenv_pkg = download_entry(DOWNLOADS['virtualenv'], build_dir)
py2exe_pkg = download_entry(DOWNLOADS['py2exe'], build_dir)
venv_path = build_dir / ('venv-inno-%s' % ('x64' if vc_x64 else 'x86'))
gettext_root = build_dir / (
'gettext-win-%s' % DOWNLOADS['gettext']['version'])
if not gettext_root.exists():
extract_zip_to_directory(gettext_pkg, gettext_root)
extract_zip_to_directory(gettext_dep_pkg, gettext_root)
with tempfile.TemporaryDirectory() as td:
td = pathlib.Path(td)
# This assumes Python 2.
extract_tar_to_directory(virtualenv_pkg, td)
extract_zip_to_directory(py2exe_pkg, td)
virtualenv_src_path = td / ('virtualenv-%s' %
DOWNLOADS['virtualenv']['version'])
py2exe_source_path = td / ('py2exe-%s' %
DOWNLOADS['py2exe']['version'])
virtualenv_py = virtualenv_src_path / 'virtualenv.py'
if not venv_path.exists():
print('creating virtualenv with dependencies')
subprocess.run(
[str(python_exe), str(virtualenv_py), str(venv_path)],
check=True)
venv_python = venv_path / 'Scripts' / 'python.exe'
venv_pip = venv_path / 'Scripts' / 'pip.exe'
requirements_txt = (source_dir / 'contrib' / 'packaging' /
'inno' / 'requirements.txt')
subprocess.run([str(venv_pip), 'install', '-r', str(requirements_txt)],
check=True)
# Force distutils to use VC++ settings from environment, which was
# validated above.
env = dict(os.environ)
env['DISTUTILS_USE_SDK'] = '1'
env['MSSdk'] = '1'
py2exe_py_path = venv_path / 'Lib' / 'site-packages' / 'py2exe'
if not py2exe_py_path.exists():
print('building py2exe')
subprocess.run([str(venv_python), 'setup.py', 'install'],
cwd=py2exe_source_path,
env=env,
check=True)
if str(sdk_path) not in os.environ['PATH'].split(os.pathsep):
print('adding %s to PATH' % sdk_path)
env['PATH'] = '%s%s%s' % (
os.environ['PATH'], os.pathsep, str(sdk_path))
# Register location of msgfmt and other binaries.
env['PATH'] = '%s%s%s' % (
env['PATH'], os.pathsep, str(gettext_root / 'bin'))
print('building Mercurial')
subprocess.run(
[str(venv_python), 'setup.py',
'py2exe', '-b', '3' if vc_x64 else '2',
'build_doc', '--html'],
cwd=str(source_dir),
env=env,
check=True)
# hg.exe depends on VC9 runtime DLLs. Copy those into place.
for f in find_vc_runtime_files(vc_x64):
if f.name.endswith('.manifest'):
basename = 'Microsoft.VC90.CRT.manifest'
else:
basename = f.name
dest_path = source_dir / 'dist' / basename
print('copying %s to %s' % (f, dest_path))
shutil.copyfile(f, dest_path)
print('creating installer')
args = [str(iscc_exe)]
if vc_x64:
args.append('/dARCH=x64')
if version:
args.append('/dVERSION=%s' % version)
args.append('/Odist')
args.append('contrib/packaging/inno/mercurial.iss')
subprocess.run(args, cwd=str(source_dir), check=True)
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--python',
required=True,
help='path to python.exe to use')
parser.add_argument('--iscc',
help='path to iscc.exe to use')
parser.add_argument('--version',
help='Mercurial version string to use '
'(detected from __version__.py if not defined')
args = parser.parse_args()
if args.iscc:
iscc = pathlib.Path(args.iscc)
else:
iscc = (pathlib.Path(os.environ['ProgramFiles(x86)']) / 'Inno Setup 5' /
'ISCC.exe')
here = pathlib.Path(os.path.abspath(os.path.dirname(__file__)))
source_dir = here.parent.parent.parent
build_dir = source_dir / 'build'
sys.path.insert(0, str(source_dir / 'contrib' / 'packaging'))
build(source_dir, build_dir, pathlib.Path(args.python), iscc,
version=args.version)

contrib/packaging/inno/readme.rst

  • This file was moved from contrib/packaging/inno/readme.txt.
The standalone Windows installer for Mercurial is built in a somewhat Requirements
jury-rigged fashion. ============
It has the following prerequisites. Ensure to take the packages Building the Inno installer requires a Windows machine.
matching the mercurial version you want to build (32-bit or 64-bit).
Python 2.6 for Windows The following system dependencies must be installed:
http://www.python.org/download/releases/
A compiler: * Python 2.7 (download from https://www.python.org/downloads/)
either MinGW * Microsoft Visual C++ Compiler for Python 2.7
http://www.mingw.org/ (https://www.microsoft.com/en-us/download/details.aspx?id=44266)
or Microsoft Visual C++ 2008 SP1 Express Edition * Windows 10 SDK (download from
http://www.microsoft.com/express/Downloads/Download-2008.aspx https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk
or install via a modern version of Visual Studio)
mfc71.dll (just download, don't install; not needed for Python 2.6) * Inno Setup (http://jrsoftware.org/isdl.php) version 5.4 or newer.
http://starship.python.net/crew/mhammond/win32/ Be sure to install the optional Inno Setup Preprocessor feature,
which is required.
Visual C++ 2008 redistributable package (needed for >= Python 2.6 or if you compile with MSVC) * Python 3.5+ (to run the ``build.py`` script)
for 32-bit:
http://www.microsoft.com/downloads/details.aspx?FamilyID=9b2da534-3e03-4391-8a4d-074b9f2bc1bf Building
for 64-bit: ========
http://www.microsoft.com/downloads/details.aspx?familyid=bd2a6171-e2d6-4230-b809-9a8d7548c1b6
The ``build.py`` script automates the process of producing an
The py2exe distutils extension Inno installer. It manages fetching and configuring the
http://sourceforge.net/projects/py2exe/ non-system dependencies (such as py2exe, gettext, and various
Python packages).
GnuWin32 gettext utility (if you want to build translations)
http://gnuwin32.sourceforge.net/packages/gettext.htm The script requires an activated ``Visual C++ 2008`` command prompt.
A shortcut to such a prompt was installed with ``Microsoft Visual C++
Inno Setup Compiler for Python 2.7``. From your Start Menu, look for
http://www.jrsoftware.org/isdl.php#qsp ``Microsoft Visual C++ Compiler Package for Python 2.7`` then launch
either ``Visual C++ 2008 32-bit Command Prompt`` or
Get and install ispack-5.3.10.exe or later (includes Inno Setup Processor), ``Visual C++ 2008 64-bit Command Prompt``.
which is necessary to package Mercurial.
From the prompt, change to the Mercurial source directory. e.g.
ISTool - optional ``cd c:\src\hg``.
http://www.istool.org/default.aspx/
Next, invoke ``build.py`` to produce an Inno installer. You will
Docutils need to supply the path to the Python interpreter to use.:
http://docutils.sourceforge.net/
$ python3.exe contrib\packaging\inno\build.py \
And, of course, Mercurial itself. --python c:\python27\python.exe
Once you have all this installed and built, clone a copy of the .. note::
Mercurial repository you want to package, and name the repo
C:\hg\hg-release. The script validates that the Visual C++ environment is
active and that the architecture of the specified Python
In a shell, build a standalone copy of the hg.exe program. interpreter matches the Visual C++ environment and errors
if not.
Building instructions for MinGW:
python setup.py build -c mingw32 If everything runs as intended, dependencies will be fetched and
python setup.py py2exe -b 2 configured into the ``build`` sub-directory, Mercurial will be built,
Note: the previously suggested combined command of "python setup.py build -c and an installer placed in the ``dist`` sub-directory. The final
mingw32 py2exe -b 2" doesn't work correctly anymore as it doesn't include the line of output should print the name of the generated installer.
extensions in the mercurial subdirectory.
If you want to create a file named setup.cfg with the contents: Additional options may be configured. Run ``build.py --help`` to
[build] see a list of program flags.
compiler=mingw32
you can skip the first build step. MinGW
=====
Building instructions with MSVC 2008 Express Edition:
for 32-bit: It is theoretically possible to generate an installer that uses
"C:\Program Files\Microsoft Visual Studio 9.0\VC\vcvarsall.bat" x86 MinGW. This isn't well tested and ``build.py`` and may properly
python setup.py py2exe -b 2 support it. See old versions of this file in version control for
for 64-bit: potentially useful hints as to how to achieve this.
"C:\Program Files\Microsoft Visual Studio 9.0\VC\vcvarsall.bat" x86_amd64
python setup.py py2exe -b 3
If you are using Python 2.6 or later, or if you are using MSVC 2008 to compile
mercurial, you must include the C runtime libraries in the installer. To do so,
install the Visual C++ 2008 redistributable package. Then in your windows\winsxs
folder, locate the folder containing the dlls version 9.0.21022.8.
For x86, it should be named like x86_Microsoft.VC90.CRT_(...)_9.0.21022.8(...).
For x64, it should be named like amd64_Microsoft.VC90.CRT_(...)_9.0.21022.8(...).
Copy the files named msvcm90.dll, msvcp90.dll and msvcr90.dll into the dist
directory.
Then in the windows\winsxs\manifests folder, locate the corresponding manifest
file (x86_Microsoft.VC90.CRT_(...)_9.0.21022.8(...).manifest for x86,
amd64_Microsoft.VC90.CRT_(...)_9.0.21022.8(...).manifest for x64), copy it in the
dist directory and rename it to Microsoft.VC90.CRT.manifest.
Before building the installer, you have to build Mercurial HTML documentation
(or fix mercurial.iss to not reference the doc directory):
cd doc
mingw32-make html
cd ..
If you use ISTool, you open the
C:\hg\hg-release\contrib\packaging\inno-installer\mercurial.iss
file and type Ctrl-F9 to compile the installer file.
Otherwise you run the Inno Setup compiler. Assuming it's in the path
you should execute:
iscc contrib\packaging\inno-installer\mercurial.iss /dVERSION=foo
Where 'foo' is the version number you would like to see in the
'Add/Remove Applications' tool. The installer will be placed into
a directory named Output/ at the root of your repository.
If the /dVERSION=foo parameter is not given in the command line, the
installer will retrieve the version information from the __version__.py file.
If you want to build an installer for a 64-bit mercurial, add /dARCH=x64 to
your command line:
iscc contrib\packaging\inno-installer\mercurial.iss /dARCH=x64
To automate the steps above you may want to create a batchfile based on the
following (MinGW build chain):
echo [build] > setup.cfg
echo compiler=mingw32 >> setup.cfg
python setup.py py2exe -b 2
cd doc
mingw32-make html
cd ..
iscc contrib\packaging\inno-installer\mercurial.iss /dVERSION=snapshot
and run it from the root of the hg repository (c:\hg\hg-release).

contrib/packaging/inno/readme.txt

  • This file was moved to contrib/packaging/inno/readme.rst.

contrib/packaging/inno/requirements.txt

  • This file was added.
#
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --generate-hashes contrib/packaging/inno/requirements.txt.in -o contrib/packaging/inno/requirements.txt
#
certifi==2018.11.29 \
--hash=sha256:47f9c83ef4c0c621eaef743f133f09fa8a74a9b75f037e8624f83bd1b6626cb7 \
--hash=sha256:993f830721089fef441cdfeb4b2c8c9df86f0c63239f06bd025a76a7daddb033 \
# via dulwich
configparser==3.7.3 \
--hash=sha256:27594cf4fc279f321974061ac69164aaebd2749af962ac8686b20503ac0bcf2d \
--hash=sha256:9d51fe0a382f05b6b117c5e601fc219fede4a8c71703324af3f7d883aef476a3 \
# via entrypoints
docutils==0.14 \
--hash=sha256:02aec4bd92ab067f6ff27a38a38a41173bf01bed8f89157768c1573f53e474a6 \
--hash=sha256:51e64ef2ebfb29cae1faa133b3710143496eca21c530f3f71424d77687764274 \
--hash=sha256:7a4bd47eaf6596e1295ecb11361139febe29b084a87bf005bf899f9a42edc3c6
dulwich==0.19.11 \
--hash=sha256:afbe070f6899357e33f63f3f3696e601731fef66c64a489dea1bc9f539f4a725
entrypoints==0.3 \
--hash=sha256:589f874b313739ad35be6e0cd7efde2a4e9b6fea91edcc34e58ecbb8dbe56d19 \
--hash=sha256:c70dd71abe5a8c85e55e12c19bd91ccfeec11a6e99044204511f9ed547d48451 \
# via keyring
keyring==18.0.0 \
--hash=sha256:12833d2b05d2055e0e25931184af9cd6a738f320a2264853cabbd8a3a0f0b65d \
--hash=sha256:ca33f5ccc542b9ffaa196ee9a33488069e5e7eac77d5b81969f8a3ce74d0230c
pygments==2.3.1 \
--hash=sha256:5ffada19f6203563680669ee7f53b64dabbeb100eb51b61996085e99c03b284a \
--hash=sha256:e8218dd399a61674745138520d0d4cf2621d7e032439341bc3f647bff125818d
pywin32-ctypes==0.2.0 \
--hash=sha256:24ffc3b341d457d48e8922352130cf2644024a4ff09762a2261fd34c36ee5942 \
--hash=sha256:9dc2d991b3479cc2df15930958b674a48a227d5361d413827a4cfd0b5876fc98 \
# via keyring
pywin32==224 \
--hash=sha256:22e218832a54ed206452c8f3ca9eff07ef327f8e597569a4c2828be5eaa09a77 \
--hash=sha256:32b37abafbfeddb0fe718008d6aada5a71efa2874f068bee1f9e703983dcc49a \
--hash=sha256:35451edb44162d2f603b5b18bd427bc88fcbc74849eaa7a7e7cfe0f507e5c0c8 \
--hash=sha256:4eda2e1e50faa706ff8226195b84fbcbd542b08c842a9b15e303589f85bfb41c \
--hash=sha256:5f265d72588806e134c8e1ede8561739071626ea4cc25c12d526aa7b82416ae5 \
--hash=sha256:6852ceac5fdd7a146b570655c37d9eacd520ed1eaeec051ff41c6fc94243d8bf \
--hash=sha256:6dbc4219fe45ece6a0cc6baafe0105604fdee551b5e876dc475d3955b77190ec \
--hash=sha256:9bd07746ce7f2198021a9fa187fa80df7b221ec5e4c234ab6f00ea355a3baf99
urllib3==1.24.1 \
--hash=sha256:61bf29cada3fc2fbefad4fdf059ea4bd1b4a86d2b6d15e1c7c0b582b9752fe39 \
--hash=sha256:de9529817c93f27c8ccbfead6985011db27bd0ddfcdb2d86f3f663385c6a9c22 \
# via dulwich

contrib/packaging/inno/requirements.txt.in

  • This file was added.
docutils
dulwich
keyring
pygments
pywin32

contrib/packaging/packagingutil.py

  • This file was added.
# packagingutil.py - Common packaging utility code.
#
# Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
# no-check-code because Python 3 native.
import gzip
import hashlib
import pathlib
import tarfile
import urllib.request
import zipfile
def hash_path(p: pathlib.Path):
h = hashlib.sha256()
with p.open('rb') as fh:
while True:
chunk = fh.read(65536)
if not chunk:
break
h.update(chunk)
return h.hexdigest()
class IntegrityError(Exception):
"""Represents an integrity error when downloading a URL."""
def secure_download_stream(url, size, sha256):
"""Securely download a URL to a stream of chunks.
If the integrity of the download fails, an IntegrityError is
raised.
"""
h = hashlib.sha256()
length = 0
with urllib.request.urlopen(url) as fh:
if not url.endswith('.gz') and fh.info().get('Content-Encoding') == 'gzip':
fh = gzip.GzipFile(fileobj=fh)
while True:
chunk = fh.read(65536)
if not chunk:
break
h.update(chunk)
length += len(chunk)
yield chunk
digest = h.hexdigest()
if length != size:
raise IntegrityError('size mismatch on %s: wanted %d; got %d' % (
url, size, length))
if digest != sha256:
raise IntegrityError('sha256 mismatch on %s: wanted %s; got %s' % (
url, sha256, digest))
def download_to_path(url: str, path: pathlib.Path, size: int, sha256: str):
"""Download a URL to a filesystem path, possibly with verification."""
# We download to a temporary file and rename at the end so there's
# no chance of the final file being partially written or containing
# bad data.
print('downloading %s to %s' % (url, path))
if path.exists():
good = True
if path.stat().st_size != size:
print('existing file size is wrong; removing')
good = False
if good:
if hash_path(path) != sha256:
print('existing file hash is wrong; removing')
good = False
if good:
print('%s exists and passes integrity checks' % path)
return
path.unlink()
tmp = path.with_name('%s.tmp' % path.name)
try:
with tmp.open('wb') as fh:
for chunk in secure_download_stream(url, size, sha256):
fh.write(chunk)
except IntegrityError:
tmp.unlink()
raise
tmp.rename(path)
print('successfully downloaded %s' % url)
def download_entry(entry: dict, dest_path: pathlib.Path, local_name=None) -> pathlib.Path:
url = entry['url']
local_name = local_name or url[url.rindex('/') + 1:]
local_path = dest_path / local_name
download_to_path(url, local_path, entry['size'], entry['sha256'])
return local_path
def extract_tar_to_directory(source: pathlib.Path, dest: pathlib.Path):
with tarfile.open(source, 'r') as tf:
tf.extractall(dest)
def extract_zip_to_directory(source: pathlib.Path, dest: pathlib.Path):
with zipfile.ZipFile(source, 'r') as zf:
zf.extractall(dest)

tests/test-check-code.t

#require test-repo #require test-repo
$ . "$TESTDIR/helpers-testrepo.sh" $ . "$TESTDIR/helpers-testrepo.sh"
$ check_code="$TESTDIR"/../contrib/check-code.py $ check_code="$TESTDIR"/../contrib/check-code.py
$ cd "$TESTDIR"/.. $ cd "$TESTDIR"/..
New errors are not allowed. Warnings are strongly discouraged. New errors are not allowed. Warnings are strongly discouraged.
(The writing "no-che?k-code" is for not skipping this file when checking.) (The writing "no-che?k-code" is for not skipping this file when checking.)
$ testrepohg locate \ $ testrepohg locate \
> -X contrib/python-zstandard \ > -X contrib/python-zstandard \
> -X hgext/fsmonitor/pywatchman \ > -X hgext/fsmonitor/pywatchman \
> -X mercurial/thirdparty \ > -X mercurial/thirdparty \
> | sed 's-\\-/-g' | "$check_code" --warnings --per-file=0 - || false > | sed 's-\\-/-g' | "$check_code" --warnings --per-file=0 - || false
Skipping contrib/packaging/inno/build.py it has no-che?k-code (glob)
Skipping contrib/packaging/packagingutil.py it has no-che?k-code (glob)
Skipping i18n/polib.py it has no-che?k-code (glob) Skipping i18n/polib.py it has no-che?k-code (glob)
Skipping mercurial/statprof.py it has no-che?k-code (glob) Skipping mercurial/statprof.py it has no-che?k-code (glob)
Skipping tests/badserverext.py it has no-che?k-code (glob) Skipping tests/badserverext.py it has no-che?k-code (glob)
@commands in debugcommands.py should be in alphabetical order. @commands in debugcommands.py should be in alphabetical order.
>>> import re >>> import re
>>> commands = [] >>> commands = []

tests/test-check-module-imports.t

$ testrepohg locate 'set:**.py or grep(r"^#!.*?python")' \ $ testrepohg locate 'set:**.py or grep(r"^#!.*?python")' \
> 'tests/**.t' \ > 'tests/**.t' \
> -X hgweb.cgi \ > -X hgweb.cgi \
> -X setup.py \ > -X setup.py \
> -X contrib/debugshell.py \ > -X contrib/debugshell.py \
> -X contrib/hgweb.fcgi \ > -X contrib/hgweb.fcgi \
> -X contrib/packaging/hg-docker \ > -X contrib/packaging/hg-docker \
> -X contrib/packaging/inno/ \
> -X contrib/packaging/packagingutil.py \
> -X contrib/python-zstandard/ \ > -X contrib/python-zstandard/ \
> -X contrib/win32/hgwebdir_wsgi.py \ > -X contrib/win32/hgwebdir_wsgi.py \
> -X contrib/perf-utils/perf-revlog-write-plot.py \ > -X contrib/perf-utils/perf-revlog-write-plot.py \
> -X doc/gendoc.py \ > -X doc/gendoc.py \
> -X doc/hgmanpage.py \ > -X doc/hgmanpage.py \
> -X i18n/posplit \ > -X i18n/posplit \
> -X mercurial/thirdparty \ > -X mercurial/thirdparty \
> -X tests/hypothesishelpers.py \ > -X tests/hypothesishelpers.py \
> -X tests/test-check-interfaces.py \ > -X tests/test-check-interfaces.py \
> -X tests/test-demandimport.py \ > -X tests/test-demandimport.py \
> -X tests/test-imports-checker.t \ > -X tests/test-imports-checker.t \
> -X tests/test-verify-repo-operations.py \ > -X tests/test-verify-repo-operations.py \
> | sed 's-\\-/-g' | "$PYTHON" "$import_checker" - > | sed 's-\\-/-g' | "$PYTHON" "$import_checker" -

tests/test-check-py3-compat.t

#require test-repo #require test-repo
$ . "$TESTDIR/helpers-testrepo.sh" $ . "$TESTDIR/helpers-testrepo.sh"
$ cd "$TESTDIR"/.. $ cd "$TESTDIR"/..
#if no-py3 #if no-py3
$ testrepohg files 'set:(**.py)' \ $ testrepohg files 'set:(**.py)' \
> -X contrib/packaging/inno/ \
> -X contrib/packaging/packagingutil.py \
> -X hgdemandimport/demandimportpy2.py \ > -X hgdemandimport/demandimportpy2.py \
> -X mercurial/thirdparty/cbor \ > -X mercurial/thirdparty/cbor \
> | sed 's|\\|/|g' | xargs "$PYTHON" contrib/check-py3-compat.py > | sed 's|\\|/|g' | xargs "$PYTHON" contrib/check-py3-compat.py
contrib/python-zstandard/setup.py not using absolute_import contrib/python-zstandard/setup.py not using absolute_import
contrib/python-zstandard/setup_zstd.py not using absolute_import contrib/python-zstandard/setup_zstd.py not using absolute_import
contrib/python-zstandard/tests/common.py not using absolute_import contrib/python-zstandard/tests/common.py not using absolute_import
contrib/python-zstandard/tests/test_buffer_util.py not using absolute_import contrib/python-zstandard/tests/test_buffer_util.py not using absolute_import
contrib/python-zstandard/tests/test_compressor.py not using absolute_import contrib/python-zstandard/tests/test_compressor.py not using absolute_import