This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8478

automation: support building Windows wheels for Python 3.7 and 3.8
ClosedPublic

Authored by indygreg on Apr 23 2020, 11:28 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG9d441f820c8b: automation: support building Windows wheels for Python 3.7 and 3.8
rHG48096e264d6c: automation: support building Windows wheels for Python 3.7 and 3.8
Summary

The time has come to support Python 3 on Windows.

Let's teach our automation code to produce Windows wheels for
Python 3.7 and 3.8.

We could theoretically support 3.5 and 3.6. But I don't think
it is worth it. People on Windows generally use the Mercurial
installers, not wheels. And I'd prefer we limit variability
and not have to worry about supporting earlier Python versions
if it can be helped.

As part of this, we change the invocation of pip to python.exe -m pip,
as this is what is being recommended in Python docs these days. And
it seemed to be required to avoid a weird build error. Why, I'm not
sure. But it looks like pip was having trouble finding a Visual Studio
files when invoked as pip.exe but not when using python.exe -m pip.
Who knows.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Apr 23 2020, 11:28 PM
martinvonz added a subscriber: martinvonz.Apr 30 2020, 12:48 AM

Why do we even give the choice between 3.7 and 3.8? Just because it was easy to do so anyway? Since the Python runtime is embedded, there's very little reason to try to match a system Python or something, right? To be clear, I don't have an issue with allowing it; I was just surprised by it and wanted to make sure I understand it correctly.

indygreg added a commit: rHG48096e264d6c: automation: support building Windows wheels for Python 3.7 and 3.8.Apr 30 2020, 1:52 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG48096e264d6c: automation: support building Windows wheels for Python 3.7 and 3.8 (authored by indygreg). · Explain Why
This revision was automatically updated to reflect the committed changes.
indygreg added a comment.Apr 30 2020, 10:04 PM
In D8478#126778, @martinvonz wrote:

Why do we even give the choice between 3.7 and 3.8? Just because it was easy to do so anyway? Since the Python runtime is embedded, there's very little reason to try to match a system Python or something, right? To be clear, I don't have an issue with allowing it; I was just surprised by it and wanted to make sure I understand it correctly.

For the installers, the runtime is embedded. The wheels are standalone, pre-compiled Python packages and installed via pip. The ABI is dependent on the OS, machine architecture, and Python version.

Personally, I would prefer that people install Mercurial via the installers so we can control the runtime. But some people (e.g. Mozilla and Python programmers) do install Mercurial via pip.

Revision Contents
Changeset List

Diff 21246

contrib/automation/hgautomation/cli.py

for a in arch: for a in arch:
windows.build_wix_installer( windows.build_wix_installer(
instance.winrm_client, a, DIST_PATH, version=version instance.winrm_client, a, DIST_PATH, version=version
) )
def build_windows_wheel( def build_windows_wheel(
hga: HGAutomation, aws_region, arch, revision, base_image_name hga: HGAutomation,
aws_region,
python_version,
arch,
revision,
base_image_name,
): ):
c = hga.aws_connection(aws_region) c = hga.aws_connection(aws_region)
image = aws.ensure_windows_dev_ami(c, base_image_name=base_image_name) image = aws.ensure_windows_dev_ami(c, base_image_name=base_image_name)
DIST_PATH.mkdir(exist_ok=True) DIST_PATH.mkdir(exist_ok=True)
with aws.temporary_windows_dev_instances(c, image, 't3.medium') as insts: with aws.temporary_windows_dev_instances(c, image, 't3.medium') as insts:
instance = insts[0] instance = insts[0]
windows.synchronize_hg(SOURCE_ROOT, revision, instance) windows.synchronize_hg(SOURCE_ROOT, revision, instance)
for py_version in python_version:
for a in arch: for a in arch:
windows.build_wheel(instance.winrm_client, a, DIST_PATH) windows.build_wheel(
instance.winrm_client, py_version, a, DIST_PATH
)
def build_all_windows_packages( def build_all_windows_packages(
hga: HGAutomation, aws_region, revision, version, base_image_name hga: HGAutomation, aws_region, revision, version, base_image_name
): ):
c = hga.aws_connection(aws_region) c = hga.aws_connection(aws_region)
image = aws.ensure_windows_dev_ami(c, base_image_name=base_image_name) image = aws.ensure_windows_dev_ami(c, base_image_name=base_image_name)
DIST_PATH.mkdir(exist_ok=True) DIST_PATH.mkdir(exist_ok=True)
with aws.temporary_windows_dev_instances(c, image, 't3.medium') as insts: with aws.temporary_windows_dev_instances(c, image, 't3.medium') as insts:
instance = insts[0] instance = insts[0]
winrm_client = instance.winrm_client winrm_client = instance.winrm_client
windows.synchronize_hg(SOURCE_ROOT, revision, instance) windows.synchronize_hg(SOURCE_ROOT, revision, instance)
for arch in ('x86', 'x64'): for py_version in ("2.7", "3.7", "3.8"):
for arch in ("x86", "x64"):
windows.purge_hg(winrm_client) windows.purge_hg(winrm_client)
windows.build_wheel(winrm_client, arch, DIST_PATH) windows.build_wheel(
winrm_client,
python_version=py_version,
arch=arch,
dest_path=DIST_PATH,
)
for arch in ('x86', 'x64'):
windows.purge_hg(winrm_client) windows.purge_hg(winrm_client)
windows.build_inno_installer( windows.build_inno_installer(
winrm_client, arch, DIST_PATH, version=version winrm_client, arch, DIST_PATH, version=version
) )
windows.purge_hg(winrm_client) windows.purge_hg(winrm_client)
windows.build_wix_installer( windows.build_wix_installer(
winrm_client, arch, DIST_PATH, version=version winrm_client, arch, DIST_PATH, version=version
) )
default=aws.WINDOWS_BASE_IMAGE_NAME, default=aws.WINDOWS_BASE_IMAGE_NAME,
) )
sp.set_defaults(func=build_inno) sp.set_defaults(func=build_inno)
sp = subparsers.add_parser( sp = subparsers.add_parser(
'build-windows-wheel', help='Build Windows wheel(s)', 'build-windows-wheel', help='Build Windows wheel(s)',
) )
sp.add_argument( sp.add_argument(
'--python-version',
help='Python version to build for',
choices={'2.7', '3.7', '3.8'},
nargs='*',
default=['3.8'],
)
sp.add_argument(
'--arch', '--arch',
help='Architecture to build for', help='Architecture to build for',
choices={'x86', 'x64'}, choices={'x86', 'x64'},
nargs='*', nargs='*',
default=['x64'], default=['x64'],
) )
sp.add_argument( sp.add_argument(
'--revision', help='Mercurial revision to build', default='.', '--revision', help='Mercurial revision to build', default='.',

contrib/automation/hgautomation/windows.py

C:\hgdev\python37-x64\python.exe contrib\packaging\packaging.py inno --python $python C:\hgdev\python37-x64\python.exe contrib\packaging\packaging.py inno --python $python
if ($LASTEXITCODE -ne 0) {{ if ($LASTEXITCODE -ne 0) {{
throw "process exited non-0: $LASTEXITCODE" throw "process exited non-0: $LASTEXITCODE"
}} }}
'''.lstrip() '''.lstrip()
BUILD_WHEEL = r''' BUILD_WHEEL = r'''
Set-Location C:\hgdev\src Set-Location C:\hgdev\src
C:\hgdev\python27-{arch}\Scripts\pip.exe wheel --wheel-dir dist . C:\hgdev\python{python_version}-{arch}\python.exe -m pip wheel --wheel-dir dist .
if ($LASTEXITCODE -ne 0) {{ if ($LASTEXITCODE -ne 0) {{
throw "process exited non-0: $LASTEXITCODE" throw "process exited non-0: $LASTEXITCODE"
}} }}
''' '''
BUILD_WIX = r''' BUILD_WIX = r'''
Set-Location C:\hgdev\src Set-Location C:\hgdev\src
$python = "C:\hgdev\python27-{arch}\python.exe" $python = "C:\hgdev\python27-{arch}\python.exe"
C:\hgdev\python37-x64\python.exe contrib\packaging\packaging.py wix --python $python {extra_args} C:\hgdev\python37-x64\python.exe contrib\packaging\packaging.py wix --python $python {extra_args}
if ($LASTEXITCODE -ne 0) {{ if ($LASTEXITCODE -ne 0) {{
throw "process exited non-0: $LASTEXITCODE" throw "process exited non-0: $LASTEXITCODE"
}} }}
''' '''
RUN_TESTS = r''' RUN_TESTS = r'''
C:\hgdev\MinGW\msys\1.0\bin\sh.exe --login -c "cd /c/hgdev/src/tests && /c/hgdev/{python_path}/python.exe run-tests.py {test_flags}" C:\hgdev\MinGW\msys\1.0\bin\sh.exe --login -c "cd /c/hgdev/src/tests && /c/hgdev/{python_path}/python.exe run-tests.py {test_flags}"
if ($LASTEXITCODE -ne 0) {{ if ($LASTEXITCODE -ne 0) {{
throw "process exited non-0: $LASTEXITCODE" throw "process exited non-0: $LASTEXITCODE"
}} }}
''' '''
X86_WHEEL_FILENAME = 'mercurial-{version}-cp27-cp27m-win32.whl' WHEEL_FILENAME_PYTHON27_X86 = 'mercurial-{version}-cp27-cp27m-win32.whl'
X64_WHEEL_FILENAME = 'mercurial-{version}-cp27-cp27m-win_amd64.whl' WHEEL_FILENAME_PYTHON27_X64 = 'mercurial-{version}-cp27-cp27m-win_amd64.whl'
WHEEL_FILENAME_PYTHON37_X86 = 'mercurial-{version}-cp37-cp37m-win32.whl'
WHEEL_FILENAME_PYTHON37_X64 = 'mercurial-{version}-cp37-cp37m-win_amd64.whl'
WHEEL_FILENAME_PYTHON38_X86 = 'mercurial-{version}-cp38-cp38-win32.whl'
WHEEL_FILENAME_PYTHON38_X64 = 'mercurial-{version}-cp38-cp38-win_amd64.whl'
X86_EXE_FILENAME = 'Mercurial-{version}.exe' X86_EXE_FILENAME = 'Mercurial-{version}.exe'
X64_EXE_FILENAME = 'Mercurial-{version}-x64.exe' X64_EXE_FILENAME = 'Mercurial-{version}-x64.exe'
X86_MSI_FILENAME = 'mercurial-{version}-x86.msi' X86_MSI_FILENAME = 'mercurial-{version}-x86.msi'
X64_MSI_FILENAME = 'mercurial-{version}-x64.msi' X64_MSI_FILENAME = 'mercurial-{version}-x64.msi'
MERCURIAL_SCM_BASE_URL = 'https://mercurial-scm.org/release/windows' MERCURIAL_SCM_BASE_URL = 'https://mercurial-scm.org/release/windows'
X86_USER_AGENT_PATTERN = '.*Windows.*' X86_USER_AGENT_PATTERN = '.*Windows.*'
ps = get_vc_prefix(arch) + BUILD_INNO.format( ps = get_vc_prefix(arch) + BUILD_INNO.format(
arch=arch, extra_args=' '.join(extra_args) arch=arch, extra_args=' '.join(extra_args)
) )
run_powershell(winrm_client, ps) run_powershell(winrm_client, ps)
copy_latest_dist(winrm_client, '*.exe', dest_path) copy_latest_dist(winrm_client, '*.exe', dest_path)
def build_wheel(winrm_client, arch: str, dest_path: pathlib.Path): def build_wheel(
winrm_client, python_version: str, arch: str, dest_path: pathlib.Path
):
"""Build Python wheels on a remote machine. """Build Python wheels on a remote machine.
Using a WinRM client, remote commands are executed to build a Python wheel Using a WinRM client, remote commands are executed to build a Python wheel
for Mercurial. for Mercurial.
""" """
print('Building Windows wheel for %s' % arch) print('Building Windows wheel for Python %s %s' % (python_version, arch))
ps = get_vc_prefix(arch) + BUILD_WHEEL.format(arch=arch)
ps = BUILD_WHEEL.format(
python_version=python_version.replace(".", ""), arch=arch
)
# Python 2.7 requires an activated environment.
if python_version == "2.7":
ps = get_vc_prefix(arch) + ps
run_powershell(winrm_client, ps) run_powershell(winrm_client, ps)
copy_latest_dist(winrm_client, '*.whl', dest_path) copy_latest_dist(winrm_client, '*.whl', dest_path)
def build_wix_installer( def build_wix_installer(
winrm_client, arch: str, dest_path: pathlib.Path, version=None winrm_client, arch: str, dest_path: pathlib.Path, version=None
): ):
"""Build the WiX installer on a remote machine. """Build the WiX installer on a remote machine.
ps = RUN_TESTS.format(python_path=python_path, test_flags=test_flags or '',) ps = RUN_TESTS.format(python_path=python_path, test_flags=test_flags or '',)
run_powershell(winrm_client, ps) run_powershell(winrm_client, ps)
def resolve_wheel_artifacts(dist_path: pathlib.Path, version: str): def resolve_wheel_artifacts(dist_path: pathlib.Path, version: str):
return ( return (
dist_path / X86_WHEEL_FILENAME.format(version=version), dist_path / WHEEL_FILENAME_PYTHON27_X86.format(version=version),
dist_path / X64_WHEEL_FILENAME.format(version=version), dist_path / WHEEL_FILENAME_PYTHON27_X64.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON37_X86.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON37_X64.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON38_X86.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON38_X64.format(version=version),
) )
def resolve_all_artifacts(dist_path: pathlib.Path, version: str): def resolve_all_artifacts(dist_path: pathlib.Path, version: str):
return ( return (
dist_path / X86_WHEEL_FILENAME.format(version=version), dist_path / WHEEL_FILENAME_PYTHON27_X86.format(version=version),
dist_path / X64_WHEEL_FILENAME.format(version=version), dist_path / WHEEL_FILENAME_PYTHON27_X64.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON37_X86.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON37_X64.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON38_X86.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON38_X64.format(version=version),
dist_path / X86_EXE_FILENAME.format(version=version), dist_path / X86_EXE_FILENAME.format(version=version),
dist_path / X64_EXE_FILENAME.format(version=version), dist_path / X64_EXE_FILENAME.format(version=version),
dist_path / X86_MSI_FILENAME.format(version=version), dist_path / X86_MSI_FILENAME.format(version=version),
dist_path / X64_MSI_FILENAME.format(version=version), dist_path / X64_MSI_FILENAME.format(version=version),
) )
def generate_latest_dat(version: str): def generate_latest_dat(version: str):