( )⚙ D11776 automation: support Python 3.10 on Windows

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D11776

automation: support Python 3.10 on Windows
ClosedPublic

Authored by indygreg on Nov 23 2021, 8:58 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHGfc1ba19ec4a0: automation: support Python 3.10 on Windows
Summary

Python 3.10 is out and we should support it. This commit teaches the
automation code to install and support building for Python 3.10.

Diff Detail

Repository
rHG Mercurial
Branch
stable
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

indygreg created this revision.Nov 23 2021, 8:58 PM
mharbison72 added a subscriber: mharbison72.Nov 23 2021, 10:19 PM

We should probably sit on this for a bit because windows_curses hasn't been updated to support py3.10 yet: https://github.com/zephyrproject-rtos/windows-curses/pull/28

Alphare added a subscriber: Alphare.Nov 29 2021, 9:33 AM
In D11776#180565, @mharbison72 wrote:

We should probably sit on this for a bit because windows_curses hasn't been updated to support py3.10 yet: https://github.com/zephyrproject-rtos/windows-curses/pull/28

I'm assuming this is good now that I queued your D11804?

indygreg added a comment.Nov 29 2021, 10:17 PM

The windows_curses dependency is only relevant for the installer Mercurial builds - the ones that ship additional Python packages. Mercurial's setup.py doesn't have a dependency on windows_curses. And this patch is only teaching the automation to produce 3.10 wheels - it doesn't touch the installers.

So I _think_ this patch (and releasing of 3.10 Windows wheels) is still reasonably safe.

We will, of course, have to upgrade windows_curses to a compatible version before switching the installer distributions to Python 3.10 (they are currently on 3.9 I believe).

mharbison72 added a comment.Nov 30 2021, 12:05 AM
In D11776#181199, @Alphare wrote:
In D11776#180565, @mharbison72 wrote:

We should probably sit on this for a bit because windows_curses hasn't been updated to support py3.10 yet: https://github.com/zephyrproject-rtos/windows-curses/pull/28

I'm assuming this is good now that I queued your D11804?

Yes.

indygreg added a commit: rHGfc1ba19ec4a0: automation: support Python 3.10 on Windows.Nov 30 2021, 12:26 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHGfc1ba19ec4a0: automation: support Python 3.10 on Windows (authored by indygreg). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 31098

contrib/automation/hgautomation/cli.py

with aws.temporary_windows_dev_instances(c, image, 't3.medium') as insts: with aws.temporary_windows_dev_instances(c, image, 't3.medium') as insts:
instance = insts[0] instance = insts[0]
winrm_client = instance.winrm_client winrm_client = instance.winrm_client
windows.synchronize_hg(SOURCE_ROOT, revision, instance) windows.synchronize_hg(SOURCE_ROOT, revision, instance)
for py_version in ("2.7", "3.7", "3.8", "3.9"): for py_version in ("2.7", "3.7", "3.8", "3.9", "3.10"):
for arch in ("x86", "x64"): for arch in ("x86", "x64"):
windows.purge_hg(winrm_client) windows.purge_hg(winrm_client)
windows.build_wheel( windows.build_wheel(
winrm_client, winrm_client,
python_version=py_version, python_version=py_version,
arch=arch, arch=arch,
dest_path=DIST_PATH, dest_path=DIST_PATH,
) )
sp = subparsers.add_parser( sp = subparsers.add_parser(
'build-windows-wheel', 'build-windows-wheel',
help='Build Windows wheel(s)', help='Build Windows wheel(s)',
) )
sp.add_argument( sp.add_argument(
'--python-version', '--python-version',
help='Python version to build for', help='Python version to build for',
choices={'2.7', '3.7', '3.8', '3.9'}, choices={'2.7', '3.7', '3.8', '3.9', '3.10'},
nargs='*', nargs='*',
default=['3.8'], default=['3.8'],
) )
sp.add_argument( sp.add_argument(
'--arch', '--arch',
help='Architecture to build for', help='Architecture to build for',
choices={'x86', 'x64'}, choices={'x86', 'x64'},
nargs='*', nargs='*',
sp.add_argument( sp.add_argument(
'--instance-type', '--instance-type',
help='EC2 instance type to use', help='EC2 instance type to use',
default='t3.medium', default='t3.medium',
) )
sp.add_argument( sp.add_argument(
'--python-version', '--python-version',
help='Python version to use', help='Python version to use',
choices={'2.7', '3.5', '3.6', '3.7', '3.8', '3.9'}, choices={'2.7', '3.5', '3.6', '3.7', '3.8', '3.9', '3.10'},
default='2.7', default='2.7',
) )
sp.add_argument( sp.add_argument(
'--arch', '--arch',
help='Architecture to test', help='Architecture to test',
choices={'x86', 'x64'}, choices={'x86', 'x64'},
default='x64', default='x64',
) )

contrib/automation/hgautomation/windows.py

WHEEL_FILENAME_PYTHON27_X86 = 'mercurial-{version}-cp27-cp27m-win32.whl' WHEEL_FILENAME_PYTHON27_X86 = 'mercurial-{version}-cp27-cp27m-win32.whl'
WHEEL_FILENAME_PYTHON27_X64 = 'mercurial-{version}-cp27-cp27m-win_amd64.whl' WHEEL_FILENAME_PYTHON27_X64 = 'mercurial-{version}-cp27-cp27m-win_amd64.whl'
WHEEL_FILENAME_PYTHON37_X86 = 'mercurial-{version}-cp37-cp37m-win32.whl' WHEEL_FILENAME_PYTHON37_X86 = 'mercurial-{version}-cp37-cp37m-win32.whl'
WHEEL_FILENAME_PYTHON37_X64 = 'mercurial-{version}-cp37-cp37m-win_amd64.whl' WHEEL_FILENAME_PYTHON37_X64 = 'mercurial-{version}-cp37-cp37m-win_amd64.whl'
WHEEL_FILENAME_PYTHON38_X86 = 'mercurial-{version}-cp38-cp38-win32.whl' WHEEL_FILENAME_PYTHON38_X86 = 'mercurial-{version}-cp38-cp38-win32.whl'
WHEEL_FILENAME_PYTHON38_X64 = 'mercurial-{version}-cp38-cp38-win_amd64.whl' WHEEL_FILENAME_PYTHON38_X64 = 'mercurial-{version}-cp38-cp38-win_amd64.whl'
WHEEL_FILENAME_PYTHON39_X86 = 'mercurial-{version}-cp39-cp39-win32.whl' WHEEL_FILENAME_PYTHON39_X86 = 'mercurial-{version}-cp39-cp39-win32.whl'
WHEEL_FILENAME_PYTHON39_X64 = 'mercurial-{version}-cp39-cp39-win_amd64.whl' WHEEL_FILENAME_PYTHON39_X64 = 'mercurial-{version}-cp39-cp39-win_amd64.whl'
WHEEL_FILENAME_PYTHON310_X86 = 'mercurial-{version}-cp310-cp310-win32.whl'
WHEEL_FILENAME_PYTHON310_X64 = 'mercurial-{version}-cp310-cp310-win_amd64.whl'
EXE_FILENAME_PYTHON2_X86 = 'Mercurial-{version}-x86-python2.exe' EXE_FILENAME_PYTHON2_X86 = 'Mercurial-{version}-x86-python2.exe'
EXE_FILENAME_PYTHON2_X64 = 'Mercurial-{version}-x64-python2.exe' EXE_FILENAME_PYTHON2_X64 = 'Mercurial-{version}-x64-python2.exe'
EXE_FILENAME_PYTHON3_X86 = 'Mercurial-{version}-x86.exe' EXE_FILENAME_PYTHON3_X86 = 'Mercurial-{version}-x86.exe'
EXE_FILENAME_PYTHON3_X64 = 'Mercurial-{version}-x64.exe' EXE_FILENAME_PYTHON3_X64 = 'Mercurial-{version}-x64.exe'
MSI_FILENAME_PYTHON2_X86 = 'mercurial-{version}-x86-python2.msi' MSI_FILENAME_PYTHON2_X86 = 'mercurial-{version}-x86-python2.msi'
MSI_FILENAME_PYTHON2_X64 = 'mercurial-{version}-x64-python2.msi' MSI_FILENAME_PYTHON2_X64 = 'mercurial-{version}-x64-python2.msi'
dist_path / WHEEL_FILENAME_PYTHON27_X86.format(version=version), dist_path / WHEEL_FILENAME_PYTHON27_X86.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON27_X64.format(version=version), dist_path / WHEEL_FILENAME_PYTHON27_X64.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON37_X86.format(version=version), dist_path / WHEEL_FILENAME_PYTHON37_X86.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON37_X64.format(version=version), dist_path / WHEEL_FILENAME_PYTHON37_X64.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON38_X86.format(version=version), dist_path / WHEEL_FILENAME_PYTHON38_X86.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON38_X64.format(version=version), dist_path / WHEEL_FILENAME_PYTHON38_X64.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON39_X86.format(version=version), dist_path / WHEEL_FILENAME_PYTHON39_X86.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON39_X64.format(version=version), dist_path / WHEEL_FILENAME_PYTHON39_X64.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON310_X86.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON310_X64.format(version=version),
) )
def resolve_all_artifacts(dist_path: pathlib.Path, version: str): def resolve_all_artifacts(dist_path: pathlib.Path, version: str):
return ( return (
dist_path / WHEEL_FILENAME_PYTHON27_X86.format(version=version), dist_path / WHEEL_FILENAME_PYTHON27_X86.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON27_X64.format(version=version), dist_path / WHEEL_FILENAME_PYTHON27_X64.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON37_X86.format(version=version), dist_path / WHEEL_FILENAME_PYTHON37_X86.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON37_X64.format(version=version), dist_path / WHEEL_FILENAME_PYTHON37_X64.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON38_X86.format(version=version), dist_path / WHEEL_FILENAME_PYTHON38_X86.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON38_X64.format(version=version), dist_path / WHEEL_FILENAME_PYTHON38_X64.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON39_X86.format(version=version), dist_path / WHEEL_FILENAME_PYTHON39_X86.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON39_X64.format(version=version), dist_path / WHEEL_FILENAME_PYTHON39_X64.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON310_X86.format(version=version),
dist_path / WHEEL_FILENAME_PYTHON310_X64.format(version=version),
dist_path / EXE_FILENAME_PYTHON2_X86.format(version=version), dist_path / EXE_FILENAME_PYTHON2_X86.format(version=version),
dist_path / EXE_FILENAME_PYTHON2_X64.format(version=version), dist_path / EXE_FILENAME_PYTHON2_X64.format(version=version),
dist_path / EXE_FILENAME_PYTHON3_X86.format(version=version), dist_path / EXE_FILENAME_PYTHON3_X86.format(version=version),
dist_path / EXE_FILENAME_PYTHON3_X64.format(version=version), dist_path / EXE_FILENAME_PYTHON3_X64.format(version=version),
dist_path / MSI_FILENAME_PYTHON2_X86.format(version=version), dist_path / MSI_FILENAME_PYTHON2_X86.format(version=version),
dist_path / MSI_FILENAME_PYTHON2_X64.format(version=version), dist_path / MSI_FILENAME_PYTHON2_X64.format(version=version),
dist_path / MSI_FILENAME_PYTHON3_X86.format(version=version), dist_path / MSI_FILENAME_PYTHON3_X86.format(version=version),
dist_path / MSI_FILENAME_PYTHON3_X64.format(version=version), dist_path / MSI_FILENAME_PYTHON3_X64.format(version=version),

contrib/install-windows-dependencies.ps1

$PYTHON38_x64_URL = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-amd64.exe" $PYTHON38_x64_URL = "https://www.python.org/ftp/python/3.8.10/python-3.8.10-amd64.exe"
$PYTHON38_x64_SHA256 = "7628244cb53408b50639d2c1287c659f4e29d3dfdb9084b11aed5870c0c6a48a" $PYTHON38_x64_SHA256 = "7628244cb53408b50639d2c1287c659f4e29d3dfdb9084b11aed5870c0c6a48a"
$PYTHON39_x86_URL = "https://www.python.org/ftp/python/3.9.9/python-3.9.9.exe" $PYTHON39_x86_URL = "https://www.python.org/ftp/python/3.9.9/python-3.9.9.exe"
$PYTHON39_x86_SHA256 = "6646a5683adf14d35e8c53aab946895bc0f0b825f7acac3a62cc85ee7d0dc71a" $PYTHON39_x86_SHA256 = "6646a5683adf14d35e8c53aab946895bc0f0b825f7acac3a62cc85ee7d0dc71a"
$PYTHON39_X64_URL = "https://www.python.org/ftp/python/3.9.9/python-3.9.9-amd64.exe" $PYTHON39_X64_URL = "https://www.python.org/ftp/python/3.9.9/python-3.9.9-amd64.exe"
$PYTHON39_x64_SHA256 = "137d59e5c0b01a8f1bdcba08344402ae658c81c6bf03b6602bd8b4e951ad0714" $PYTHON39_x64_SHA256 = "137d59e5c0b01a8f1bdcba08344402ae658c81c6bf03b6602bd8b4e951ad0714"
$PYTHON310_x86_URL = "https://www.python.org/ftp/python/3.10.0/python-3.10.0.exe"
$PYTHON310_x86_SHA256 = "ea896eeefb1db9e12fb89ec77a6e28c9fe52b4a162a34c85d9688be2ec2392e8"
$PYTHON310_X64_URL = "https://www.python.org/ftp/python/3.10.0/python-3.10.0-amd64.exe"
$PYTHON310_x64_SHA256 = "cb580eb7dc55f9198e650f016645023e8b2224cf7d033857d12880b46c5c94ef"
# PIP 19.2.3. # PIP 19.2.3.
$PIP_URL = "https://github.com/pypa/get-pip/raw/309a56c5fd94bd1134053a541cb4657a4e47e09d/get-pip.py" $PIP_URL = "https://github.com/pypa/get-pip/raw/309a56c5fd94bd1134053a541cb4657a4e47e09d/get-pip.py"
$PIP_SHA256 = "57e3643ff19f018f8a00dfaa6b7e4620e3c1a7a2171fd218425366ec006b3bfe" $PIP_SHA256 = "57e3643ff19f018f8a00dfaa6b7e4620e3c1a7a2171fd218425366ec006b3bfe"
$INNO_SETUP_URL = "http://files.jrsoftware.org/is/5/innosetup-5.6.1-unicode.exe" $INNO_SETUP_URL = "http://files.jrsoftware.org/is/5/innosetup-5.6.1-unicode.exe"
$INNO_SETUP_SHA256 = "27D49E9BC769E9D1B214C153011978DB90DC01C2ACD1DDCD9ED7B3FE3B96B538" $INNO_SETUP_SHA256 = "27D49E9BC769E9D1B214C153011978DB90DC01C2ACD1DDCD9ED7B3FE3B96B538"
$MINGW_BIN_URL = "https://osdn.net/frs/redir.php?m=constant&f=mingw%2F68260%2Fmingw-get-0.6.3-mingw32-pre-20170905-1-bin.zip" $MINGW_BIN_URL = "https://osdn.net/frs/redir.php?m=constant&f=mingw%2F68260%2Fmingw-get-0.6.3-mingw32-pre-20170905-1-bin.zip"
$pip = "${prefix}\assets\get-pip.py" $pip = "${prefix}\assets\get-pip.py"
Secure-Download $PYTHON37_x86_URL ${prefix}\assets\python37-x86.exe $PYTHON37_x86_SHA256 Secure-Download $PYTHON37_x86_URL ${prefix}\assets\python37-x86.exe $PYTHON37_x86_SHA256
Secure-Download $PYTHON37_x64_URL ${prefix}\assets\python37-x64.exe $PYTHON37_x64_SHA256 Secure-Download $PYTHON37_x64_URL ${prefix}\assets\python37-x64.exe $PYTHON37_x64_SHA256
Secure-Download $PYTHON38_x86_URL ${prefix}\assets\python38-x86.exe $PYTHON38_x86_SHA256 Secure-Download $PYTHON38_x86_URL ${prefix}\assets\python38-x86.exe $PYTHON38_x86_SHA256
Secure-Download $PYTHON38_x64_URL ${prefix}\assets\python38-x64.exe $PYTHON38_x64_SHA256 Secure-Download $PYTHON38_x64_URL ${prefix}\assets\python38-x64.exe $PYTHON38_x64_SHA256
Secure-Download $PYTHON39_x86_URL ${prefix}\assets\python39-x86.exe $PYTHON39_x86_SHA256 Secure-Download $PYTHON39_x86_URL ${prefix}\assets\python39-x86.exe $PYTHON39_x86_SHA256
Secure-Download $PYTHON39_x64_URL ${prefix}\assets\python39-x64.exe $PYTHON39_x64_SHA256 Secure-Download $PYTHON39_x64_URL ${prefix}\assets\python39-x64.exe $PYTHON39_x64_SHA256
Secure-Download $PYTHON310_x86_URL ${prefix}\assets\python310-x86.exe $PYTHON310_x86_SHA256
Secure-Download $PYTHON310_x64_URL ${prefix}\assets\python310-x64.exe $PYTHON310_x64_SHA256
Secure-Download $PIP_URL ${pip} $PIP_SHA256 Secure-Download $PIP_URL ${pip} $PIP_SHA256
Secure-Download $VS_BUILD_TOOLS_URL ${prefix}\assets\vs_buildtools.exe $VS_BUILD_TOOLS_SHA256 Secure-Download $VS_BUILD_TOOLS_URL ${prefix}\assets\vs_buildtools.exe $VS_BUILD_TOOLS_SHA256
Secure-Download $INNO_SETUP_URL ${prefix}\assets\InnoSetup.exe $INNO_SETUP_SHA256 Secure-Download $INNO_SETUP_URL ${prefix}\assets\InnoSetup.exe $INNO_SETUP_SHA256
Secure-Download $MINGW_BIN_URL ${prefix}\assets\mingw-get-bin.zip $MINGW_BIN_SHA256 Secure-Download $MINGW_BIN_URL ${prefix}\assets\mingw-get-bin.zip $MINGW_BIN_SHA256
Secure-Download $MERCURIAL_WHEEL_URL ${prefix}\assets\${MERCURIAL_WHEEL_FILENAME} $MERCURIAL_WHEEL_SHA256 Secure-Download $MERCURIAL_WHEEL_URL ${prefix}\assets\${MERCURIAL_WHEEL_FILENAME} $MERCURIAL_WHEEL_SHA256
Secure-Download $RUSTUP_INIT_URL ${prefix}\assets\rustup-init.exe $RUSTUP_INIT_SHA256 Secure-Download $RUSTUP_INIT_URL ${prefix}\assets\rustup-init.exe $RUSTUP_INIT_SHA256
Secure-Download $PYOXIDIZER_URL ${prefix}\assets\PyOxidizer.msi $PYOXIDIZER_SHA256 Secure-Download $PYOXIDIZER_URL ${prefix}\assets\PyOxidizer.msi $PYOXIDIZER_SHA256
Install-Python3 "Python 3.7 32-bit" ${prefix}\assets\python37-x86.exe ${prefix}\python37-x86 ${pip} Install-Python3 "Python 3.7 32-bit" ${prefix}\assets\python37-x86.exe ${prefix}\python37-x86 ${pip}
Install-Python3 "Python 3.7 64-bit" ${prefix}\assets\python37-x64.exe ${prefix}\python37-x64 ${pip} Install-Python3 "Python 3.7 64-bit" ${prefix}\assets\python37-x64.exe ${prefix}\python37-x64 ${pip}
Install-Python3 "Python 3.8 32-bit" ${prefix}\assets\python38-x86.exe ${prefix}\python38-x86 ${pip} Install-Python3 "Python 3.8 32-bit" ${prefix}\assets\python38-x86.exe ${prefix}\python38-x86 ${pip}
# Install-Python3 "Python 3.8 64-bit" ${prefix}\assets\python38-x64.exe ${prefix}\python38-x64 ${pip} # Install-Python3 "Python 3.8 64-bit" ${prefix}\assets\python38-x64.exe ${prefix}\python38-x64 ${pip}
Install-Python3 "Python 3.9 32-bit" ${prefix}\assets\python39-x86.exe ${prefix}\python39-x86 ${pip} Install-Python3 "Python 3.9 32-bit" ${prefix}\assets\python39-x86.exe ${prefix}\python39-x86 ${pip}
Install-Python3 "Python 3.9 64-bit" ${prefix}\assets\python39-x64.exe ${prefix}\python39-x64 ${pip} Install-Python3 "Python 3.9 64-bit" ${prefix}\assets\python39-x64.exe ${prefix}\python39-x64 ${pip}
Install-Python3 "Python 3.10 32-bit" ${prefix}\assets\python310-x86.exe ${prefix}\python310-x86 ${pip}
Install-Python3 "Python 3.10 64-bit" ${prefix}\assets\python310-x64.exe ${prefix}\python310-x64 ${pip}
Write-Output "installing Visual Studio 2017 Build Tools and SDKs" Write-Output "installing Visual Studio 2017 Build Tools and SDKs"
Invoke-Process ${prefix}\assets\vs_buildtools.exe "--quiet --wait --norestart --nocache --channelUri https://aka.ms/vs/15/release/channel --add Microsoft.VisualStudio.Workload.MSBuildTools --add Microsoft.VisualStudio.Component.Windows10SDK.17763 --add Microsoft.VisualStudio.Workload.VCTools --add Microsoft.VisualStudio.Component.Windows10SDK --add Microsoft.VisualStudio.Component.VC.140" Invoke-Process ${prefix}\assets\vs_buildtools.exe "--quiet --wait --norestart --nocache --channelUri https://aka.ms/vs/15/release/channel --add Microsoft.VisualStudio.Workload.MSBuildTools --add Microsoft.VisualStudio.Component.Windows10SDK.17763 --add Microsoft.VisualStudio.Workload.VCTools --add Microsoft.VisualStudio.Component.Windows10SDK --add Microsoft.VisualStudio.Component.VC.140"
Write-Output "installing PyOxidizer" Write-Output "installing PyOxidizer"
Invoke-Process msiexec.exe "/i ${prefix}\assets\PyOxidizer.msi /l* ${prefix}\assets\PyOxidizer.log /quiet" Invoke-Process msiexec.exe "/i ${prefix}\assets\PyOxidizer.msi /l* ${prefix}\assets\PyOxidizer.log /quiet"
Install-Rust ${prefix} Install-Rust ${prefix}