This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10680

contrib: install PyOxidizer 0.16.0
ClosedPublic

Authored by indygreg on May 6 2021, 8:00 PM.

Details

Reviewers
mharbison72
Group Reviewers
hg-reviewers
Commits
rHG0de5eefbe4b2: contrib: install PyOxidizer 0.16.0
Summary

PyOxidizer now provides MSI installers and pre-built Linux binaries.
So we install that way. This significantly reduces the time to bootstrap
a new machine in automation, as building PyOxidizer from source on a low
core count machine takes several minutes.

This change temporarily breaks the ability of the automated environment
to use the in-repo pyoxidizer.bzl configuration file, as there are
backwards-incompatible changes with the upgrade that need to be
reflected. We'll handle those in subsequent commits.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.May 6 2021, 8:00 PM
indygreg edited the summary of this revision. (Show Details)May 6 2021, 9:38 PM
indygreg updated this revision to Diff 27732.
baymax updated this revision to Diff 27751.May 7 2021, 11:30 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

indygreg edited the summary of this revision. (Show Details)May 7 2021, 12:42 PM
indygreg updated this revision to Diff 27762.
indygreg planned changes to this revision.May 7 2021, 8:38 PM

I'll need to release a new version of PyOxidizer to fix wonky behavior due to interaction of file mtimes and pip (https://github.com/indygreg/PyOxidizer/commit/a0bdde179aaed8521d3a668e518554814bcffdba).

indygreg retitled this revision from contrib: install PyOxidizer 0.15.0 to contrib: install PyOxidizer 0.16.0.May 9 2021, 1:08 PM
indygreg edited the summary of this revision. (Show Details)
indygreg updated this revision to Diff 27769.
baymax updated this revision to Diff 27788.May 10 2021, 11:37 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

mharbison72 accepted this revision.May 10 2021, 6:35 PM
This revision is now accepted and ready to land.May 10 2021, 6:36 PM
indygreg added a commit: rHG0de5eefbe4b2: contrib: install PyOxidizer 0.16.0.May 15 2021, 7:23 PM
Closed by commit rHG0de5eefbe4b2: contrib: install PyOxidizer 0.16.0 (authored by indygreg). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 27940

contrib/automation/hgautomation/linux.py

${PYENV_ROOT}/versions/${v}/bin/pip install -r /hgdev/${REQUIREMENTS} ${PYENV_ROOT}/versions/${v}/bin/pip install -r /hgdev/${REQUIREMENTS}
done done
pyenv global ${PYENV2_VERSIONS} ${PYENV3_VERSIONS} system pyenv global ${PYENV2_VERSIONS} ${PYENV3_VERSIONS} system
'''.lstrip().replace( '''.lstrip().replace(
'\r\n', '\n' '\r\n', '\n'
) )
INSTALL_PYOXIDIZER = r'''
PYOXIDIZER_VERSION=0.16.0
PYOXIDIZER_SHA256=8875471c270312fbb934007fd30f65f1904cc0f5da6188d61c90ed2129b9f9c1
PYOXIDIZER_URL=https://github.com/indygreg/PyOxidizer/releases/download/pyoxidizer%2F${PYOXIDIZER_VERSION}/pyoxidizer-${PYOXIDIZER_VERSION}-linux_x86_64.zip
wget -O pyoxidizer.zip --progress dot:mega ${PYOXIDIZER_URL}
echo "${PYOXIDIZER_SHA256} pyoxidizer.zip" | sha256sum --check -
unzip pyoxidizer.zip
chmod +x pyoxidizer
sudo mv pyoxidizer /usr/local/bin/pyoxidizer
'''
INSTALL_RUST = r''' INSTALL_RUST = r'''
RUSTUP_INIT_SHA256=a46fe67199b7bcbbde2dcbc23ae08db6f29883e260e23899a88b9073effc9076 RUSTUP_INIT_SHA256=a46fe67199b7bcbbde2dcbc23ae08db6f29883e260e23899a88b9073effc9076
wget -O rustup-init --progress dot:mega https://static.rust-lang.org/rustup/archive/1.18.3/x86_64-unknown-linux-gnu/rustup-init wget -O rustup-init --progress dot:mega https://static.rust-lang.org/rustup/archive/1.18.3/x86_64-unknown-linux-gnu/rustup-init
echo "${RUSTUP_INIT_SHA256} rustup-init" | sha256sum --check - echo "${RUSTUP_INIT_SHA256} rustup-init" | sha256sum --check -
chmod +x rustup-init chmod +x rustup-init
sudo -H -u hg -g hg ./rustup-init -y sudo -H -u hg -g hg ./rustup-init -y
sudo -H -u hg -g hg /home/hg/.cargo/bin/rustup install 1.41.1 1.52.0 sudo -H -u hg -g hg /home/hg/.cargo/bin/rustup install 1.41.1 1.52.0
sudo -H -u hg -g hg /home/hg/.cargo/bin/rustup component add clippy sudo -H -u hg -g hg /home/hg/.cargo/bin/rustup component add clippy
sudo -H -u hg -g hg /home/hg/.cargo/bin/cargo install --version 0.10.3 pyoxidizer
''' '''
BOOTSTRAP_VIRTUALENV = r''' BOOTSTRAP_VIRTUALENV = r'''
/usr/bin/virtualenv /hgdev/venv-bootstrap /usr/bin/virtualenv /hgdev/venv-bootstrap
HG_SHA256=35fc8ba5e0379c1b3affa2757e83fb0509e8ac314cbd9f1fd133cf265d16e49f HG_SHA256=35fc8ba5e0379c1b3affa2757e83fb0509e8ac314cbd9f1fd133cf265d16e49f
HG_TARBALL=mercurial-5.1.1.tar.gz HG_TARBALL=mercurial-5.1.1.tar.gz
sudo update-alternatives --install /usr/bin/clang-format clang-format \ sudo update-alternatives --install /usr/bin/clang-format clang-format \
/usr/bin/clang-format-6.0 1000 /usr/bin/clang-format-6.0 1000
sudo mkdir /hgdev sudo mkdir /hgdev
# Will be normalized to hg:hg later. # Will be normalized to hg:hg later.
sudo chown `whoami` /hgdev sudo chown `whoami` /hgdev
{install_rust} {install_rust}
{install_pyoxidizer}
cp requirements-*.txt /hgdev/ cp requirements-*.txt /hgdev/
# Disable the pip version check because it uses the network and can # Disable the pip version check because it uses the network and can
# be annoying. # be annoying.
cat << EOF | sudo tee -a /etc/pip.conf cat << EOF | sudo tee -a /etc/pip.conf
[global] [global]
disable-pip-version-check = True disable-pip-version-check = True
[phases] [phases]
publish = false publish = false
EOF EOF
sudo chown -R hg:hg /hgdev sudo chown -R hg:hg /hgdev
'''.lstrip() '''.lstrip()
.format( .format(
install_rust=INSTALL_RUST, install_rust=INSTALL_RUST,
install_pyoxidizer=INSTALL_PYOXIDIZER,
install_pythons=INSTALL_PYTHONS, install_pythons=INSTALL_PYTHONS,
bootstrap_virtualenv=BOOTSTRAP_VIRTUALENV, bootstrap_virtualenv=BOOTSTRAP_VIRTUALENV,
) )
.replace('\r\n', '\n') .replace('\r\n', '\n')
) )
# Prepares /hgdev for operations. # Prepares /hgdev for operations.

contrib/install-windows-dependencies.ps1

$MERCURIAL_WHEEL_FILENAME = "mercurial-5.1.2-cp27-cp27m-win_amd64.whl" $MERCURIAL_WHEEL_FILENAME = "mercurial-5.1.2-cp27-cp27m-win_amd64.whl"
$MERCURIAL_WHEEL_URL = "https://files.pythonhosted.org/packages/6d/47/e031e47f7fe9b16e4e3383da47e2b0a7eae6e603996bc67a03ec4fa1b3f4/$MERCURIAL_WHEEL_FILENAME" $MERCURIAL_WHEEL_URL = "https://files.pythonhosted.org/packages/6d/47/e031e47f7fe9b16e4e3383da47e2b0a7eae6e603996bc67a03ec4fa1b3f4/$MERCURIAL_WHEEL_FILENAME"
$MERCURIAL_WHEEL_SHA256 = "1d18c7f6ca1456f0f62ee65c9a50c14cbba48ce6e924930cdb10537f5c9eaf5f" $MERCURIAL_WHEEL_SHA256 = "1d18c7f6ca1456f0f62ee65c9a50c14cbba48ce6e924930cdb10537f5c9eaf5f"
$RUSTUP_INIT_URL = "https://static.rust-lang.org/rustup/archive/1.21.1/x86_64-pc-windows-gnu/rustup-init.exe" $RUSTUP_INIT_URL = "https://static.rust-lang.org/rustup/archive/1.21.1/x86_64-pc-windows-gnu/rustup-init.exe"
$RUSTUP_INIT_SHA256 = "d17df34ba974b9b19cf5c75883a95475aa22ddc364591d75d174090d55711c72" $RUSTUP_INIT_SHA256 = "d17df34ba974b9b19cf5c75883a95475aa22ddc364591d75d174090d55711c72"
$PYOXIDIZER_URL = "https://github.com/indygreg/PyOxidizer/releases/download/pyoxidizer%2F0.16.0/PyOxidizer-0.16.0-x64.msi"
$PYOXIDIZER_SHA256 = "2a9c58add9161c272c418d5e6dec13fbe648f624b5d26770190357e4d664f24e"
# Writing progress slows down downloads substantially. So disable it. # Writing progress slows down downloads substantially. So disable it.
$progressPreference = 'silentlyContinue' $progressPreference = 'silentlyContinue'
function Secure-Download($url, $path, $sha256) { function Secure-Download($url, $path, $sha256) {
if (Test-Path -Path $path) { if (Test-Path -Path $path) {
Get-FileHash -Path $path -Algorithm SHA256 -OutVariable hash Get-FileHash -Path $path -Algorithm SHA256 -OutVariable hash
if ($hash.Hash -eq $sha256) { if ($hash.Hash -eq $sha256) {
Write-Output "installing Rust" Write-Output "installing Rust"
$Env:RUSTUP_HOME = "${prefix}\rustup" $Env:RUSTUP_HOME = "${prefix}\rustup"
$Env:CARGO_HOME = "${prefix}\cargo" $Env:CARGO_HOME = "${prefix}\cargo"
Invoke-Process "${prefix}\assets\rustup-init.exe" "-y --default-host x86_64-pc-windows-msvc" Invoke-Process "${prefix}\assets\rustup-init.exe" "-y --default-host x86_64-pc-windows-msvc"
Invoke-Process "${prefix}\cargo\bin\rustup.exe" "target add i686-pc-windows-msvc" Invoke-Process "${prefix}\cargo\bin\rustup.exe" "target add i686-pc-windows-msvc"
Invoke-Process "${prefix}\cargo\bin\rustup.exe" "install 1.52.0" Invoke-Process "${prefix}\cargo\bin\rustup.exe" "install 1.52.0"
Invoke-Process "${prefix}\cargo\bin\rustup.exe" "component add clippy" Invoke-Process "${prefix}\cargo\bin\rustup.exe" "component add clippy"
# Install PyOxidizer for packaging.
Invoke-Process "${prefix}\cargo\bin\cargo.exe" "install --version 0.10.3 pyoxidizer"
} }
function Install-Dependencies($prefix) { function Install-Dependencies($prefix) {
if (!(Test-Path -Path $prefix\assets)) { if (!(Test-Path -Path $prefix\assets)) {
New-Item -Path $prefix\assets -ItemType Directory New-Item -Path $prefix\assets -ItemType Directory
} }
$pip = "${prefix}\assets\get-pip.py" $pip = "${prefix}\assets\get-pip.py"
Secure-Download $PYTHON39_x64_URL ${prefix}\assets\python39-x64.exe $PYTHON39_x64_SHA256 Secure-Download $PYTHON39_x64_URL ${prefix}\assets\python39-x64.exe $PYTHON39_x64_SHA256
Secure-Download $PIP_URL ${pip} $PIP_SHA256 Secure-Download $PIP_URL ${pip} $PIP_SHA256
Secure-Download $VIRTUALENV_URL ${prefix}\assets\virtualenv.tar.gz $VIRTUALENV_SHA256 Secure-Download $VIRTUALENV_URL ${prefix}\assets\virtualenv.tar.gz $VIRTUALENV_SHA256
Secure-Download $VS_BUILD_TOOLS_URL ${prefix}\assets\vs_buildtools.exe $VS_BUILD_TOOLS_SHA256 Secure-Download $VS_BUILD_TOOLS_URL ${prefix}\assets\vs_buildtools.exe $VS_BUILD_TOOLS_SHA256
Secure-Download $INNO_SETUP_URL ${prefix}\assets\InnoSetup.exe $INNO_SETUP_SHA256 Secure-Download $INNO_SETUP_URL ${prefix}\assets\InnoSetup.exe $INNO_SETUP_SHA256
Secure-Download $MINGW_BIN_URL ${prefix}\assets\mingw-get-bin.zip $MINGW_BIN_SHA256 Secure-Download $MINGW_BIN_URL ${prefix}\assets\mingw-get-bin.zip $MINGW_BIN_SHA256
Secure-Download $MERCURIAL_WHEEL_URL ${prefix}\assets\${MERCURIAL_WHEEL_FILENAME} $MERCURIAL_WHEEL_SHA256 Secure-Download $MERCURIAL_WHEEL_URL ${prefix}\assets\${MERCURIAL_WHEEL_FILENAME} $MERCURIAL_WHEEL_SHA256
Secure-Download $RUSTUP_INIT_URL ${prefix}\assets\rustup-init.exe $RUSTUP_INIT_SHA256 Secure-Download $RUSTUP_INIT_URL ${prefix}\assets\rustup-init.exe $RUSTUP_INIT_SHA256
Secure-Download $PYOXIDIZER_URL ${prefix}\assets\PyOxidizer.msi $PYOXIDIZER_SHA256
Write-Output "installing Python 2.7 32-bit" Write-Output "installing Python 2.7 32-bit"
Invoke-Process msiexec.exe "/i ${prefix}\assets\python27-x86.msi /l* ${prefix}\assets\python27-x86.log /q TARGETDIR=${prefix}\python27-x86 ALLUSERS=" Invoke-Process msiexec.exe "/i ${prefix}\assets\python27-x86.msi /l* ${prefix}\assets\python27-x86.log /q TARGETDIR=${prefix}\python27-x86 ALLUSERS="
Invoke-Process ${prefix}\python27-x86\python.exe ${prefix}\assets\get-pip.py Invoke-Process ${prefix}\python27-x86\python.exe ${prefix}\assets\get-pip.py
Invoke-Process ${prefix}\python27-x86\Scripts\pip.exe "install ${prefix}\assets\virtualenv.tar.gz" Invoke-Process ${prefix}\python27-x86\Scripts\pip.exe "install ${prefix}\assets\virtualenv.tar.gz"
Write-Output "installing Python 2.7 64-bit" Write-Output "installing Python 2.7 64-bit"
Invoke-Process msiexec.exe "/i ${prefix}\assets\python27-x64.msi /l* ${prefix}\assets\python27-x64.log /q TARGETDIR=${prefix}\python27-x64 ALLUSERS=" Invoke-Process msiexec.exe "/i ${prefix}\assets\python27-x64.msi /l* ${prefix}\assets\python27-x64.log /q TARGETDIR=${prefix}\python27-x64 ALLUSERS="
Invoke-Process ${prefix}\python27-x64\python.exe ${prefix}\assets\get-pip.py Invoke-Process ${prefix}\python27-x64\python.exe ${prefix}\assets\get-pip.py
Invoke-Process ${prefix}\python27-x64\Scripts\pip.exe "install ${prefix}\assets\virtualenv.tar.gz" Invoke-Process ${prefix}\python27-x64\Scripts\pip.exe "install ${prefix}\assets\virtualenv.tar.gz"
Install-Python3 "Python 3.7 32-bit" ${prefix}\assets\python37-x86.exe ${prefix}\python37-x86 ${pip} Install-Python3 "Python 3.7 32-bit" ${prefix}\assets\python37-x86.exe ${prefix}\python37-x86 ${pip}
Install-Python3 "Python 3.7 64-bit" ${prefix}\assets\python37-x64.exe ${prefix}\python37-x64 ${pip} Install-Python3 "Python 3.7 64-bit" ${prefix}\assets\python37-x64.exe ${prefix}\python37-x64 ${pip}
Install-Python3 "Python 3.8 32-bit" ${prefix}\assets\python38-x86.exe ${prefix}\python38-x86 ${pip} Install-Python3 "Python 3.8 32-bit" ${prefix}\assets\python38-x86.exe ${prefix}\python38-x86 ${pip}
Install-Python3 "Python 3.8 64-bit" ${prefix}\assets\python38-x64.exe ${prefix}\python38-x64 ${pip} Install-Python3 "Python 3.8 64-bit" ${prefix}\assets\python38-x64.exe ${prefix}\python38-x64 ${pip}
Install-Python3 "Python 3.9 32-bit" ${prefix}\assets\python39-x86.exe ${prefix}\python39-x86 ${pip} Install-Python3 "Python 3.9 32-bit" ${prefix}\assets\python39-x86.exe ${prefix}\python39-x86 ${pip}
Install-Python3 "Python 3.9 64-bit" ${prefix}\assets\python39-x64.exe ${prefix}\python39-x64 ${pip} Install-Python3 "Python 3.9 64-bit" ${prefix}\assets\python39-x64.exe ${prefix}\python39-x64 ${pip}
Write-Output "installing Visual Studio 2017 Build Tools and SDKs" Write-Output "installing Visual Studio 2017 Build Tools and SDKs"
Invoke-Process ${prefix}\assets\vs_buildtools.exe "--quiet --wait --norestart --nocache --channelUri https://aka.ms/vs/15/release/channel --add Microsoft.VisualStudio.Workload.MSBuildTools --add Microsoft.VisualStudio.Component.Windows10SDK.17763 --add Microsoft.VisualStudio.Workload.VCTools --add Microsoft.VisualStudio.Component.Windows10SDK --add Microsoft.VisualStudio.Component.VC.140" Invoke-Process ${prefix}\assets\vs_buildtools.exe "--quiet --wait --norestart --nocache --channelUri https://aka.ms/vs/15/release/channel --add Microsoft.VisualStudio.Workload.MSBuildTools --add Microsoft.VisualStudio.Component.Windows10SDK.17763 --add Microsoft.VisualStudio.Workload.VCTools --add Microsoft.VisualStudio.Component.Windows10SDK --add Microsoft.VisualStudio.Component.VC.140"
Write-Output "installing PyOxidizer"
Invoke-Process msiexec.exe "/i ${prefix}\assets\PyOxidizer.msi /l* ${prefix}\assets\PyOxidizer.log /quiet"
Install-Rust ${prefix} Install-Rust ${prefix}
Write-Output "installing Visual C++ 9.0 for Python 2.7" Write-Output "installing Visual C++ 9.0 for Python 2.7"
Invoke-Process msiexec.exe "/i ${prefix}\assets\VCForPython27.msi /l* ${prefix}\assets\VCForPython27.log /q" Invoke-Process msiexec.exe "/i ${prefix}\assets\VCForPython27.msi /l* ${prefix}\assets\VCForPython27.log /q"
Write-Output "installing Inno Setup" Write-Output "installing Inno Setup"
Invoke-Process ${prefix}\assets\InnoSetup.exe "/SP- /VERYSILENT /SUPPRESSMSGBOXES" Invoke-Process ${prefix}\assets\InnoSetup.exe "/SP- /VERYSILENT /SUPPRESSMSGBOXES"