( )⚙ D10680 contrib: install PyOxidizer 0.16.0

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10680

contrib: install PyOxidizer 0.16.0
ClosedPublic

Authored by indygreg on May 6 2021, 8:00 PM.

Details

Reviewers
mharbison72
Group Reviewers
hg-reviewers
Commits
rHG0de5eefbe4b2: contrib: install PyOxidizer 0.16.0
Summary

PyOxidizer now provides MSI installers and pre-built Linux binaries.
So we install that way. This significantly reduces the time to bootstrap
a new machine in automation, as building PyOxidizer from source on a low
core count machine takes several minutes.

This change temporarily breaks the ability of the automated environment
to use the in-repo pyoxidizer.bzl configuration file, as there are
backwards-incompatible changes with the upgrade that need to be
reflected. We'll handle those in subsequent commits.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

indygreg created this revision.May 6 2021, 8:00 PM
indygreg edited the summary of this revision. (Show Details)May 6 2021, 9:38 PM
indygreg updated this revision to Diff 27732.
baymax updated this revision to Diff 27751.May 7 2021, 11:30 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

indygreg edited the summary of this revision. (Show Details)May 7 2021, 12:42 PM
indygreg updated this revision to Diff 27762.
indygreg planned changes to this revision.May 7 2021, 8:38 PM

I'll need to release a new version of PyOxidizer to fix wonky behavior due to interaction of file mtimes and pip (https://github.com/indygreg/PyOxidizer/commit/a0bdde179aaed8521d3a668e518554814bcffdba).

indygreg retitled this revision from contrib: install PyOxidizer 0.15.0 to contrib: install PyOxidizer 0.16.0.May 9 2021, 1:08 PM
indygreg edited the summary of this revision. (Show Details)
indygreg updated this revision to Diff 27769.
baymax updated this revision to Diff 27788.May 10 2021, 11:37 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

mharbison72 accepted this revision.May 10 2021, 6:35 PM
This revision is now accepted and ready to land.May 10 2021, 6:36 PM
indygreg added a commit: rHG0de5eefbe4b2: contrib: install PyOxidizer 0.16.0.May 15 2021, 7:23 PM
Closed by commit rHG0de5eefbe4b2: contrib: install PyOxidizer 0.16.0 (authored by indygreg). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 27751

contrib/automation/hgautomation/linux.py

${PYENV_ROOT}/versions/${v}/bin/pip install -r /hgdev/${REQUIREMENTS} ${PYENV_ROOT}/versions/${v}/bin/pip install -r /hgdev/${REQUIREMENTS}
done done
pyenv global ${PYENV2_VERSIONS} ${PYENV3_VERSIONS} system pyenv global ${PYENV2_VERSIONS} ${PYENV3_VERSIONS} system
'''.lstrip().replace( '''.lstrip().replace(
'\r\n', '\n' '\r\n', '\n'
) )
INSTALL_PYOXIDIZER = r'''
PYOXIDIZER_VERSION=0.15.0
PYOXIDIZER_SHA256=396c9b271174b108dcd9ec1f8b9b93839930890b41a859e7945948bcec57fae9
PYOXIDIZER_URL=https://github.com/indygreg/PyOxidizer/releases/download/pyoxidizer%2F${PYOXIDIZER_VERSION}/pyoxidizer-${PYOXIDIZER_VERSION}-linux_x86_64.zip
wget -O pyoxidizer.zip --progress dot:mega ${PYOXIDIZER_URL}
echo "${PYOXIDIZER_SHA256} pyoxidizer.zip" | sha256sum --check -
unzip pyoxidizer.zip
chmod +x pyoxidizer
sudo mv pyoxidizer /usr/local/bin/pyoxidizer
'''
INSTALL_RUST = r''' INSTALL_RUST = r'''
RUSTUP_INIT_SHA256=a46fe67199b7bcbbde2dcbc23ae08db6f29883e260e23899a88b9073effc9076 RUSTUP_INIT_SHA256=a46fe67199b7bcbbde2dcbc23ae08db6f29883e260e23899a88b9073effc9076
wget -O rustup-init --progress dot:mega https://static.rust-lang.org/rustup/archive/1.18.3/x86_64-unknown-linux-gnu/rustup-init wget -O rustup-init --progress dot:mega https://static.rust-lang.org/rustup/archive/1.18.3/x86_64-unknown-linux-gnu/rustup-init
echo "${RUSTUP_INIT_SHA256} rustup-init" | sha256sum --check - echo "${RUSTUP_INIT_SHA256} rustup-init" | sha256sum --check -
chmod +x rustup-init chmod +x rustup-init
sudo -H -u hg -g hg ./rustup-init -y sudo -H -u hg -g hg ./rustup-init -y
sudo -H -u hg -g hg /home/hg/.cargo/bin/rustup install 1.41.1 1.52.0 sudo -H -u hg -g hg /home/hg/.cargo/bin/rustup install 1.41.1 1.52.0
sudo -H -u hg -g hg /home/hg/.cargo/bin/rustup component add clippy sudo -H -u hg -g hg /home/hg/.cargo/bin/rustup component add clippy
sudo -H -u hg -g hg /home/hg/.cargo/bin/cargo install --version 0.10.3 pyoxidizer
''' '''
BOOTSTRAP_VIRTUALENV = r''' BOOTSTRAP_VIRTUALENV = r'''
/usr/bin/virtualenv /hgdev/venv-bootstrap /usr/bin/virtualenv /hgdev/venv-bootstrap
HG_SHA256=35fc8ba5e0379c1b3affa2757e83fb0509e8ac314cbd9f1fd133cf265d16e49f HG_SHA256=35fc8ba5e0379c1b3affa2757e83fb0509e8ac314cbd9f1fd133cf265d16e49f
HG_TARBALL=mercurial-5.1.1.tar.gz HG_TARBALL=mercurial-5.1.1.tar.gz
sudo update-alternatives --install /usr/bin/clang-format clang-format \ sudo update-alternatives --install /usr/bin/clang-format clang-format \
/usr/bin/clang-format-6.0 1000 /usr/bin/clang-format-6.0 1000
sudo mkdir /hgdev sudo mkdir /hgdev
# Will be normalized to hg:hg later. # Will be normalized to hg:hg later.
sudo chown `whoami` /hgdev sudo chown `whoami` /hgdev
{install_rust} {install_rust}
{install_pyoxidizer}
cp requirements-*.txt /hgdev/ cp requirements-*.txt /hgdev/
# Disable the pip version check because it uses the network and can # Disable the pip version check because it uses the network and can
# be annoying. # be annoying.
cat << EOF | sudo tee -a /etc/pip.conf cat << EOF | sudo tee -a /etc/pip.conf
[global] [global]
disable-pip-version-check = True disable-pip-version-check = True
[phases] [phases]
publish = false publish = false
EOF EOF
sudo chown -R hg:hg /hgdev sudo chown -R hg:hg /hgdev
'''.lstrip() '''.lstrip()
.format( .format(
install_rust=INSTALL_RUST, install_rust=INSTALL_RUST,
install_pyoxidizer=INSTALL_PYOXIDIZER,
install_pythons=INSTALL_PYTHONS, install_pythons=INSTALL_PYTHONS,
bootstrap_virtualenv=BOOTSTRAP_VIRTUALENV, bootstrap_virtualenv=BOOTSTRAP_VIRTUALENV,
) )
.replace('\r\n', '\n') .replace('\r\n', '\n')
) )
# Prepares /hgdev for operations. # Prepares /hgdev for operations.

contrib/install-windows-dependencies.ps1

$MERCURIAL_WHEEL_FILENAME = "mercurial-5.1.2-cp27-cp27m-win_amd64.whl" $MERCURIAL_WHEEL_FILENAME = "mercurial-5.1.2-cp27-cp27m-win_amd64.whl"
$MERCURIAL_WHEEL_URL = "https://files.pythonhosted.org/packages/6d/47/e031e47f7fe9b16e4e3383da47e2b0a7eae6e603996bc67a03ec4fa1b3f4/$MERCURIAL_WHEEL_FILENAME" $MERCURIAL_WHEEL_URL = "https://files.pythonhosted.org/packages/6d/47/e031e47f7fe9b16e4e3383da47e2b0a7eae6e603996bc67a03ec4fa1b3f4/$MERCURIAL_WHEEL_FILENAME"
$MERCURIAL_WHEEL_SHA256 = "1d18c7f6ca1456f0f62ee65c9a50c14cbba48ce6e924930cdb10537f5c9eaf5f" $MERCURIAL_WHEEL_SHA256 = "1d18c7f6ca1456f0f62ee65c9a50c14cbba48ce6e924930cdb10537f5c9eaf5f"
$RUSTUP_INIT_URL = "https://static.rust-lang.org/rustup/archive/1.21.1/x86_64-pc-windows-gnu/rustup-init.exe" $RUSTUP_INIT_URL = "https://static.rust-lang.org/rustup/archive/1.21.1/x86_64-pc-windows-gnu/rustup-init.exe"
$RUSTUP_INIT_SHA256 = "d17df34ba974b9b19cf5c75883a95475aa22ddc364591d75d174090d55711c72" $RUSTUP_INIT_SHA256 = "d17df34ba974b9b19cf5c75883a95475aa22ddc364591d75d174090d55711c72"
$PYOXIDIZER_URL = "https://github.com/indygreg/PyOxidizer/releases/download/pyoxidizer%2F0.15.0/PyOxidizer-0.15.0-x64.msi"
$PYOXIDIZER_SHA256 = "c866b70431fc5a2690937e265e31834d4be6127c8c8315d1f1412c792cbc29f1"
# Writing progress slows down downloads substantially. So disable it. # Writing progress slows down downloads substantially. So disable it.
$progressPreference = 'silentlyContinue' $progressPreference = 'silentlyContinue'
function Secure-Download($url, $path, $sha256) { function Secure-Download($url, $path, $sha256) {
if (Test-Path -Path $path) { if (Test-Path -Path $path) {
Get-FileHash -Path $path -Algorithm SHA256 -OutVariable hash Get-FileHash -Path $path -Algorithm SHA256 -OutVariable hash
if ($hash.Hash -eq $sha256) { if ($hash.Hash -eq $sha256) {
Write-Output "installing Rust" Write-Output "installing Rust"
$Env:RUSTUP_HOME = "${prefix}\rustup" $Env:RUSTUP_HOME = "${prefix}\rustup"
$Env:CARGO_HOME = "${prefix}\cargo" $Env:CARGO_HOME = "${prefix}\cargo"
Invoke-Process "${prefix}\assets\rustup-init.exe" "-y --default-host x86_64-pc-windows-msvc" Invoke-Process "${prefix}\assets\rustup-init.exe" "-y --default-host x86_64-pc-windows-msvc"
Invoke-Process "${prefix}\cargo\bin\rustup.exe" "target add i686-pc-windows-msvc" Invoke-Process "${prefix}\cargo\bin\rustup.exe" "target add i686-pc-windows-msvc"
Invoke-Process "${prefix}\cargo\bin\rustup.exe" "install 1.52.0" Invoke-Process "${prefix}\cargo\bin\rustup.exe" "install 1.52.0"
Invoke-Process "${prefix}\cargo\bin\rustup.exe" "component add clippy" Invoke-Process "${prefix}\cargo\bin\rustup.exe" "component add clippy"
# Install PyOxidizer for packaging.
Invoke-Process "${prefix}\cargo\bin\cargo.exe" "install --version 0.10.3 pyoxidizer"
} }
function Install-Dependencies($prefix) { function Install-Dependencies($prefix) {
if (!(Test-Path -Path $prefix\assets)) { if (!(Test-Path -Path $prefix\assets)) {
New-Item -Path $prefix\assets -ItemType Directory New-Item -Path $prefix\assets -ItemType Directory
} }
$pip = "${prefix}\assets\get-pip.py" $pip = "${prefix}\assets\get-pip.py"
Secure-Download $PYTHON39_x64_URL ${prefix}\assets\python39-x64.exe $PYTHON39_x64_SHA256 Secure-Download $PYTHON39_x64_URL ${prefix}\assets\python39-x64.exe $PYTHON39_x64_SHA256
Secure-Download $PIP_URL ${pip} $PIP_SHA256 Secure-Download $PIP_URL ${pip} $PIP_SHA256
Secure-Download $VIRTUALENV_URL ${prefix}\assets\virtualenv.tar.gz $VIRTUALENV_SHA256 Secure-Download $VIRTUALENV_URL ${prefix}\assets\virtualenv.tar.gz $VIRTUALENV_SHA256
Secure-Download $VS_BUILD_TOOLS_URL ${prefix}\assets\vs_buildtools.exe $VS_BUILD_TOOLS_SHA256 Secure-Download $VS_BUILD_TOOLS_URL ${prefix}\assets\vs_buildtools.exe $VS_BUILD_TOOLS_SHA256
Secure-Download $INNO_SETUP_URL ${prefix}\assets\InnoSetup.exe $INNO_SETUP_SHA256 Secure-Download $INNO_SETUP_URL ${prefix}\assets\InnoSetup.exe $INNO_SETUP_SHA256
Secure-Download $MINGW_BIN_URL ${prefix}\assets\mingw-get-bin.zip $MINGW_BIN_SHA256 Secure-Download $MINGW_BIN_URL ${prefix}\assets\mingw-get-bin.zip $MINGW_BIN_SHA256
Secure-Download $MERCURIAL_WHEEL_URL ${prefix}\assets\${MERCURIAL_WHEEL_FILENAME} $MERCURIAL_WHEEL_SHA256 Secure-Download $MERCURIAL_WHEEL_URL ${prefix}\assets\${MERCURIAL_WHEEL_FILENAME} $MERCURIAL_WHEEL_SHA256
Secure-Download $RUSTUP_INIT_URL ${prefix}\assets\rustup-init.exe $RUSTUP_INIT_SHA256 Secure-Download $RUSTUP_INIT_URL ${prefix}\assets\rustup-init.exe $RUSTUP_INIT_SHA256
Secure-Download $PYOXIDIZER_URL ${prefix}\assets\PyOxidizer.msi $PYOXIDIZER_SHA256
Write-Output "installing Python 2.7 32-bit" Write-Output "installing Python 2.7 32-bit"
Invoke-Process msiexec.exe "/i ${prefix}\assets\python27-x86.msi /l* ${prefix}\assets\python27-x86.log /q TARGETDIR=${prefix}\python27-x86 ALLUSERS=" Invoke-Process msiexec.exe "/i ${prefix}\assets\python27-x86.msi /l* ${prefix}\assets\python27-x86.log /q TARGETDIR=${prefix}\python27-x86 ALLUSERS="
Invoke-Process ${prefix}\python27-x86\python.exe ${prefix}\assets\get-pip.py Invoke-Process ${prefix}\python27-x86\python.exe ${prefix}\assets\get-pip.py
Invoke-Process ${prefix}\python27-x86\Scripts\pip.exe "install ${prefix}\assets\virtualenv.tar.gz" Invoke-Process ${prefix}\python27-x86\Scripts\pip.exe "install ${prefix}\assets\virtualenv.tar.gz"
Write-Output "installing Python 2.7 64-bit" Write-Output "installing Python 2.7 64-bit"
Invoke-Process msiexec.exe "/i ${prefix}\assets\python27-x64.msi /l* ${prefix}\assets\python27-x64.log /q TARGETDIR=${prefix}\python27-x64 ALLUSERS=" Invoke-Process msiexec.exe "/i ${prefix}\assets\python27-x64.msi /l* ${prefix}\assets\python27-x64.log /q TARGETDIR=${prefix}\python27-x64 ALLUSERS="
Invoke-Process ${prefix}\python27-x64\python.exe ${prefix}\assets\get-pip.py Invoke-Process ${prefix}\python27-x64\python.exe ${prefix}\assets\get-pip.py
Invoke-Process ${prefix}\python27-x64\Scripts\pip.exe "install ${prefix}\assets\virtualenv.tar.gz" Invoke-Process ${prefix}\python27-x64\Scripts\pip.exe "install ${prefix}\assets\virtualenv.tar.gz"
Install-Python3 "Python 3.7 32-bit" ${prefix}\assets\python37-x86.exe ${prefix}\python37-x86 ${pip} Install-Python3 "Python 3.7 32-bit" ${prefix}\assets\python37-x86.exe ${prefix}\python37-x86 ${pip}
Install-Python3 "Python 3.7 64-bit" ${prefix}\assets\python37-x64.exe ${prefix}\python37-x64 ${pip} Install-Python3 "Python 3.7 64-bit" ${prefix}\assets\python37-x64.exe ${prefix}\python37-x64 ${pip}
Install-Python3 "Python 3.8 32-bit" ${prefix}\assets\python38-x86.exe ${prefix}\python38-x86 ${pip} Install-Python3 "Python 3.8 32-bit" ${prefix}\assets\python38-x86.exe ${prefix}\python38-x86 ${pip}
Install-Python3 "Python 3.8 64-bit" ${prefix}\assets\python38-x64.exe ${prefix}\python38-x64 ${pip} Install-Python3 "Python 3.8 64-bit" ${prefix}\assets\python38-x64.exe ${prefix}\python38-x64 ${pip}
Install-Python3 "Python 3.9 32-bit" ${prefix}\assets\python39-x86.exe ${prefix}\python39-x86 ${pip} Install-Python3 "Python 3.9 32-bit" ${prefix}\assets\python39-x86.exe ${prefix}\python39-x86 ${pip}
Install-Python3 "Python 3.9 64-bit" ${prefix}\assets\python39-x64.exe ${prefix}\python39-x64 ${pip} Install-Python3 "Python 3.9 64-bit" ${prefix}\assets\python39-x64.exe ${prefix}\python39-x64 ${pip}
Write-Output "installing Visual Studio 2019 Build Tools and SDKs" Write-Output "installing Visual Studio 2019 Build Tools and SDKs"
Invoke-Process ${prefix}\assets\vs_buildtools.exe "--quiet --wait --norestart --nocache --channelUri https://aka.ms/vs/16/release/channel --add Microsoft.VisualStudio.Workload.MSBuildTools --add Microsoft.VisualStudio.Component.Windows10SDK.19041 --add Microsoft.VisualStudio.Workload.VCTools --add Microsoft.VisualStudio.Component.Windows10SDK --add Microsoft.VisualStudio.Component.VC.v141.x86.x64" Invoke-Process ${prefix}\assets\vs_buildtools.exe "--quiet --wait --norestart --nocache --channelUri https://aka.ms/vs/16/release/channel --add Microsoft.VisualStudio.Workload.MSBuildTools --add Microsoft.VisualStudio.Component.Windows10SDK.19041 --add Microsoft.VisualStudio.Workload.VCTools --add Microsoft.VisualStudio.Component.Windows10SDK --add Microsoft.VisualStudio.Component.VC.v141.x86.x64"
Write-Output "installing PyOxidizer"
Invoke-Process msiexec.exe "/i ${prefix}\assets\PyOxidizer.msi /l* ${prefix}\assets\PyOxidizer.log /quiet"
Install-Rust ${prefix} Install-Rust ${prefix}
Write-Output "installing Visual C++ 9.0 for Python 2.7" Write-Output "installing Visual C++ 9.0 for Python 2.7"
Invoke-Process msiexec.exe "/i ${prefix}\assets\VCForPython27.msi /l* ${prefix}\assets\VCForPython27.log /q" Invoke-Process msiexec.exe "/i ${prefix}\assets\VCForPython27.msi /l* ${prefix}\assets\VCForPython27.log /q"
Write-Output "installing Inno Setup" Write-Output "installing Inno Setup"
Invoke-Process ${prefix}\assets\InnoSetup.exe "/SP- /VERYSILENT /SUPPRESSMSGBOXES" Invoke-Process ${prefix}\assets\InnoSetup.exe "/SP- /VERYSILENT /SUPPRESSMSGBOXES"