This is an archive of the discontinued Mercurial Phabricator instance.

Differential D6754

contrib: proof of concept script to build Mac packages without system python
AbandonedPublic

Authored by mharbison72 on Aug 20 2019, 6:34 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Summary

This isn't close to being ready- I'm just looking for ideas before I get too far
off into the weeds. It will build a *.pkg that does install, and both
hg version and hg debuginstall look healthy. As noted in the script though,
there are still references to /Library/Frameworks/Python.framework (which is the
python.org installed stuff), so it may not work on a fresh system.

At this point, I'm wondering if we should just curl the source for OpenSSL and
Python, and build the non-framework version with the destination set to the
proper directory. The packaging for thg does this (though it probably doesn't
have to), and it takes about 10 minutes on a Mac Mini. 5 Minutes of that is
running the tests on the python build, so maybe there's a way to shut that off.

As far as alternatives, I started down the path of using virtualenv here, and
shipping that. I'm skeptical of the experimental nature of --relocatable, and
grepping on the tree showed the build path in the *.pyc files. I have no idea
if that matters. I left pip in place so that things like evolve could be
installed, but it errored out when specifying --user. I didn't look into why.

I also looked at pyinstaller. I couldn't figure out how to convince it to
bundle the C extensions, and it appears to append its own output when the hg
script exits with a non zero value. I briefly looked at bbfreeze and cx_freeze,
but didn't get as far. I've had enough frustration with py2exe and py2app that
I'm not wanting to struggle with yet another packager.

Ultimately I think we should keep the target for building against system python
(there was some special functionality unlocked with a specifically crafted dummy
cert), and this should possibly be a different target. (I don't see a way to
switch between the two by simply overriding $PYTHON the way this is going.) I
basically copy/pasted the makefile as a starting point, but moved it into a
shell script so the environment could be activated. This either needs to be
moved to contrib/, or folded back into the makefile. I kinda lean towards the
former, and have that called by make. The readme in the installer will also
have to be conditionalize/updated, since it mentions relying on system python.

It would be nice to open up the range of supported platforms back to say, 10.9.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

mharbison72 created this revision.Aug 20 2019, 6:34 PM
indygreg added a subscriber: indygreg.Aug 25 2019, 12:18 PM

This looks promising.

I don't want to tell you not to work on this, but my bold plan is to get Mercurial using PyOxidizer and leaning on PyOxidizer for packaging. This will require shipping a Python 3 Mercurial, however.

Until PyOxidizer is ready, you may want to look into Beeware's Briefcase tool for packaging macOS applications. It also provides its own self-contained Python distribution. I'm not sure if it works with Python 2.7, however.

You may also be interested in https://github.com/indygreg/python-build-standalone for self-contained Python distributions. Only works with Python 3.7+ at the moment now, though. The pre-built distributions on GitHub having a working Python install in them. All the Python C extensions are statically linked into the python executable.

make_osx.sh
16

We should be pinning versions and hashes when installing anything from the network. For reproducibility and security.

mharbison72 added a subscriber: durin42.Aug 25 2019, 5:11 PM
In D6754#99255, @indygreg wrote:

This looks promising.
I don't want to tell you not to work on this, but my bold plan is to get Mercurial using PyOxidizer and leaning on PyOxidizer for packaging. This will require shipping a Python 3 Mercurial, however.

Thanks for the pointers.

I'm broadly aware of the plans to use PyOxidizer for py3, and hope to use it for thg too. I started this with nothing more in mind than making the installer support more than 10.14. But since it links against the system python, setting the MACOS_DEPLOYMENT_TARGET (or whatever it's called) errors out. So I tried allowing PYTHON to be overridden in the makefile, so it could be pointed to the python.org install and go back to 10.9. But @durin42 locked it into the system python in da1848f07c6a, and things just kinda snowballed from there. He mentioned on IRC that a pyinstaller dependency wouldn't be great, and also maybe we don't want to be completely tied to PyOxidizer. (I took it as don't preclude other methods, but maybe that needs to be clarified.)

Until PyOxidizer is ready, you may want to look into Beeware's Briefcase tool for packaging macOS applications. It also provides its own self-contained Python distribution. I'm not sure if it works with Python 2.7, however.

There at least appears to be a package visible in pip27, but the docs talk about building a *.app, which isn't going to be useful for us.

You may also be interested in https://github.com/indygreg/python-build-standalone for self-contained Python distributions. Only works with Python 3.7+ at the moment now, though. The pre-built distributions on GitHub having a working Python install in them. All the Python C extensions are statically linked into the python executable.

I'll try to find time to play with this in the next week or so. I'm not sure what it would take to support py2, but I'd rather everybody poured their effort in elsewhere at this point. @durin42 mentioned on IRC that it's long past time to get off the system python, but the only real deadline is that it won't be in 10.16 a year from now. Hopefully the codebase is still supporting py2 then, but maybe the py3 support is good enough that we don't need an official py2 installer? End of py2 support doesn't seem like a big deal, unless some horrible TLS bug is found.

I suppose a reasonable fix for supporting more OS versions is to spin up a 10.9 VM and build there. I don't have one right now, but I do have a machine that is going to be setup to host various Mac VMs, and I don't have a problem building the installer on it if there's interest.

durin42 added a comment.Sep 9 2019, 11:51 AM

This is a start, but I suspect where we should end up with this is downloading and compiling the whole Python distribution, rather than relying on a virtualenv to be portable?

mharbison72 added a comment.Sep 9 2019, 9:01 PM
In D6754#100159, @durin42 wrote:

This is a start, but I suspect where we should end up with this is downloading and compiling the whole Python distribution, rather than relying on a virtualenv to be portable?

Agreed. I think we will need to build OpenSSL too, and bundle certifi?

The script for building thg makes it look easy, but the result I ended up with failed to support zlib and sqlite3. (We need the latter too, right?) Maybe it's an issue with non-framework builds, as that's the only difference I'm aware of from what thg build is doing.

mharbison72 abandoned this revision.Dec 28 2019, 12:55 PM

D6846 will download and build from source without relying on virtualenv stuff.

Revision Contents
Changeset List

PathPackages
AMmake_osx.sh (87 lines)

Diff 16288

make_osx.sh

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
#!/bin/sh
set -e
# Extra args passed in, not version
OSXVERSIONFLAGS=
# TODO: default MACOSX_DEPLOYMENT_TARGET=10.9
# TODO: adjust ${OSXVER} accordingly
rm -rf build/mercurial
python -m virtualenv build/mercurial/usr/local/mercurial
source build/mercurial/usr/local/mercurial/bin/activate
# TODO: Don't install this into the virtual environment. Also, use docutils-python3
python -m pip install docutils
indygregUnsubmitted
Not Done

We should be pinning versions and hashes when installing anything from the network. For reproducibility and security.

indygreg: We should be pinning versions and hashes when installing anything from the network. For…
python setup.py install --optimize=1 \
--root=build/mercurial/ --prefix=/usr/local/mercurial
make -C doc all install DESTDIR="`pwd`/build/mercurial/"
# Place a bogon .DS_Store file in the target dir so we can be
# sure it doesn't get included in the final package.
touch build/mercurial/.DS_Store
# install zsh completions - this location appears to be
# searched by default as of macOS Sierra.
install -d build/mercurial/usr/local/share/zsh/site-functions/
install -m 0644 contrib/zsh_completion build/mercurial/usr/local/share/zsh/site-functions/_hg
# install bash completions - there doesn't appear to be a
# place that's searched by default for bash, so we'll follow
# the lead of Apple's git install and just put it in a
# location of our own.
install -d build/mercurial/usr/local/hg/contrib/
install -m 0644 contrib/bash_completion build/mercurial/usr/local/hg/contrib/hg-completion.bash
make -C contrib/chg \
HGPATH=/usr/local/bin/hg \
PYTHON=python \
HGEXTDIR=/usr/local/mercurial/lib/python2.7/site-packages/hgext \
DESTDIR=../../build/mercurial \
PREFIX=/usr/local \
clean install
# Keep pip around so that things like evolve can be installed.
rm -f build/mercurial/usr/local/mercurial/bin/rst* \
build/mercurial/usr/local/mercurial/bin/activate* \
build/mercurial/usr/local/mercurial/bin/wheel \
build/mercurial/usr/local/mercurial/bin/easy_install
# LANG=C' to avoid "sed: RE error: illegal byte sequence"
LANG=C sed -i '' -e "s|`pwd`/build/mercurial||g" build/mercurial/usr/local/mercurial/bin/*
# The virtualenv contains a symlink back to the real python install. We can
# either slim this down somehow (to avoid distributing whatever random packages
# are already installed), or build python from source with a custom --dest.
mkdir -p build/mercurial/usr/local/mercurial/Library/Frameworks/Python.framework/Versions
cp -R /Library/Frameworks/Python.framework/Versions/2.7 build/mercurial/usr/local/mercurial/Library/Frameworks/Python.framework/Versions
ln -s -f /usr/local/mercurial/Python.framework/Versions/2.7/Python build/mercurial/usr/local/mercurial/.Python
# Running /usr/local/mercurial/Python.framework/Versions/2.7/bin/python2.7 and
# printing sys.paths lists out stuff in /Library/Frameworks/Python.framework,
# so this may not be totally detached from a standard python.org installation.
# sys.exec_prefix and sys.prefix also point there. That stuff appears to live
# in /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_sysconfigdata.py.
# The various *.pyc files that are copied over also have /Library/Frameworks strings
# in them. Then again, so do *.pyc files in a virtualenv.
ln -s /usr/local/mercurial/bin/hg build/mercurial/usr/local/bin/hg
mkdir -p ${OUTPUTDIR:-dist}
HGVER=$(python2 contrib/genosxversion.py ${OSXVERSIONFLAGS} \
build/mercurial/usr/local/mercurial/lib/python2.7/site-packages/mercurial/__version__.py) && \
OSXVER=$(sw_vers -productVersion | cut -d. -f1,2) && \
PYVER=$(python -V 2>&1 | cut -d' ' -f2 | cut -d. -f1,2) && \
pkgbuild --filter .DS_Store --root build/mercurial/ \
--identifier org.mercurial-scm.mercurial \
--version "${HGVER}" \
build/mercurial.pkg && \
productbuild --distribution contrib/packaging/macosx/distribution.xml \
--package-path build/ \
--version "${HGVER}" \
--resources contrib/packaging/macosx/ \
"${OUTPUTDIR:-dist/}"/Mercurial-"${HGVER}"-py"${PYVER}"-macosx"${OSXVER}".pkg