( )⚙ D6846 packaging: script the building of a MacOS installer using a custom python

This is an archive of the discontinued Mercurial Phabricator instance.

packaging: script the building of a MacOS installer using a custom python
Changes PlannedPublic

Authored by mharbison72 on Sep 12 2019, 6:21 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Summary

The intent here is to get away from relying on system python (which is going
away in 10.16), so that an installer can easily be built to work on multiple
versions of the OS. 10.9 was chosen as the minimum platform here because that's
the SDK needed to notarize, and also what the binary python installer targets.
A lot of this was adapted from the script that builds the TortoiseHg DMG.

There's a useful thg customization (not duplicated here), which extends
sys.path to include $HOME/Library/Python/2.7/lib/python/site-packages, because
there's no way to add to the bundled python after the app is signed. As it
stands now, this installation won't see that, but does see
$HOME/.local/lib/python2.7/site-packages, and the bundled pip will install
there with --user. It does seem like only seeing the second --user location
will get confusing, especially since the current behavior is that
pip install --user ... just works for pip in PATH.

The prerequisite is to have python3 to run the script, and a working python2
with docutils installed to feed to the script. (The latter is used to build and
install the documentation for Mercurial without polluting the python distro to
be packaged, so maybe we can use python3 there.)

The general flow is to download, build and install OpenSSL off to the side, and
then use that to download, build and install Python to a separate staging area.
The certifi package is bundled for the root certificates. I'm not an expert in
what OpenSSL compilation options should be used, so if there's a way to tighten
things up, let me know.

I don't like how the other packaging scripts assume an existing path on the
system, so this uses the build directory to stage the Python and OpenSSL builds.
The existing system python installer stages in build/mercurial, but this
builds Mercurial into the staged python directory. (That path includes version
info, so it shouldn't be much to add python3 support and build both at the same
time.) I also wasn't sure where to stash the downloads (they seem somewhat
useful to cache), so I stuffed them into the packages directory (which gets
clobbered by make clean).

To distinguish from the system python installer (and with an eye towards
python3), -py2.7 is added after the Mercurial version to the file name.
Altogether, this takes about 10 minutes to build on a 2018 Mac Mini.

There's additional followup work to do, in addition to the obvious py3 support-
the readme still mentions depending on system python, and this could probably be
driven by the makefile as a separate target. It should be trivial to sign the
binaries and installer, and slightly less trivial to notarize the *.pkg, but
definitely doable. The files in the final *.pkg can be listed like so:

pkgutil --payload-files dist/Mercurial-*.pkg

Looking at that, there's probably a bunch of junk that can be removed:

  • include/
  • distutils/tests/
  • lib-tk/test/
  • ctypes/test/,
  • unittest
  • test
  • lib2to3
  • idle
  • pydoc

I didn't see config options for that, and maybe it isn't worth it. The
installer ends up around 39MB.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

mharbison72 created this revision.Sep 12 2019, 6:21 PM

This seems to work (though the shebang line hack is painful)- until the original python build directory is deleted. Then a lot of things complain about unsupported hash type for md5 and sha{1,224,256,384,512}. Other modules like json can be imported with the installed python executable. I grepped around for the build directory, and it is in a bunch of *.so files (though this could be __FILE__ for all I know). Any ideas?

This seems to work (though the shebang line hack is painful)- until the original python build directory is deleted. Then a lot of things complain about unsupported hash type for md5 and sha{1,224,256,384,512}. Other modules like json can be imported with the installed python executable. I grepped around for the build directory, and it is in a bunch of *.so files (though this could be __FILE__ for all I know). Any ideas?

So, the problem here is that openssl libraries are being built and installed with the temporary install path, and the _ssl.so module is expecting to find them there. I tried a custom Setup.local that statically links _ssl.so against openssl before configuring python, but the build summary said dependencies were missing to build _ssl (among other things). For some reason, the resulting build still seemed to be able to talk to https servers. help(_ssl) said it was builtin, whereas help(_ssl) on the original build mentioned openssl (IIRC).

The _ssl.so module in the thg app has @executable_path in the library name, so that's why that works. I also tried messing with @rpath, but I can't get that working either. I'm not sure what else to try.

What's the latest with this diff ? It seems to still be a work in progress. Should it be in the "need-review" state?

What's the latest with this diff ? It seems to still be a work in progress. Should it be in the "need-review" state?

The hangup here is that the installed code seems to want to look in the path where it was built to load libraries. I have no idea why. It would be good if somebody figured out why, because I have a similar problem with the thg build on Mac. (But unlike here, that's not fatal because it falls back to a library inside the bundle.)

My thought with this was to be able to build an official py2 installer that worked on more than the very latest platform. But I also thought that we would hang on to py2 for awhile. If we're not, then this probably isn't worth pursuing, assuming PyOxidizer works out on OS X (so far, so good). I'm not so excited about dropping py2 because of the amount of work still needed on hg on Windows, thg on Windows and Mac, and the packaging on both. But 10.16 will remove all builtin python, so something will probably need to be done one way or the other by September or so.

are we talking about python library or compiled library ? Go you have any "empty" item in the various PATH used for the build ?

are we talking about python library or compiled library ? Go you have any "empty" item in the various PATH used for the build ?

See https://phab.mercurial-scm.org/D6846#100512.

I'm not sure what you mean about empty items. The build directory is out of the way, not on PATH. It's just that _ssl.so wants to load OpenSSL libraries from the path where they were built. It actually works fine after installing- until you do a make clean, which is how I noticed the problem.

macOS supports a @loader_path and related magic tokens in rpath to load libraries relative to the current binary. See e.g. https://blogs.oracle.com/dipol/dynamic-libraries,-rpath,-and-mac-os and https://medium.com/@donblas/fun-with-rpath-otool-and-install-name-tool-e3e41ae86172 for examples.

indygreg added inline comments.Feb 11 2020, 9:56 PM
contrib/packaging/hgpackaging/downloads.py
29

1.0.2u is out and should be used.

contrib/packaging/hgpackaging/python.py
54

Where did you get this line and enable-cms from?

120

I _think_ we also want --enable-lto for more speed wins.

contrib/packaging/macosx/build.py
170

fp.truncate()?

macOS supports a @loader_path and related magic tokens in rpath to load libraries relative to the current binary. See e.g. https://blogs.oracle.com/dipol/dynamic-libraries,-rpath,-and-mac-os and https://medium.com/@donblas/fun-with-rpath-otool-and-install-name-tool-e3e41ae86172 for examples.

I saw those, thanks. I guess what's confusing is I don't see RPATH being configured when building OpenSSL for the TortoiseHg build, nor do I see it being overwritten when running py2app. I also interpreted this to mean that is should be automatic on Mac, but I guess not:

https://wiki.openssl.org/index.php/Compilation_and_Installation#Using_RPATHs

contrib/packaging/hgpackaging/python.py
54
contrib/packaging/macosx/build.py
170

The original shebang line was much longer, so after writing it out with the shorter one, I think this was meant to trim the file. too the proper content.

What's the status of this ? @mharbison72 id the linking problem got solved ?

What's the status of this ? @mharbison72 id the linking problem got solved ?

I've been too busy to get back to this, and probably won't for the short term, anyway.

It would probably also be a good to have a general idea when we drop py2 completely. I'd be content leaving py2 around for awhile for various reasons, but I don't want to sink a bunch of time into something that lasts for one release. (I think a PyOxidizer build would end up being a lot different looking than this.)

So as I understand this is not ready to land and should be moved to change-requested (or change planned), right ?

mharbison72 planned changes to this revision.Apr 17 2020, 2:56 PM

So as I understand this is not ready to land and should be moved to change-requested (or change planned), right ?

Yeah, I'll move this out of the way, and can abandon it when PyOxidizer builds are in place.