This is an archive of the discontinued Mercurial Phabricator instance.

Differential D3759

packaging: replace dockerlib.sh with a Python script
ClosedPublic

Authored by indygreg on Jun 16 2018, 2:19 PM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHGe5916f1236f3: packaging: replace dockerlib.sh with a Python script
Summary

I want to do some more advanced things with Docker in upcoming
commits. Trying to do that with shell scripts will be a bit too
painful for my liking. Implementing things in Python will be
vastly simpler in the long run.

This commit essentially ports dockerlib.sh to a Python script.
dockerdeb and dockerrpm have been ported to use the new hg-docker
script.

hg-docker requires Python 3. I've only tested on Python 3.5.

Unlike the local packaging scripts which may need to run on old
distros, the Docker packaging scripts don't have these constraints.
So I think it is acceptable to require Python 3.5.

As part of the transition, the Docker image tags changed slightly.
I don't think that's a big deal: the Docker image names are
effectively arbitrary and are a means to an end to achieve
running commands in Docker containers.

The code for resolving the Dockerfile content allows substituting
values passed as arguments. This will be used in a subsequent commit.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Jun 16 2018, 2:19 PM
durin42 accepted this revision.Jun 25 2018, 10:50 AM
This revision is now accepted and ready to land.Jun 25 2018, 10:50 AM
Closed by commit rHGe5916f1236f3: packaging: replace dockerlib.sh with a Python script (authored by indygreg). · Explain WhyJun 25 2018, 11:20 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 9285

contrib/packaging/dockerdeb

#!/bin/bash -eu #!/bin/bash -eu
. $(dirname $0)/dockerlib.sh
. $(dirname $0)/packagelib.sh . $(dirname $0)/packagelib.sh
BUILDDIR=$(dirname $0) BUILDDIR=$(dirname $0)
export ROOTDIR=$(cd $BUILDDIR/../.. > /dev/null; pwd) export ROOTDIR=$(cd $BUILDDIR/../.. > /dev/null; pwd)
checkdocker
DISTID="$1" DISTID="$1"
CODENAME="$2" CODENAME="$2"
PLATFORM="$1-$2" PLATFORM="$1-$2"
shift; shift # extra params are passed to build process shift; shift # extra params are passed to build process
OUTPUTDIR=${OUTPUTDIR:=$ROOTDIR/packages/$PLATFORM} OUTPUTDIR=${OUTPUTDIR:=$ROOTDIR/packages/$PLATFORM}
CONTAINER=hg-docker-$PLATFORM
DOCKER=$($BUILDDIR/hg-docker docker-path)
initcontainer $PLATFORM $BUILDDIR/hg-docker build $BUILDDIR/docker/$PLATFORM $CONTAINER
# debuild only appears to be able to save built debs etc to .., so we # debuild only appears to be able to save built debs etc to .., so we
# have to share the .. of the current directory with the docker # have to share the .. of the current directory with the docker
# container and hope it's writable. Whee. # container and hope it's writable. Whee.
dn=$(basename $ROOTDIR) dn=$(basename $ROOTDIR)
DBUILDUSER=build DBUILDUSER=build

contrib/packaging/dockerlib.sh

  • This file was deleted.
#!/bin/sh -eu
# This function exists to set up the DOCKER variable and verify that
# it's the binary we expect. It also verifies that the docker service
# is running on the system and we can talk to it.
function checkdocker() {
if which docker.io >> /dev/null 2>&1 ; then
DOCKER=docker.io
elif which docker >> /dev/null 2>&1 ; then
DOCKER=docker
else
echo "Error: docker must be installed"
exit 1
fi
$DOCKER -h 2> /dev/null | grep -q Jansens && { echo "Error: $DOCKER is the Docking System Tray - install docker.io instead"; exit 1; }
$DOCKER version | grep -Eq "^Client( version)?:" || { echo "Error: unexpected output from \"$DOCKER version\""; exit 1; }
$DOCKER version | grep -Eq "^Server( version)?:" || { echo "Error: could not get docker server version - check it is running and your permissions"; exit 1; }
}
# Construct a container and leave its name in $CONTAINER for future use.
function initcontainer() {
[ "$1" ] || { echo "Error: platform name must be specified"; exit 1; }
DFILE="$ROOTDIR/contrib/packaging/docker/$1"
[ -f "$DFILE" ] || { echo "Error: docker file $DFILE not found"; exit 1; }
CONTAINER="hg-dockerrpm-$1"
cat $DFILE | $DOCKER build --build-arg http_proxy --build-arg https_proxy --tag $CONTAINER -
}

contrib/packaging/dockerrpm

#!/bin/bash -e #!/bin/bash -e
. $(dirname $0)/dockerlib.sh
BUILDDIR=$(dirname $0) BUILDDIR=$(dirname $0)
export ROOTDIR=$(cd $BUILDDIR/../..; pwd) export ROOTDIR=$(cd $BUILDDIR/../..; pwd)
checkdocker
PLATFORM="$1" PLATFORM="$1"
shift # extra params are passed to buildrpm shift # extra params are passed to buildrpm
initcontainer $PLATFORM DOCKER=$($BUILDDIR/hg-docker docker-path)
CONTAINER=hg-docker-$PLATFORM
$BUILDDIR/hg-docker build $BUILDDIR/docker/$PLATFORM $CONTAINER
RPMBUILDDIR=$ROOTDIR/packages/$PLATFORM RPMBUILDDIR=$ROOTDIR/packages/$PLATFORM
$ROOTDIR/contrib/packaging/buildrpm --rpmbuilddir $RPMBUILDDIR --prepare $* $ROOTDIR/contrib/packaging/buildrpm --rpmbuilddir $RPMBUILDDIR --prepare $*
DSHARED=/mnt/shared DSHARED=/mnt/shared
DBUILDUSER=build DBUILDUSER=build
$DOCKER run -e http_proxy -e https_proxy -u $DBUILDUSER --rm -v $RPMBUILDDIR:$DSHARED $CONTAINER \ $DOCKER run -e http_proxy -e https_proxy -u $DBUILDUSER --rm -v $RPMBUILDDIR:$DSHARED $CONTAINER \

contrib/packaging/hg-docker

  • This file was added.
PropertyOld ValueNew Value
File Modenull100755
#!/usr/bin/env python3
#
# Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
import argparse
import pathlib
import shutil
import subprocess
import sys
def get_docker() -> str:
docker = shutil.which('docker.io') or shutil.which('docker')
if not docker:
print('could not find docker executable')
return 1
try:
out = subprocess.check_output([docker, '-h'], stderr=subprocess.STDOUT)
if b'Jansens' in out:
print('%s is the Docking System Tray; try installing docker.io' %
docker)
sys.exit(1)
except subprocess.CalledProcessError as e:
print('error calling `%s -h`: %s' % (docker, e.output))
sys.exit(1)
out = subprocess.check_output([docker, 'version'],
stderr=subprocess.STDOUT)
lines = out.splitlines()
if not any(l.startswith((b'Client:', b'Client version:')) for l in lines):
print('`%s version` does not look like Docker' % docker)
sys.exit(1)
if not any(l.startswith((b'Server:', b'Server version:')) for l in lines):
print('`%s version` does not look like Docker' % docker)
sys.exit(1)
return docker
def get_dockerfile(path: pathlib.Path, args: list) -> bytes:
with path.open('rb') as fh:
df = fh.read()
for k, v in args:
df = df.replace(b'%%%s%%' % k, v)
return df
def build_docker_image(dockerfile: pathlib.Path, params: list, tag: str):
"""Build a Docker image from a templatized Dockerfile."""
docker = get_docker()
dockerfile_path = pathlib.Path(dockerfile)
dockerfile = get_dockerfile(dockerfile_path, params)
print('building Dockerfile:')
print(dockerfile.decode('utf-8', 'replace'))
args = [
docker,
'build',
'--build-arg', 'http_proxy',
'--build-arg', 'https_proxy',
'--tag', tag,
'-',
]
print('executing: %r' % args)
subprocess.run(args, input=dockerfile, check=True)
def command_build(args):
build_args = []
for arg in args.build_arg:
k, v = arg.split('=', 1)
build_args.append((k.encode('utf-8'), v.encode('utf-8')))
build_docker_image(pathlib.Path(args.dockerfile),
build_args,
args.tag)
def command_docker(args):
print(get_docker())
def main() -> int:
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(title='subcommands')
build = subparsers.add_parser('build', help='Build a Docker image')
build.set_defaults(func=command_build)
build.add_argument('--build-arg', action='append', default=[],
help='Substitution to perform in Dockerfile; '
'format: key=value')
build.add_argument('dockerfile', help='path to Dockerfile to use')
build.add_argument('tag', help='Tag to apply to created image')
docker = subparsers.add_parser('docker-path', help='Resolve path to Docker')
docker.set_defaults(func=command_docker)
args = parser.parse_args()
return args.func(args)
if __name__ == '__main__':
sys.exit(main())

tests/test-check-module-imports.t

outputs, which should be fixed later. outputs, which should be fixed later.
$ testrepohg locate 'set:**.py or grep(r"^#!.*?python")' \ $ testrepohg locate 'set:**.py or grep(r"^#!.*?python")' \
> 'tests/**.t' \ > 'tests/**.t' \
> -X hgweb.cgi \ > -X hgweb.cgi \
> -X setup.py \ > -X setup.py \
> -X contrib/debugshell.py \ > -X contrib/debugshell.py \
> -X contrib/hgweb.fcgi \ > -X contrib/hgweb.fcgi \
> -X contrib/packaging/hg-docker \
> -X contrib/python-zstandard/ \ > -X contrib/python-zstandard/ \
> -X contrib/win32/hgwebdir_wsgi.py \ > -X contrib/win32/hgwebdir_wsgi.py \
> -X doc/gendoc.py \ > -X doc/gendoc.py \
> -X doc/hgmanpage.py \ > -X doc/hgmanpage.py \
> -X i18n/posplit \ > -X i18n/posplit \
> -X mercurial/thirdparty \ > -X mercurial/thirdparty \
> -X tests/hypothesishelpers.py \ > -X tests/hypothesishelpers.py \
> -X tests/test-check-interfaces.py \ > -X tests/test-check-interfaces.py \
contrib/packaging/dockerdeb