This is an archive of the discontinued Mercurial Phabricator instance.

			Path	Packages
P			contrib/packaging/hgpackaging/downloads.py (14 lines) Copied from contrib/packaging/hgpackaging/util.py
P			contrib/packaging/hgpackaging/util.py (106 lines) Copied to contrib/packaging/hgpackaging/downloads.py
M			contrib/packaging/inno/build.py (4 lines)
M			tests/test-check-code.t (1 line)

Commit	Parents	Author	Summary	Date
		Gregory Szorc		Mar 7 2019, 6:42 PM

Status	Author	Revision
Closed	indygreg	D6101 wix: remove enum and future packages
Closed	indygreg	D6100 wix: remove pywin32
Closed	indygreg	D6099 wix: remove sphinx and dependencies
Closed	indygreg	D6098 wix: restore COPYING.rtf
Closed	indygreg	D6097 wix: functionality to automate building WiX installers
Closed	indygreg	D6096 wix: move contrib/wix to contrib/packaging/wix
Closed	indygreg	D6095 wix: remove hg.cmd
Closed	indygreg	D6094 setup: include hgext3rd package in py2exe builds
Closed	indygreg	D6093 setup: properly install build_hgextindex for py2exe builds
Closed	indygreg	D6092 setup: configure py2exe config via environment variables
Closed	indygreg	D6091 packaging: extract py2exe functionality to own module
Closed	indygreg	D6090 packaging: extract python exe info to own function
Closed	indygreg	D6089 packaging: don't use temporary directory
Closed	indygreg	D6088 packaging: extract virtualenv and py2exe to build directory
Closed	indygreg	D6087 packaging: move Inno Setup core logic into a module
Closed	indygreg	D6086 packaging: move find_vc_runtime_files() into hgpackaging.util
Closed	indygreg	D6085 packaging: move DOWNLOADS dict to hgpackaging.downloads
Closed	indygreg	D6084 packaging: split downloading code into own module
Closed	indygreg	D6083 packaging: establish hgpackaging package

Diff 14391

contrib/packaging/hgpackaging/downloads.py

This file was copied from contrib/packaging/hgpackaging/util.py.

	# util.py - Common packaging utility code.			# downloads.py - Code for downloading dependencies.
	#			#
	# Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>			# Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>
	#			#
	# This software may be used and distributed according to the terms of the			# This software may be used and distributed according to the terms of the
	# GNU General Public License version 2 or any later version.			# GNU General Public License version 2 or any later version.

	# no-check-code because Python 3 native.			# no-check-code because Python 3 native.

	import gzip			import gzip
	import hashlib			import hashlib
	import pathlib			import pathlib
	import tarfile
	import urllib.request			import urllib.request
	import zipfile


	def hash_path(p: pathlib.Path):			def hash_path(p: pathlib.Path):
	h = hashlib.sha256()			h = hashlib.sha256()

	with p.open('rb') as fh:			with p.open('rb') as fh:
	while True:			while True:
	chunk = fh.read(65536)			chunk = fh.read(65536)
	url = entry['url']			url = entry['url']

	local_name = local_name or url[url.rindex('/') + 1:]			local_name = local_name or url[url.rindex('/') + 1:]

	local_path = dest_path / local_name			local_path = dest_path / local_name
	download_to_path(url, local_path, entry['size'], entry['sha256'])			download_to_path(url, local_path, entry['size'], entry['sha256'])

	return local_path			return local_path


	def extract_tar_to_directory(source: pathlib.Path, dest: pathlib.Path):
	with tarfile.open(source, 'r') as tf:
	tf.extractall(dest)


	def extract_zip_to_directory(source: pathlib.Path, dest: pathlib.Path):
	with zipfile.ZipFile(source, 'r') as zf:
	zf.extractall(dest)

contrib/packaging/hgpackaging/util.py

This file was copied to contrib/packaging/hgpackaging/downloads.py.

	# util.py - Common packaging utility code.			# util.py - Common packaging utility code.
	#			#
	# Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>			# Copyright 2019 Gregory Szorc <gregory.szorc@gmail.com>
	#			#
	# This software may be used and distributed according to the terms of the			# This software may be used and distributed according to the terms of the
	# GNU General Public License version 2 or any later version.			# GNU General Public License version 2 or any later version.

	# no-check-code because Python 3 native.			# no-check-code because Python 3 native.

	import gzip
	import hashlib
	import pathlib			import pathlib
	import tarfile			import tarfile
	import urllib.request
	import zipfile			import zipfile


	def hash_path(p: pathlib.Path):
	h = hashlib.sha256()

	with p.open('rb') as fh:
	while True:
	chunk = fh.read(65536)
	if not chunk:
	break

	h.update(chunk)

	return h.hexdigest()


	class IntegrityError(Exception):
	"""Represents an integrity error when downloading a URL."""


	def secure_download_stream(url, size, sha256):
	"""Securely download a URL to a stream of chunks.

	If the integrity of the download fails, an IntegrityError is
	raised.
	"""
	h = hashlib.sha256()
	length = 0

	with urllib.request.urlopen(url) as fh:
	if not url.endswith('.gz') and fh.info().get('Content-Encoding') == 'gzip':
	fh = gzip.GzipFile(fileobj=fh)

	while True:
	chunk = fh.read(65536)
	if not chunk:
	break

	h.update(chunk)
	length += len(chunk)

	yield chunk

	digest = h.hexdigest()

	if length != size:
	raise IntegrityError('size mismatch on %s: wanted %d; got %d' % (
	url, size, length))

	if digest != sha256:
	raise IntegrityError('sha256 mismatch on %s: wanted %s; got %s' % (
	url, sha256, digest))


	def download_to_path(url: str, path: pathlib.Path, size: int, sha256: str):
	"""Download a URL to a filesystem path, possibly with verification."""

	# We download to a temporary file and rename at the end so there's
	# no chance of the final file being partially written or containing
	# bad data.
	print('downloading %s to %s' % (url, path))

	if path.exists():
	good = True

	if path.stat().st_size != size:
	print('existing file size is wrong; removing')
	good = False

	if good:
	if hash_path(path) != sha256:
	print('existing file hash is wrong; removing')
	good = False

	if good:
	print('%s exists and passes integrity checks' % path)
	return

	path.unlink()

	tmp = path.with_name('%s.tmp' % path.name)

	try:
	with tmp.open('wb') as fh:
	for chunk in secure_download_stream(url, size, sha256):
	fh.write(chunk)
	except IntegrityError:
	tmp.unlink()
	raise

	tmp.rename(path)
	print('successfully downloaded %s' % url)


	def download_entry(entry: dict, dest_path: pathlib.Path, local_name=None) -> pathlib.Path:
	url = entry['url']

	local_name = local_name or url[url.rindex('/') + 1:]

	local_path = dest_path / local_name
	download_to_path(url, local_path, entry['size'], entry['sha256'])

	return local_path


	def extract_tar_to_directory(source: pathlib.Path, dest: pathlib.Path):			def extract_tar_to_directory(source: pathlib.Path, dest: pathlib.Path):
	with tarfile.open(source, 'r') as tf:			with tarfile.open(source, 'r') as tf:
	tf.extractall(dest)			tf.extractall(dest)


	def extract_zip_to_directory(source: pathlib.Path, dest: pathlib.Path):			def extract_zip_to_directory(source: pathlib.Path, dest: pathlib.Path):
	with zipfile.ZipFile(source, 'r') as zf:			with zipfile.ZipFile(source, 'r') as zf:
	zf.extractall(dest)			zf.extractall(dest)

contrib/packaging/inno/build.py

	"""Build the Inno installer.			"""Build the Inno installer.

	Build files will be placed in ``build_dir``.			Build files will be placed in ``build_dir``.

	py2exe's setup.py doesn't use setuptools. It doesn't have modern logic			py2exe's setup.py doesn't use setuptools. It doesn't have modern logic
	for finding the Python 2.7 toolchain. So, we require the environment			for finding the Python 2.7 toolchain. So, we require the environment
	to already be configured with an active toolchain.			to already be configured with an active toolchain.
	"""			"""
	from hgpackaging.util import (			from hgpackaging.downloads import (
	download_entry,			download_entry,
				)
				from hgpackaging.util import (
	extract_tar_to_directory,			extract_tar_to_directory,
	extract_zip_to_directory,			extract_zip_to_directory,
	)			)

	if not iscc.exists():			if not iscc.exists():
	raise Exception('%s does not exist' % iscc)			raise Exception('%s does not exist' % iscc)

	if 'VCINSTALLDIR' not in os.environ:			if 'VCINSTALLDIR' not in os.environ:

tests/test-check-code.t

	#require test-repo			#require test-repo

	$ . "$TESTDIR/helpers-testrepo.sh"			$ . "$TESTDIR/helpers-testrepo.sh"
	$ check_code="$TESTDIR"/../contrib/check-code.py			$ check_code="$TESTDIR"/../contrib/check-code.py
	$ cd "$TESTDIR"/..			$ cd "$TESTDIR"/..

	New errors are not allowed. Warnings are strongly discouraged.			New errors are not allowed. Warnings are strongly discouraged.
	(The writing "no-che?k-code" is for not skipping this file when checking.)			(The writing "no-che?k-code" is for not skipping this file when checking.)

	$ testrepohg locate \			$ testrepohg locate \
	> -X contrib/python-zstandard \			> -X contrib/python-zstandard \
	> -X hgext/fsmonitor/pywatchman \			> -X hgext/fsmonitor/pywatchman \
	> -X mercurial/thirdparty \			> -X mercurial/thirdparty \
	> \| sed 's-\\-/-g' \| "$check_code" --warnings --per-file=0 - \|\| false			> \| sed 's-\\-/-g' \| "$check_code" --warnings --per-file=0 - \|\| false
				Skipping contrib/packaging/hgpackaging/downloads.py it has no-che?k-code (glob)
	Skipping contrib/packaging/hgpackaging/util.py it has no-che?k-code (glob)			Skipping contrib/packaging/hgpackaging/util.py it has no-che?k-code (glob)
	Skipping contrib/packaging/inno/build.py it has no-che?k-code (glob)			Skipping contrib/packaging/inno/build.py it has no-che?k-code (glob)
	Skipping i18n/polib.py it has no-che?k-code (glob)			Skipping i18n/polib.py it has no-che?k-code (glob)
	Skipping mercurial/statprof.py it has no-che?k-code (glob)			Skipping mercurial/statprof.py it has no-che?k-code (glob)
	Skipping tests/badserverext.py it has no-che?k-code (glob)			Skipping tests/badserverext.py it has no-che?k-code (glob)

	@commands in debugcommands.py should be in alphabetical order.			@commands in debugcommands.py should be in alphabetical order.

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	14391		Mar 7 2019, 7:49 PM	★	★
Diff 2	14439	rHGc2237fe1359eee4213f2eef843b86368a3a410e3	Mar 7 2019, 6:42 PM	★	★

This is an archive of the discontinued Mercurial Phabricator instance.

packaging: split downloading code into own moduleClosedPublic

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 14391

contrib/packaging/hgpackaging/downloads.py

contrib/packaging/hgpackaging/util.py

contrib/packaging/inno/build.py

tests/test-check-code.t

packaging: split downloading code into own module
ClosedPublic

Revision Contents
Changeset List