This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8351

hgcli: customize for Mercurial
ClosedPublic

Authored by indygreg on Mar 31 2020, 11:29 PM.

Details

Reviewers
marmoute
Group Reviewers
hg-reviewers
Commits
rHGbc847878f4c0: hgcli: customize for Mercurial
Summary

Now that we have a shiny new PyOxidizer-based hgcli project, let's
customize it for Mercurial!

This commit replaces the auto-generated pyoxidizer.bzl with one
that installs Mercurial from the local source repository.

A README.md with build instructions has been added.

The Cargo.toml file has been updated to reflect the proper license
and reference the added README.md.

In my Linux environment, running the test suite yields 27 failures.

It's worth noting the run time of the test harness on Linux on my
Ryzen 3950X:

before: 378s wall; 9982s user; 1195s sys
after: 353s wall; 8996s user; 958s sys
% orig: 93.4 wall; 90.1 user; 80.2 sys

While I haven't measured explicitly, I suspect the performance win is
due to in-memory resource loading (which is known to be faster than
Python's filesystem importer).

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Mar 31 2020, 11:29 PM
marmoute accepted this revision.Apr 1 2020, 4:15 AM
marmoute added a subscriber: marmoute.

Seems overall good. I added a request for comments.

rust/hgcli/pyoxidizer.bzl
15

It would be nice to have a comment abotu why this is necesssary (as for the other entry in this call)

indygreg added a commit: rHGbc847878f4c0: hgcli: customize for Mercurial.Apr 1 2020, 11:22 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHGbc847878f4c0: hgcli: customize for Mercurial (authored by indygreg). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mrust/hgcli/Cargo.toml (6 lines)
AMrust/hgcli/README.md (50 lines)
Mrust/hgcli/pyoxidizer.bzl (162 lines)

Diff 20938

rust/hgcli/Cargo.toml

[package] [package]
name = "hgcli" name = "hgcli"
version = "0.1.0" version = "0.1.0"
build = "build.rs" build = "build.rs"
authors = ["Gregory Szorc <gregory.szorc@gmail.com>"] authors = ["Gregory Szorc <gregory.szorc@gmail.com>"]
edition = "2018" edition = "2018"
license = "GPL-2.0"
readme = "README.md"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html [[bin]]
name = "hg"
path = "src/main.rs"
[dependencies] [dependencies]
jemallocator-global = { version = "0.3", optional = true } jemallocator-global = { version = "0.3", optional = true }
pyembed = { git = "https://github.com/indygreg/PyOxidizer.git", rev = "c772a1379c3026314eda1c8ea244b86c0658951d", default-features=false } pyembed = { git = "https://github.com/indygreg/PyOxidizer.git", rev = "c772a1379c3026314eda1c8ea244b86c0658951d", default-features=false }
[features] [features]
default = ["build-mode-pyoxidizer-exe"] default = ["build-mode-pyoxidizer-exe"]
jemalloc = ["jemallocator-global", "pyembed/jemalloc"] jemalloc = ["jemallocator-global", "pyembed/jemalloc"]
build-mode-pyoxidizer-exe = ["pyembed/build-mode-pyoxidizer-exe"] build-mode-pyoxidizer-exe = ["pyembed/build-mode-pyoxidizer-exe"]
build-mode-prebuilt-artifacts = ["pyembed/build-mode-prebuilt-artifacts"] build-mode-prebuilt-artifacts = ["pyembed/build-mode-prebuilt-artifacts"]
cpython-link-unresolved-static = ["pyembed/cpython-link-unresolved-static"] cpython-link-unresolved-static = ["pyembed/cpython-link-unresolved-static"]
cpython-link-default = ["pyembed/cpython-link-default"] cpython-link-default = ["pyembed/cpython-link-default"]

rust/hgcli/README.md

  • This file was added.
# Oxidized Mercurial
This project provides a Rust implementation of the Mercurial (`hg`)
version control tool.
Under the hood, the project uses
[PyOxidizer](https://github.com/indygreg/PyOxidizer) to embed a Python
interpreter in a binary built with Rust. At run-time, the Rust `fn main()`
is called and Rust code handles initial process startup. An in-process
Python interpreter is started (if needed) to provide additional
functionality.
# Building
This project currently requires an unreleased version of PyOxidizer
(0.7.0-pre). For best results, build the exact PyOxidizer commit
as defined in the `pyoxidizer.bzl` file:
$ git clone https://github.com/indygreg/PyOxidizer.git
$ cd PyOxidizer
$ git checkout <Git commit from pyoxidizer.bzl>
$ cargo build --release
Then build this Rust project using the built `pyoxidizer` executable::
$ /path/to/pyoxidizer/target/release/pyoxidizer build
If all goes according to plan, there should be an assembled application
under `build/<arch>/debug/app/` with an `hg` executable:
$ build/x86_64-unknown-linux-gnu/debug/app/hg version
Mercurial Distributed SCM (version 5.3.1+433-f99cd77d53dc+20200331)
(see https://mercurial-scm.org for more information)
Copyright (C) 2005-2020 Matt Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
# Running Tests
To run tests with a built `hg` executable, you can use the `--with-hg`
argument to `run-tests.py`. But there's a wrinkle: many tests run custom
Python scripts that need to `import` modules provided by Mercurial. Since
these modules are embedded in the produced `hg` executable, a regular
Python interpreter can't access them! To work around this, set `PYTHONPATH`
to the Mercurial source directory. e.g.:
$ cd /path/to/hg/src/tests
$ PYTHONPATH=`pwd`/.. python3.7 run-tests.py \
--with-hg `pwd`/../rust/hgcli/build/x86_64-unknown-linux-gnu/debug/app/hg

rust/hgcli/pyoxidizer.bzl

# This file defines how PyOxidizer application building and packaging is ROOT = CWD + "/../.."
# performed. See the pyoxidizer crate's documentation for extensive
# documentation on this file format.
# Obtain the default PythonDistribution for our build target. We link
# this distribution into our produced executable and extract the Python
# standard library from it.
def make_dist():
return default_python_distribution()
# Configuration files consist of functions which define build "targets."
# This function creates a Python executable and installs it in a destination
# directory.
def make_exe(dist):
# This variable defines the configuration of the
# embedded Python interpreter.
python_config = PythonInterpreterConfig(
# bytes_warning=0,
# dont_write_bytecode=True,
# ignore_environment=True,
# inspect=False,
# interactive=False,
# isolated=False,
# legacy_windows_fs_encoding=False,
# legacy_windows_stdio=False,
# no_site=True,
# no_user_site_directory=True,
# optimize_level=0,
# parser_debug=False,
# stdio_encoding=None,
# unbuffered_stdio=False,
# filesystem_importer=False,
# sys_frozen=False,
# sys_meipass=False,
# sys_paths=None,
# raw_allocator=None,
# terminfo_resolution="dynamic",
# terminfo_dirs=None,
# use_hash_seed=False,
# verbose=0,
# write_modules_directory_env=None,
# run_eval=None,
# run_module=None,
# run_noop=False,
# run_repl=True,
)
# The run_eval, run_module, run_noop, and run_repl arguments are mutually def make_exe():
# exclusive controls over what the interpreter should do once it initializes. dist = default_python_distribution()
#
# run_eval -- Run the specified string value via `eval()`.
# run_module -- Import the specified module as __main__ and run it.
# run_noop -- Do nothing.
# run_repl -- Start a Python REPL.
#
# These arguments can be ignored if you are providing your own Rust code for
# starting the interpreter, as Rust code has full control over interpreter
# behavior.
# Produce a PythonExecutable from a Python distribution, embedded
# resources, and other options. The returned object represents the
# standalone executable that will be built.
exe = dist.to_python_executable(
name = "hgcli",
config = python_config,
# Embed all extension modules, making this a fully-featured Python.
extension_module_filter = "all",
# Only package the minimal set of extension modules needed to initialize code = "import hgdemandimport; hgdemandimport.enable(); from mercurial import dispatch; dispatch.run()"
# a Python interpreter. Many common packages in Python's standard
# library won't work with this setting.
#extension_module_filter='minimal',
# Only package extension modules that don't require linking against
# non-Python libraries. e.g. will exclude support for OpenSSL, SQLite3,
# other features that require external libraries.
#extension_module_filter='no-libraries',
# Only package extension modules that don't link against GPL licensed
# libraries.
#extension_module_filter='no-gpl',
# Include Python module sources. This isn't strictly required and it does
# make binary sizes larger. But having the sources can be useful for
# activities such as debugging.
include_sources = True,
# Whether to include non-module resource data/files. config = PythonInterpreterConfig(
include_resources = False, raw_allocator = "system",
run_eval = code,
# We want to let the user load extensions from the file system
filesystem_importer = True,
# We need this to make resourceutil happy, since it looks for sys.frozen.
sys_frozen = True,
legacy_windows_stdio = True,
marmouteUnsubmitted
Not Done

It would be nice to have a comment abotu why this is necesssary (as for the other entry in this call)

marmoute: It would be nice to have a comment abotu why this is necesssary (as for the other entry in this…
)
# Do not include functionality for testing Python itself. exe = dist.to_python_executable(
include_test = False, name = "hg",
resources_policy = "prefer-in-memory-fallback-filesystem-relative:lib",
config = config,
# Extension may depend on any Python functionality. Include all
# extensions.
extension_module_filter = "all",
) )
# Invoke `pip install` with our Python distribution to install a single package. exe.add_python_resources(dist.pip_install([ROOT]))
# `pip_install()` returns objects representing installed files.
# `add_in_memory_python_resources()` adds these objects to the binary,
# marking them for in-memory loading.
#exe.add_in_memory_python_resources(dist.pip_install(["appdirs"]))
# Invoke `pip install` using a requirements file and add the collected resources
# to our binary.
#exe.add_in_memory_python_resources(dist.pip_install(["-r", "requirements.txt"]))
# Read Python files from a local directory and add them to our embedded
# context, taking just the resources belonging to the `foo` and `bar`
# Python packages.
#exe.add_in_memory_python_resources(dist.read_package_root(
# path="/src/mypackage",
# packages=["foo", "bar"],
#))
# Discover Python files from a virtualenv and add them to our embedded
# context.
#exe.add_in_memory_python_resources(dist.read_virtualenv(path="/path/to/venv"))
# Filter all resources collected so far through a filter of names
# in a file.
#exe.filter_from_files(files=["/path/to/filter-file"]))
# Return our `PythonExecutable` instance so it can be built and
# referenced by other consumers of this target.
return exe return exe
def make_embedded_resources(exe):
return exe.to_embedded_resources()
def make_install(exe): def make_install(exe):
# Create an object that represents our installed application file layout. m = FileManifest()
files = FileManifest()
# Add the generated executable to our install layout in the root directory. # `hg` goes in root directory.
files.add_python_resource(".", exe) m.add_python_resource(".", exe)
return files templates = glob(
include = [ROOT + "/mercurial/templates/**/*"],
strip_prefix = ROOT + "/mercurial/",
)
m.add_manifest(templates)
# Tell PyOxidizer about the build targets defined above. return m
register_target("dist", make_dist)
register_target("exe", make_exe, depends = ["dist"], default = True) def make_embedded_resources(exe):
register_target("resources", make_embedded_resources, depends = ["exe"], default_build_script = True) return exe.to_embedded_resources()
register_target("install", make_install, depends = ["exe"])
# Resolve whatever targets the invoker of this configuration file is requesting register_target("exe", make_exe)
# be resolved. register_target("app", make_install, depends = ["exe"], default = True)
register_target("embedded", make_embedded_resources, depends = ["exe"], default_build_script = True)
resolve_targets() resolve_targets()
# END OF COMMON USER-ADJUSTED SETTINGS. # END OF COMMON USER-ADJUSTED SETTINGS.
# #
# Everything below this is typically managed by PyOxidizer and doesn't need # Everything below this is typically managed by PyOxidizer and doesn't need
# to be updated by people. # to be updated by people.
PYOXIDIZER_VERSION = "0.7.0-pre" PYOXIDIZER_VERSION = "0.7.0-pre"
PYOXIDIZER_COMMIT = "c772a1379c3026314eda1c8ea244b86c0658951d" PYOXIDIZER_COMMIT = "c772a1379c3026314eda1c8ea244b86c0658951d"