( )⚙ D8594 rust: remove support for `re2`

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8594

rust: remove support for `re2`
ClosedPublic

Authored by Alphare on May 29 2020, 6:34 AM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG9f96beb9bafe: rust: remove support for `re2`
Summary

With the performance issues with regex figured out and fixed in previous
patches and regex newly gaining support for empty alternations, there is no
reason to keep re2 around anymore. It's only *marginally* faster at creating
the regex which saves at most a couple of ms, but gets beaten by regex in
every other aspect.

This removes the Rust/C/C++ bridge (hooray!), the with-re2 feature, the
conditional code that goes with it, the documentation and relevant part of the
debug/module output.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Alphare created this revision.May 29 2020, 6:34 AM
Alphare added a commit: rHG9f96beb9bafe: rust: remove support for `re2`.May 29 2020, 11:28 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG9f96beb9bafe: rust: remove support for `re2` (authored by Alphare). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 21523

mercurial/debugcommands.py

), ),
) )
re2 = b'missing' re2 = b'missing'
if util._re2: if util._re2:
re2 = b'available' re2 = b'available'
fm.plain(_(b'checking "re2" regexp engine (%s)\n') % re2) fm.plain(_(b'checking "re2" regexp engine (%s)\n') % re2)
fm.data(re2=bool(util._re2)) fm.data(re2=bool(util._re2))
rust_debug_mod = policy.importrust("debug")
if rust_debug_mod is not None:
re2_rust = b'installed' if rust_debug_mod.re2_installed else b'missing'
msg = b'checking "re2" regexp engine Rust bindings (%s)\n'
fm.plain(_(msg % re2_rust))
# templates # templates
p = templater.templatepaths() p = templater.templatepaths()
fm.write(b'templatedirs', b'checking templates (%s)...\n', b' '.join(p)) fm.write(b'templatedirs', b'checking templates (%s)...\n', b' '.join(p))
fm.condwrite(not p, b'', _(b" no template directories found\n")) fm.condwrite(not p, b'', _(b" no template directories found\n"))
if p: if p:
m = templater.templatepath(b"map-cmdline.default") m = templater.templatepath(b"map-cmdline.default")
if m: if m:
# template found, check if it is working # template found, check if it is working

rust/Cargo.lock

source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]] [[package]]
name = "byteorder" name = "byteorder"
version = "1.3.4" version = "1.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]] [[package]]
name = "cc"
version = "1.0.50"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "cfg-if" name = "cfg-if"
version = "0.1.10" version = "0.1.10"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]] [[package]]
name = "chrono" name = "chrono"
version = "0.4.11" version = "0.4.11"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
version = "0.4.2" version = "0.4.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]] [[package]]
name = "hg-core" name = "hg-core"
version = "0.1.0" version = "0.1.0"
dependencies = [ dependencies = [
"byteorder 1.3.4 (registry+https://github.com/rust-lang/crates.io-index)", "byteorder 1.3.4 (registry+https://github.com/rust-lang/crates.io-index)",
"cc 1.0.50 (registry+https://github.com/rust-lang/crates.io-index)",
"clap 2.33.0 (registry+https://github.com/rust-lang/crates.io-index)", "clap 2.33.0 (registry+https://github.com/rust-lang/crates.io-index)",
"crossbeam 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)", "crossbeam 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)",
"hex 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)", "hex 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)",
"lazy_static 1.4.0 (registry+https://github.com/rust-lang/crates.io-index)", "lazy_static 1.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
"libc 0.2.67 (registry+https://github.com/rust-lang/crates.io-index)",
"log 0.4.8 (registry+https://github.com/rust-lang/crates.io-index)", "log 0.4.8 (registry+https://github.com/rust-lang/crates.io-index)",
"memchr 2.3.3 (registry+https://github.com/rust-lang/crates.io-index)", "memchr 2.3.3 (registry+https://github.com/rust-lang/crates.io-index)",
"memmap 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)", "memmap 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)",
"micro-timer 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)", "micro-timer 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
"pretty_assertions 0.6.1 (registry+https://github.com/rust-lang/crates.io-index)", "pretty_assertions 0.6.1 (registry+https://github.com/rust-lang/crates.io-index)",
"rand 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)", "rand 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)",
"rand_distr 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)", "rand_distr 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
"rand_pcg 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)", "rand_pcg 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
[metadata] [metadata]
"checksum aho-corasick 0.7.10 (registry+https://github.com/rust-lang/crates.io-index)" = "8716408b8bc624ed7f65d223ddb9ac2d044c0547b6fa4b0d554f3a9540496ada" "checksum aho-corasick 0.7.10 (registry+https://github.com/rust-lang/crates.io-index)" = "8716408b8bc624ed7f65d223ddb9ac2d044c0547b6fa4b0d554f3a9540496ada"
"checksum ansi_term 0.11.0 (registry+https://github.com/rust-lang/crates.io-index)" = "ee49baf6cb617b853aa8d93bf420db2383fab46d314482ca2803b40d5fde979b" "checksum ansi_term 0.11.0 (registry+https://github.com/rust-lang/crates.io-index)" = "ee49baf6cb617b853aa8d93bf420db2383fab46d314482ca2803b40d5fde979b"
"checksum atty 0.2.14 (registry+https://github.com/rust-lang/crates.io-index)" = "d9b39be18770d11421cdb1b9947a45dd3f37e93092cbf377614828a319d5fee8" "checksum atty 0.2.14 (registry+https://github.com/rust-lang/crates.io-index)" = "d9b39be18770d11421cdb1b9947a45dd3f37e93092cbf377614828a319d5fee8"
"checksum autocfg 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)" = "f8aac770f1885fd7e387acedd76065302551364496e46b3dd00860b2f8359b9d" "checksum autocfg 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)" = "f8aac770f1885fd7e387acedd76065302551364496e46b3dd00860b2f8359b9d"
"checksum bitflags 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693" "checksum bitflags 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "cf1de2fe8c75bc145a2f577add951f8134889b4795d47466a54a5c846d691693"
"checksum byteorder 1.3.4 (registry+https://github.com/rust-lang/crates.io-index)" = "08c48aae112d48ed9f069b33538ea9e3e90aa263cfa3d1c24309612b1f7472de" "checksum byteorder 1.3.4 (registry+https://github.com/rust-lang/crates.io-index)" = "08c48aae112d48ed9f069b33538ea9e3e90aa263cfa3d1c24309612b1f7472de"
"checksum cc 1.0.50 (registry+https://github.com/rust-lang/crates.io-index)" = "95e28fa049fda1c330bcf9d723be7663a899c4679724b34c81e9f5a326aab8cd"
"checksum cfg-if 0.1.10 (registry+https://github.com/rust-lang/crates.io-index)" = "4785bdd1c96b2a846b2bd7cc02e86b6b3dbf14e7e53446c4f54c92a361040822" "checksum cfg-if 0.1.10 (registry+https://github.com/rust-lang/crates.io-index)" = "4785bdd1c96b2a846b2bd7cc02e86b6b3dbf14e7e53446c4f54c92a361040822"
"checksum chrono 0.4.11 (registry+https://github.com/rust-lang/crates.io-index)" = "80094f509cf8b5ae86a4966a39b3ff66cd7e2a3e594accec3743ff3fabeab5b2" "checksum chrono 0.4.11 (registry+https://github.com/rust-lang/crates.io-index)" = "80094f509cf8b5ae86a4966a39b3ff66cd7e2a3e594accec3743ff3fabeab5b2"
"checksum clap 2.33.0 (registry+https://github.com/rust-lang/crates.io-index)" = "5067f5bb2d80ef5d68b4c87db81601f0b75bca627bc2ef76b141d7b846a3c6d9" "checksum clap 2.33.0 (registry+https://github.com/rust-lang/crates.io-index)" = "5067f5bb2d80ef5d68b4c87db81601f0b75bca627bc2ef76b141d7b846a3c6d9"
"checksum colored 1.9.3 (registry+https://github.com/rust-lang/crates.io-index)" = "f4ffc801dacf156c5854b9df4f425a626539c3a6ef7893cc0c5084a23f0b6c59" "checksum colored 1.9.3 (registry+https://github.com/rust-lang/crates.io-index)" = "f4ffc801dacf156c5854b9df4f425a626539c3a6ef7893cc0c5084a23f0b6c59"
"checksum cpython 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)" = "bfaf3847ab963e40c4f6dd8d6be279bdf74007ae2413786a0dcbb28c52139a95" "checksum cpython 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)" = "bfaf3847ab963e40c4f6dd8d6be279bdf74007ae2413786a0dcbb28c52139a95"
"checksum crossbeam 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)" = "69323bff1fb41c635347b8ead484a5ca6c3f11914d784170b158d8449ab07f8e" "checksum crossbeam 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)" = "69323bff1fb41c635347b8ead484a5ca6c3f11914d784170b158d8449ab07f8e"
"checksum crossbeam-channel 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)" = "cced8691919c02aac3cb0a1bc2e9b73d89e832bf9a06fc579d4e71b68a2da061" "checksum crossbeam-channel 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)" = "cced8691919c02aac3cb0a1bc2e9b73d89e832bf9a06fc579d4e71b68a2da061"
"checksum crossbeam-deque 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)" = "9f02af974daeee82218205558e51ec8768b48cf524bd01d550abe5573a608285" "checksum crossbeam-deque 0.7.3 (registry+https://github.com/rust-lang/crates.io-index)" = "9f02af974daeee82218205558e51ec8768b48cf524bd01d550abe5573a608285"

rust/README.rst

If the environment variable ``HGWITHRUSTEXT=cpython`` is set, the Rust If the environment variable ``HGWITHRUSTEXT=cpython`` is set, the Rust
extension will be used by default unless ``--no-rust``. extension will be used by default unless ``--no-rust``.
One day we may use this environment variable to switch to new experimental One day we may use this environment variable to switch to new experimental
binding crates like a hypothetical ``HGWITHRUSTEXT=hpy``. binding crates like a hypothetical ``HGWITHRUSTEXT=hpy``.
Using the fastest ``hg status``
-------------------------------
The code for ``hg status`` needs to conform to ``.hgignore`` rules, which are
all translated into regex.
In the first version, for compatibility and ease of development reasons, the
Re2 regex engine was chosen until we figured out if the ``regex`` crate had
similar enough behavior.
Now that that work has been done, the default behavior is to use the ``regex``
crate, that provides a significant performance boost compared to the standard
Python + C path in many commands such as ``status``, ``diff`` and ``commit``,
However, the ``Re2`` path remains slightly faster for our use cases and remains
a better option for getting the most speed out of your Mercurial.
If you want to use ``Re2``, you need to install ``Re2`` following Google's
guidelines: https://github.com/google/re2/wiki/Install.
Then, use ``HG_RUST_FEATURES=with-re2`` and
``HG_RE2_PATH=system|<path to your re2 install>`` when building ``hg`` to
signal the use of Re2. Using the local path instead of the "system" RE2 links
it statically.
For example::
$ HG_RUST_FEATURES=with-re2 HG_RE2_PATH=system make PURE=--rust
$ # OR
$ HG_RUST_FEATURES=with-re2 HG_RE2_PATH=/path/to/re2 make PURE=--rust
Developing Rust Developing Rust
=============== ===============
The current version of Rust in use is ``1.34.2``, because it's what Debian The current version of Rust in use is ``1.34.2``, because it's what Debian
stable has. You can use ``rustup override set 1.34.2`` at the root of the repo stable has. You can use ``rustup override set 1.34.2`` at the root of the repo
to make it easier on you. to make it easier on you.
Go to the ``hg-cpython`` folder:: Go to the ``hg-cpython`` folder::
using the nightly version because it has been stable enough and provides using the nightly version because it has been stable enough and provides
comment folding. comment folding.
To format the entire Rust workspace:: To format the entire Rust workspace::
$ cargo +nightly fmt $ cargo +nightly fmt
This requires you to have the nightly toolchain installed. This requires you to have the nightly toolchain installed.
Additional features
-------------------
As mentioned in the section about ``hg status``, code paths using ``re2`` are
opt-in.
For example::
$ cargo check --features with-re2

rust/hg-core/Cargo.toml

[package] [package]
name = "hg-core" name = "hg-core"
version = "0.1.0" version = "0.1.0"
authors = ["Georges Racinet <gracinet@anybox.fr>"] authors = ["Georges Racinet <gracinet@anybox.fr>"]
description = "Mercurial pure Rust core library, with no assumption on Python bindings (FFI)" description = "Mercurial pure Rust core library, with no assumption on Python bindings (FFI)"
edition = "2018" edition = "2018"
build = "build.rs"
[lib] [lib]
name = "hg" name = "hg"
[dependencies] [dependencies]
byteorder = "1.3.4" byteorder = "1.3.4"
hex = "0.4.2" hex = "0.4.2"
lazy_static = "1.4.0" lazy_static = "1.4.0"
libc = { version = "0.2.66", optional = true }
memchr = "2.3.3" memchr = "2.3.3"
rand = "0.7.3" rand = "0.7.3"
rand_pcg = "0.2.1" rand_pcg = "0.2.1"
rand_distr = "0.2.2" rand_distr = "0.2.2"
rayon = "1.3.0" rayon = "1.3.0"
regex = "1.3.9" regex = "1.3.9"
twox-hash = "1.5.0" twox-hash = "1.5.0"
same-file = "1.0.6" same-file = "1.0.6"
crossbeam = "0.7.3" crossbeam = "0.7.3"
micro-timer = "0.2.1" micro-timer = "0.2.1"
log = "0.4.8" log = "0.4.8"
[dev-dependencies] [dev-dependencies]
clap = "*" clap = "*"
memmap = "0.7.0" memmap = "0.7.0"
pretty_assertions = "0.6.1" pretty_assertions = "0.6.1"
tempfile = "3.1.0" tempfile = "3.1.0"
[build-dependencies]
cc = { version = "1.0.48", optional = true }
[features]
default = []
with-re2 = ["cc", "libc"]

rust/hg-core/build.rs

  • This file was deleted.
// build.rs
//
// Copyright 2020 Raphaël Gomès <rgomes@octobus.net>
//
// This software may be used and distributed according to the terms of the
// GNU General Public License version 2 or any later version.
#[cfg(feature = "with-re2")]
use cc;
/// Uses either the system Re2 install as a dynamic library or the provided
/// build as a static library
#[cfg(feature = "with-re2")]
fn compile_re2() {
use cc;
use std::path::Path;
use std::process::exit;
let msg = r"HG_RE2_PATH must be one of `system|<path to build source clone of Re2>`";
let re2 = match std::env::var_os("HG_RE2_PATH") {
None => {
eprintln!("{}", msg);
exit(1)
}
Some(v) => {
if v == "system" {
None
} else {
Some(v)
}
}
};
let mut options = cc::Build::new();
options
.cpp(true)
.flag("-std=c++11")
.file("src/re2/rust_re2.cpp");
if let Some(ref source) = re2 {
options.include(Path::new(source));
};
options.compile("librustre.a");
if let Some(ref source) = &re2 {
// Link the local source statically
println!(
"cargo:rustc-link-search=native={}",
Path::new(source).join(Path::new("obj")).display()
);
println!("cargo:rustc-link-lib=static=re2");
} else {
println!("cargo:rustc-link-lib=re2");
}
}
fn main() {
#[cfg(feature = "with-re2")]
compile_re2();
}

rust/hg-core/src/lib.rs

}, },
CopyMap, CopyMapIter, DirstateEntry, DirstateParents, EntryState, CopyMap, CopyMapIter, DirstateEntry, DirstateParents, EntryState,
StateMap, StateMapIter, StateMap, StateMapIter,
}; };
mod filepatterns; mod filepatterns;
pub mod matchers; pub mod matchers;
pub mod revlog; pub mod revlog;
pub use revlog::*; pub use revlog::*;
#[cfg(feature = "with-re2")]
pub mod re2;
pub mod utils; pub mod utils;
// Remove this to see (potential) non-artificial compile failures. MacOS // Remove this to see (potential) non-artificial compile failures. MacOS
// *should* compile, but fail to compile tests for example as of 2020-03-06 // *should* compile, but fail to compile tests for example as of 2020-03-06
#[cfg(not(target_os = "linux"))] #[cfg(not(target_os = "linux"))]
compile_error!( compile_error!(
"`hg-core` has only been tested on Linux and will most \ "`hg-core` has only been tested on Linux and will most \
likely not behave correctly on other platforms." likely not behave correctly on other platforms."
Path(HgPathError), Path(HgPathError),
UnsupportedSyntax(String), UnsupportedSyntax(String),
UnsupportedSyntaxInFile(String, String, usize), UnsupportedSyntaxInFile(String, String, usize),
TooLong(usize), TooLong(usize),
IO(std::io::Error), IO(std::io::Error),
/// Needed a pattern that can be turned into a regex but got one that /// Needed a pattern that can be turned into a regex but got one that
/// can't. This should only happen through programmer error. /// can't. This should only happen through programmer error.
NonRegexPattern(IgnorePattern), NonRegexPattern(IgnorePattern),
/// This is temporary, see `re2/mod.rs`.
/// This will cause a fallback to Python.
Re2NotInstalled,
} }
impl ToString for PatternError { impl ToString for PatternError {
fn to_string(&self) -> String { fn to_string(&self) -> String {
match self { match self {
PatternError::UnsupportedSyntax(syntax) => { PatternError::UnsupportedSyntax(syntax) => {
format!("Unsupported syntax {}", syntax) format!("Unsupported syntax {}", syntax)
} }
PatternError::UnsupportedSyntaxInFile(syntax, file_path, line) => { PatternError::UnsupportedSyntaxInFile(syntax, file_path, line) => {
format!( format!(
"{}:{}: unsupported syntax {}", "{}:{}: unsupported syntax {}",
file_path, line, syntax file_path, line, syntax
) )
} }
PatternError::TooLong(size) => { PatternError::TooLong(size) => {
format!("matcher pattern is too long ({} bytes)", size) format!("matcher pattern is too long ({} bytes)", size)
} }
PatternError::IO(e) => e.to_string(), PatternError::IO(e) => e.to_string(),
PatternError::Path(e) => e.to_string(), PatternError::Path(e) => e.to_string(),
PatternError::NonRegexPattern(pattern) => { PatternError::NonRegexPattern(pattern) => {
format!("'{:?}' cannot be turned into a regex", pattern) format!("'{:?}' cannot be turned into a regex", pattern)
} }
PatternError::Re2NotInstalled => {
"Re2 is not installed, cannot use regex functionality."
.to_string()
}
} }
} }
} }
impl From<DirstateMapError> for DirstateError { impl From<DirstateMapError> for DirstateError {
fn from(e: DirstateMapError) -> Self { fn from(e: DirstateMapError) -> Self {
DirstateError::Map(e) DirstateError::Map(e)
} }

rust/hg-core/src/matchers.rs

// matchers.rs // matchers.rs
// //
// Copyright 2019 Raphaël Gomès <rgomes@octobus.net> // Copyright 2019 Raphaël Gomès <rgomes@octobus.net>
// //
// This software may be used and distributed according to the terms of the // This software may be used and distributed according to the terms of the
// GNU General Public License version 2 or any later version. // GNU General Public License version 2 or any later version.
//! Structs and types for matching files and directories. //! Structs and types for matching files and directories.
#[cfg(feature = "with-re2")]
use crate::re2::Re2;
use crate::{ use crate::{
dirstate::dirs_multiset::DirsChildrenMultiset, dirstate::dirs_multiset::DirsChildrenMultiset,
filepatterns::{ filepatterns::{
build_single_regex, filter_subincludes, get_patterns_from_file, build_single_regex, filter_subincludes, get_patterns_from_file,
PatternFileWarning, PatternResult, SubInclude, PatternFileWarning, PatternResult, SubInclude,
}, },
utils::{ utils::{
files::find_dirs, files::find_dirs,
false false
} }
fn is_exact(&self) -> bool { fn is_exact(&self) -> bool {
true true
} }
} }
/// Matches files that are included in the ignore rules. /// Matches files that are included in the ignore rules.
#[cfg_attr( /// ```
feature = "with-re2", /// use hg::{
doc = r##" /// matchers::{IncludeMatcher, Matcher},
``` /// IgnorePattern,
use hg::{ /// PatternSyntax,
matchers::{IncludeMatcher, Matcher}, /// utils::hg_path::HgPath
IgnorePattern, /// };
PatternSyntax, /// use std::path::Path;
utils::hg_path::HgPath /// ///
}; /// let ignore_patterns =
use std::path::Path; /// vec![IgnorePattern::new(PatternSyntax::RootGlob, b"this*", Path::new(""))];
/// /// let (matcher, _) = IncludeMatcher::new(ignore_patterns, "").unwrap();
let ignore_patterns = /// ///
vec![IgnorePattern::new(PatternSyntax::RootGlob, b"this*", Path::new(""))]; /// assert_eq!(matcher.matches(HgPath::new(b"testing")), false);
let (matcher, _) = IncludeMatcher::new(ignore_patterns, "").unwrap(); /// assert_eq!(matcher.matches(HgPath::new(b"this should work")), true);
/// /// assert_eq!(matcher.matches(HgPath::new(b"this also")), true);
assert_eq!(matcher.matches(HgPath::new(b"testing")), false); /// assert_eq!(matcher.matches(HgPath::new(b"but not this")), false);
assert_eq!(matcher.matches(HgPath::new(b"this should work")), true); /// ```
assert_eq!(matcher.matches(HgPath::new(b"this also")), true);
assert_eq!(matcher.matches(HgPath::new(b"but not this")), false);
```
"##
)]
pub struct IncludeMatcher<'a> { pub struct IncludeMatcher<'a> {
patterns: Vec<u8>, patterns: Vec<u8>,
match_fn: Box<dyn for<'r> Fn(&'r HgPath) -> bool + 'a + Sync>, match_fn: Box<dyn for<'r> Fn(&'r HgPath) -> bool + 'a + Sync>,
/// Whether all the patterns match a prefix (i.e. recursively) /// Whether all the patterns match a prefix (i.e. recursively)
prefix: bool, prefix: bool,
roots: HashSet<HgPathBuf>, roots: HashSet<HgPathBuf>,
dirs: HashSet<HgPathBuf>, dirs: HashSet<HgPathBuf>,
parents: HashSet<HgPathBuf>, parents: HashSet<HgPathBuf>,
false false
} }
fn is_exact(&self) -> bool { fn is_exact(&self) -> bool {
false false
} }
} }
#[cfg(feature = "with-re2")]
/// Returns a function that matches an `HgPath` against the given regex
/// pattern.
///
/// This can fail when the pattern is invalid or not supported by the
/// underlying engine `Re2`, for instance anything with back-references.
#[timed]
fn re_matcher(
pattern: &[u8],
) -> PatternResult<impl Fn(&HgPath) -> bool + Sync> {
let regex = Re2::new(pattern);
let regex = regex.map_err(|e| PatternError::UnsupportedSyntax(e))?;
Ok(move |path: &HgPath| regex.is_match(path.as_bytes()))
}
#[cfg(not(feature = "with-re2"))]
/// Returns a function that matches an `HgPath` against the given regex /// Returns a function that matches an `HgPath` against the given regex
/// pattern. /// pattern.
/// ///
/// This can fail when the pattern is invalid or not supported by the /// This can fail when the pattern is invalid or not supported by the
/// underlying engine (the `regex` crate), for instance anything with /// underlying engine (the `regex` crate), for instance anything with
/// back-references. /// back-references.
#[timed] #[timed]
fn re_matcher( fn re_matcher(
VisitChildrenSet::Empty VisitChildrenSet::Empty
); );
assert_eq!( assert_eq!(
matcher.visit_children_set(HgPath::new(b"folder")), matcher.visit_children_set(HgPath::new(b"folder")),
VisitChildrenSet::Empty VisitChildrenSet::Empty
); );
} }
#[cfg(feature = "with-re2")]
#[test] #[test]
fn test_includematcher() { fn test_includematcher() {
// VisitchildrensetPrefix // VisitchildrensetPrefix
let (matcher, _) = IncludeMatcher::new( let (matcher, _) = IncludeMatcher::new(
vec![IgnorePattern::new( vec![IgnorePattern::new(
PatternSyntax::RelPath, PatternSyntax::RelPath,
b"dir/subdir", b"dir/subdir",
Path::new(""), Path::new(""),

rust/hg-core/src/re2/mod.rs

  • This file was deleted.
/// re2 module
///
/// The Python implementation of Mercurial uses the Re2 regex engine when
/// possible and if the bindings are installed, falling back to Python's `re`
/// in case of unsupported syntax (Re2 is a non-backtracking engine).
///
/// Using it from Rust is not ideal. We need C++ bindings, a C++ compiler,
/// Re2 needs to be installed... why not just use the `regex` crate?
///
/// Using Re2 from the Rust implementation guarantees backwards compatibility.
/// We know it will work out of the box without needing to figure out the
/// subtle differences in syntax. For example, `regex` currently does not
/// support empty alternations (regex like `a||b`) which happens more often
/// than we might think. Old benchmarks also showed worse performance from
/// regex than with Re2, but the methodology and results were lost, so take
/// this with a grain of salt.
///
/// The idea is to use Re2 for now as a temporary phase and then investigate
/// how much work would be needed to use `regex`.
mod re2;
pub use re2::Re2;

rust/hg-core/src/re2/re2.rs

  • This file was deleted.
/*
re2.rs
Rust FFI bindings to Re2.
Copyright 2020 Valentin Gatien-Baron
This software may be used and distributed according to the terms of the
GNU General Public License version 2 or any later version.
*/
use libc::{c_int, c_void};
type Re2Ptr = *const c_void;
pub struct Re2(Re2Ptr);
/// `re2.h` says:
/// "An "RE2" object is safe for concurrent use by multiple threads."
unsafe impl Sync for Re2 {}
/// These bind to the C ABI in `rust_re2.cpp`.
extern "C" {
fn rust_re2_create(data: *const u8, len: usize) -> Re2Ptr;
fn rust_re2_destroy(re2: Re2Ptr);
fn rust_re2_ok(re2: Re2Ptr) -> bool;
fn rust_re2_error(
re2: Re2Ptr,
outdata: *mut *const u8,
outlen: *mut usize,
) -> bool;
fn rust_re2_match(
re2: Re2Ptr,
data: *const u8,
len: usize,
anchor: c_int,
) -> bool;
}
impl Re2 {
pub fn new(pattern: &[u8]) -> Result<Re2, String> {
unsafe {
let re2 = rust_re2_create(pattern.as_ptr(), pattern.len());
if rust_re2_ok(re2) {
Ok(Re2(re2))
} else {
let mut data: *const u8 = std::ptr::null();
let mut len: usize = 0;
rust_re2_error(re2, &mut data, &mut len);
Err(String::from_utf8_lossy(std::slice::from_raw_parts(
data, len,
))
.to_string())
}
}
}
pub fn is_match(&self, data: &[u8]) -> bool {
unsafe { rust_re2_match(self.0, data.as_ptr(), data.len(), 1) }
}
}
impl Drop for Re2 {
fn drop(&mut self) {
unsafe { rust_re2_destroy(self.0) }
}
}

rust/hg-core/src/re2/rust_re2.cpp

  • This file was deleted.
/*
rust_re2.cpp
C ABI export of Re2's C++ interface for Rust FFI.
Copyright 2020 Valentin Gatien-Baron
This software may be used and distributed according to the terms of the
GNU General Public License version 2 or any later version.
*/
#include <re2/re2.h>
using namespace re2;
extern "C" {
RE2* rust_re2_create(const char* data, size_t len) {
RE2::Options o;
o.set_encoding(RE2::Options::Encoding::EncodingLatin1);
o.set_log_errors(false);
o.set_max_mem(50000000);
return new RE2(StringPiece(data, len), o);
}
void rust_re2_destroy(RE2* re) {
delete re;
}
bool rust_re2_ok(RE2* re) {
return re->ok();
}
void rust_re2_error(RE2* re, const char** outdata, size_t* outlen) {
const std::string& e = re->error();
*outdata = e.data();
*outlen = e.length();
}
bool rust_re2_match(RE2* re, char* data, size_t len, int ianchor) {
const StringPiece sp = StringPiece(data, len);
RE2::Anchor anchor =
ianchor == 0 ? RE2::Anchor::UNANCHORED :
(ianchor == 1 ? RE2::Anchor::ANCHOR_START :
RE2::Anchor::ANCHOR_BOTH);
return re->Match(sp, 0, len, anchor, NULL, 0);
}
}

rust/hg-cpython/Cargo.toml

[package] [package]
name = "hg-cpython" name = "hg-cpython"
version = "0.1.0" version = "0.1.0"
authors = ["Georges Racinet <gracinet@anybox.fr>"] authors = ["Georges Racinet <gracinet@anybox.fr>"]
edition = "2018" edition = "2018"
[lib] [lib]
name='rusthg' name='rusthg'
crate-type = ["cdylib"] crate-type = ["cdylib"]
[features] [features]
default = ["python27"] default = ["python27"]
with-re2 = ["hg-core/with-re2"]
# Features to build an extension module: # Features to build an extension module:
python27 = ["cpython/python27-sys", "cpython/extension-module-2-7"] python27 = ["cpython/python27-sys", "cpython/extension-module-2-7"]
python3 = ["cpython/python3-sys", "cpython/extension-module"] python3 = ["cpython/python3-sys", "cpython/extension-module"]
# Enable one of these features to build a test executable linked to libpython: # Enable one of these features to build a test executable linked to libpython:
# e.g. cargo test --no-default-features --features python27-bin # e.g. cargo test --no-default-features --features python27-bin
python27-bin = ["cpython/python27-sys"] python27-bin = ["cpython/python27-sys"]

rust/hg-cpython/src/debug.rs

/// Create the module, with `__package__` given from parent /// Create the module, with `__package__` given from parent
pub fn init_module(py: Python, package: &str) -> PyResult<PyModule> { pub fn init_module(py: Python, package: &str) -> PyResult<PyModule> {
let dotted_name = &format!("{}.debug", package); let dotted_name = &format!("{}.debug", package);
let m = PyModule::new(py, dotted_name)?; let m = PyModule::new(py, dotted_name)?;
m.add(py, "__package__", package)?; m.add(py, "__package__", package)?;
m.add(py, "__doc__", "Rust debugging information")?; m.add(py, "__doc__", "Rust debugging information")?;
m.add(py, "re2_installed", cfg!(feature = "with-re2"))?;
let sys = PyModule::import(py, "sys")?; let sys = PyModule::import(py, "sys")?;
let sys_modules: PyDict = sys.get(py, "modules")?.extract(py)?; let sys_modules: PyDict = sys.get(py, "modules")?.extract(py)?;
sys_modules.set_item(py, dotted_name, &m)?; sys_modules.set_item(py, dotted_name, &m)?;
Ok(m) Ok(m)
} }

tests/test-install.t

checking Mercurial version (*) (glob) checking Mercurial version (*) (glob)
checking Mercurial custom build (*) (glob) checking Mercurial custom build (*) (glob)
checking module policy (*) (glob) checking module policy (*) (glob)
checking installed modules (*mercurial)... (glob) checking installed modules (*mercurial)... (glob)
checking registered compression engines (*zlib*) (glob) checking registered compression engines (*zlib*) (glob)
checking available compression engines (*zlib*) (glob) checking available compression engines (*zlib*) (glob)
checking available compression engines for wire protocol (*zlib*) (glob) checking available compression engines for wire protocol (*zlib*) (glob)
checking "re2" regexp engine \((available|missing)\) (re) checking "re2" regexp engine \((available|missing)\) (re)
checking "re2" regexp engine Rust bindings \((installed|missing)\) (re) (rust !)
checking templates (*mercurial?templates)... (glob) checking templates (*mercurial?templates)... (glob)
checking default template (*mercurial?templates?map-cmdline.default) (glob) checking default template (*mercurial?templates?map-cmdline.default) (glob)
checking commit editor... (*) (glob) checking commit editor... (*) (glob)
checking username (test) checking username (test)
no problems detected no problems detected
hg debuginstall JSON hg debuginstall JSON
$ hg debuginstall -Tjson | sed 's|\\\\|\\|g' $ hg debuginstall -Tjson | sed 's|\\\\|\\|g'
checking Mercurial version (*) (glob) checking Mercurial version (*) (glob)
checking Mercurial custom build (*) (glob) checking Mercurial custom build (*) (glob)
checking module policy (*) (glob) checking module policy (*) (glob)
checking installed modules (*mercurial)... (glob) checking installed modules (*mercurial)... (glob)
checking registered compression engines (*zlib*) (glob) checking registered compression engines (*zlib*) (glob)
checking available compression engines (*zlib*) (glob) checking available compression engines (*zlib*) (glob)
checking available compression engines for wire protocol (*zlib*) (glob) checking available compression engines for wire protocol (*zlib*) (glob)
checking "re2" regexp engine \((available|missing)\) (re) checking "re2" regexp engine \((available|missing)\) (re)
checking "re2" regexp engine Rust bindings \((installed|missing)\) (re) (rust !)
checking templates (*mercurial?templates)... (glob) checking templates (*mercurial?templates)... (glob)
checking default template (*mercurial?templates?map-cmdline.default) (glob) checking default template (*mercurial?templates?map-cmdline.default) (glob)
checking commit editor... (*) (glob) checking commit editor... (*) (glob)
checking username... checking username...
no username supplied no username supplied
(specify a username in your configuration file) (specify a username in your configuration file)
1 problems detected, please check your install! 1 problems detected, please check your install!
[1] [1]
checking Mercurial version (*) (glob) checking Mercurial version (*) (glob)
checking Mercurial custom build (*) (glob) checking Mercurial custom build (*) (glob)
checking module policy (*) (glob) checking module policy (*) (glob)
checking installed modules (*mercurial)... (glob) checking installed modules (*mercurial)... (glob)
checking registered compression engines (*zlib*) (glob) checking registered compression engines (*zlib*) (glob)
checking available compression engines (*zlib*) (glob) checking available compression engines (*zlib*) (glob)
checking available compression engines for wire protocol (*zlib*) (glob) checking available compression engines for wire protocol (*zlib*) (glob)
checking "re2" regexp engine \((available|missing)\) (re) checking "re2" regexp engine \((available|missing)\) (re)
checking "re2" regexp engine Rust bindings \((installed|missing)\) (re) (rust !)
checking templates (*mercurial?templates)... (glob) checking templates (*mercurial?templates)... (glob)
checking default template (*mercurial?templates?map-cmdline.default) (glob) checking default template (*mercurial?templates?map-cmdline.default) (glob)
checking commit editor... ($TESTTMP/tools/testeditor.exe) checking commit editor... ($TESTTMP/tools/testeditor.exe)
checking username (test) checking username (test)
no problems detected no problems detected
print out the binary post-shlexsplit in the error message when commit editor is print out the binary post-shlexsplit in the error message when commit editor is
not found (this is intentionally using backslashes to mimic a windows usecase). not found (this is intentionally using backslashes to mimic a windows usecase).
checking Mercurial version (*) (glob) checking Mercurial version (*) (glob)
checking Mercurial custom build (*) (glob) checking Mercurial custom build (*) (glob)
checking module policy (*) (glob) checking module policy (*) (glob)
checking installed modules (*mercurial)... (glob) checking installed modules (*mercurial)... (glob)
checking registered compression engines (*zlib*) (glob) checking registered compression engines (*zlib*) (glob)
checking available compression engines (*zlib*) (glob) checking available compression engines (*zlib*) (glob)
checking available compression engines for wire protocol (*zlib*) (glob) checking available compression engines for wire protocol (*zlib*) (glob)
checking "re2" regexp engine \((available|missing)\) (re) checking "re2" regexp engine \((available|missing)\) (re)
checking "re2" regexp engine Rust bindings \((installed|missing)\) (re) (rust !)
checking templates (*mercurial?templates)... (glob) checking templates (*mercurial?templates)... (glob)
checking default template (*mercurial?templates?map-cmdline.default) (glob) checking default template (*mercurial?templates?map-cmdline.default) (glob)
checking commit editor... (c:\foo\bar\baz.exe) (windows !) checking commit editor... (c:\foo\bar\baz.exe) (windows !)
Can't find editor 'c:\foo\bar\baz.exe' in PATH (windows !) Can't find editor 'c:\foo\bar\baz.exe' in PATH (windows !)
checking commit editor... (c:foobarbaz.exe) (no-windows !) checking commit editor... (c:foobarbaz.exe) (no-windows !)
Can't find editor 'c:foobarbaz.exe' in PATH (no-windows !) Can't find editor 'c:foobarbaz.exe' in PATH (no-windows !)
(specify a commit editor in your configuration file) (specify a commit editor in your configuration file)
checking username (test) checking username (test)
checking Mercurial version (*) (glob) checking Mercurial version (*) (glob)
checking Mercurial custom build (*) (glob) checking Mercurial custom build (*) (glob)
checking module policy (*) (glob) checking module policy (*) (glob)
checking installed modules (*/mercurial)... (glob) checking installed modules (*/mercurial)... (glob)
checking registered compression engines (*) (glob) checking registered compression engines (*) (glob)
checking available compression engines (*) (glob) checking available compression engines (*) (glob)
checking available compression engines for wire protocol (*) (glob) checking available compression engines for wire protocol (*) (glob)
checking "re2" regexp engine \((available|missing)\) (re) checking "re2" regexp engine \((available|missing)\) (re)
checking "re2" regexp engine Rust bindings \((installed|missing)\) (re) (rust !)
checking templates ($TESTTMP/installenv/*/site-packages/mercurial/templates)... (glob) checking templates ($TESTTMP/installenv/*/site-packages/mercurial/templates)... (glob)
checking default template ($TESTTMP/installenv/*/site-packages/mercurial/templates/map-cmdline.default) (glob) checking default template ($TESTTMP/installenv/*/site-packages/mercurial/templates/map-cmdline.default) (glob)
checking commit editor... (*) (glob) checking commit editor... (*) (glob)
checking username (test) checking username (test)
no problems detected no problems detected
#endif #endif
#if no-py3 virtualenv #if no-py3 virtualenv
checking Mercurial version (*) (glob) checking Mercurial version (*) (glob)
checking Mercurial custom build (*) (glob) checking Mercurial custom build (*) (glob)
checking module policy (*) (glob) checking module policy (*) (glob)
checking installed modules (*/mercurial)... (glob) checking installed modules (*/mercurial)... (glob)
checking registered compression engines (*) (glob) checking registered compression engines (*) (glob)
checking available compression engines (*) (glob) checking available compression engines (*) (glob)
checking available compression engines for wire protocol (*) (glob) checking available compression engines for wire protocol (*) (glob)
checking "re2" regexp engine \((available|missing)\) (re) checking "re2" regexp engine \((available|missing)\) (re)
checking "re2" regexp engine Rust bindings \((installed|missing)\) (re) (rust !)
checking templates ($TESTTMP/installenv/*/site-packages/mercurial/templates)... (glob) checking templates ($TESTTMP/installenv/*/site-packages/mercurial/templates)... (glob)
checking default template ($TESTTMP/installenv/*/site-packages/mercurial/templates/map-cmdline.default) (glob) checking default template ($TESTTMP/installenv/*/site-packages/mercurial/templates/map-cmdline.default) (glob)
checking commit editor... (*) (glob) checking commit editor... (*) (glob)
checking username (test) checking username (test)
no problems detected no problems detected
#endif #endif
rust/hg-core/src/re2/re2.rs