Page MenuHomePhabricator

examplerust: add an example extension written in Rust
AbandonedPublic

Authored by quark on Sep 13 2017, 12:41 PM.

Details

Reviewers
jsgf
mbthomas
Group Reviewers
Restricted Project
Summary

The example extension doesn't do much, but serves as an example of how
Mercurial extensions may be written in Rust.

Test Plan

Build and run the extension. It doesn't integrate with the main build
as it is not a proper extension.

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Branch
default
Lint
Lint OK
Unit
Unit Tests OK

Event Timeline

mbthomas created this revision.Sep 13 2017, 12:41 PM
mbthomas updated this revision to Diff 1784.Sep 13 2017, 12:44 PM

Fix indentation

durin42 added inline comments.
examplerust/README.md
15

A couple of thoughts:

nightly rust is probably not going to fly for distro-packaging reasons (eg the Debian I need to target at work is giving me rustc 1.14, so selfishly I'd like to stick to that)

https://github.com/mitsuhiko/snaek/ might be better than pyo3, as it'll be portable to non-cpython Pythons)

quark added a subscriber: quark.Sep 13 2017, 1:50 PM
quark added inline comments.
examplerust/README.md
15

I personally prefer native CPython bindings for 2 reasons:

  • ffi is not distributed with Python. CPython Mercurial does not depend on any 3rd-party Python library yet.
  • Performance. Since we use Rust for its performance, and the default hg distribution uses CPython, there is no reason to slow default UX down, even a little bit.

I agree nightly is an issue. If we can find a way to workaround around that, I'd still prefer native CPython solutions.

durin42 added inline comments.Sep 13 2017, 1:52 PM
examplerust/README.md
15

I don't want to make pypy worse though (and cpyext on pypy is *awful*), and your choices are to:

  1. not use cpython's API
  2. maintain a C version of anything that's in Rust
  3. spend time optimizing the pure-python code so that pypy isn't penalized

1 seems easy, and I doubt I can sell anyone on 2 or 3. :)

akushner added subscribers: jsgf, akushner.

@jsgf - I'm sure you have some thoughts here.

mbthomas added inline comments.Sep 13 2017, 2:08 PM
examplerust/README.md
15

snaek (and ffi in general) also doesn't let you construct Python objects in Rust that you can return to the Python runtime. You're limited to passing primitives and C structures between the two languages. If you want to have a Rust version of a Python object you have to construct the Python object in a Python wrapper and marshal the arguments for each method call manually.

I'm going to investigate exactly which bits of nightly PyO3 needs and why. There is also rust-cpython, of which PyO3 is a fork, which I think uses older techniques for describing the python interface, making it clumsier to use but perhaps more compatible with Rust stable.

quark added inline comments.Sep 13 2017, 2:08 PM
examplerust/README.md
15

I think:

  1. PyPy could often optimize Python code to be as fast as native code. In this case, it does not even need C or Rust.
  2. If writing bridge code is easy, then it seems fine to define proper APIs with multiple bridges.

I don't think we must limit the bridge choice to just one.

mbthomas updated this revision to Diff 1814.Sep 14 2017, 6:23 AM

Use rust-cpython to interface between Rust and Python

I've re-implemented the example using rust-cpython instead of PyO3. This works on the stable compiler. There are a few drawbacks using this interface:

  • The code to create modules and classes is a bit more verbose.
  • Functions that are interfacing with Python must pass round an instance of a Python marker object that represents the holding of the GIL. (PyO3 uses this too, but the macro magic hides it).
  • Function signatures are more limited. Only simple declarations with a fixed number of parameters are supported. If you want to use variable arguments, keyword arguments, or default values then you have to implement them yourself. I've included an example of how this is done.

Since PyO3 is based on rust-cpython, there are considerable similarities. If we can limit the interface code to just the glue between Rust and Python, i.e. avoid polluting the Rust code with too many Python concepts, then it may be possible to start with rust-cpython and migrate to PyO3 once the compiler features it relies on are stabilised.

I checked that the rust-cpython version also works on 1.14.0 as well as current stable (1.20.0).

jsgf added a comment.Sep 18 2017, 6:36 PM

I don't have any particular opinion about which Python binding to use - they all seem a bit rough. I think there's some interesting scope for doing things like using custom derive to automate converting between Rust and Python representations, and iterators/futures/generators to do the conversions of large structures lazily.

I'm completely on board with restricting Rust to a stable version. But constraining Rust to 1.14 is unreasonably old - that's getting on for a year old, which 1/3rd of Rust's total life. crates.io 3rd-party packages also tend to update to use new language features quickly, and there's no way to set a Cargo.toml constraint on a particular version of a the language, so you'd need to pin package versions to language-version compatible ones manually. Personally I'd try to keep a window of no more than 2 or 3 stable releases (ie, 1.18, 1.19, 1.20 as of today).

I like the explicitness of passing around a Python object. In principle if the underlying runtime doesn't have a GIL (ie, PyPy?) then you could have multiple instances of it for parallelism. If it's stashed away in some secret state, then you're stuck with GIL limitations. (But perhaps this is too hypothetical to be worth considering.)

In D698#12205, @jsgf wrote:

I'm completely on board with restricting Rust to a stable version. But constraining Rust to 1.14 is unreasonably old - that's getting on for a year old, which 1/3rd of Rust's total life. crates.io 3rd-party packages also tend to update to use new language features quickly, and there's no way to set a Cargo.toml constraint on a particular version of a the language, so you'd need to pin package versions to language-version compatible ones manually. Personally I'd try to keep a window of no more than 2 or 3 stable releases (ie, 1.18, 1.19, 1.20 as of today).

I can't see that keeping Debian releases possible. In my view if "no more than 2 or 3 stable releases" is a hard requirement for rust, we may as well stop having the discussion now.

(Alternatively: convince me with documented success cases that binaries built by a newer Rust than is in a given Debian release are somehow plausible and acceptable.)

sid0 added a subscriber: sid0.EditedSep 18 2017, 8:03 PM

I can't see that keeping Debian releases possible. In my view if "no more than 2 or 3 stable releases" is a hard requirement for rust, we may as well stop having the discussion now.
(Alternatively: convince me with documented success cases that binaries built by a newer Rust than is in a given Debian release are somehow plausible and acceptable.)

So So I dug into this a bit, and it turns out that Debian stable doesn't even have cargo. Any third-party dependencies we vendor in will need cargo for their build system.

If a dependency requires a build system that Debian stable doesn't have, what do we do?

In my opinion this is a fundamental limitation in the Debian stable model. I really don't think we should hold ourselves back because of it. People that opt into it are opting into many, many tradeoffs (like not even getting the latest version of their web browser -- see link above for details), and IMO getting a slower Mercurial experience would be just another tradeoff.

sid0 added a comment.Sep 18 2017, 8:11 PM

And if the concern is about deb packages that we provide -- Rust links its runtime statically I believe.

jsgf added a comment.Sep 22 2017, 3:55 PM
In D698#12222, @durin42 wrote:

I can't see that keeping Debian releases possible. In my view if "no more than 2 or 3 stable releases" is a hard requirement for rust, we may as well stop having the discussion now.
(Alternatively: convince me with documented success cases that binaries built by a newer Rust than is in a given Debian release are somehow plausible and acceptable.)

Rust artifacts are compiled in a self-contained way - executables are statically linked with all their deps except for things like libc/libpthread, likewise shared objects contain all their Rust dependencies. Dynamic linking Rust crates is extremely unusual and only for special cases. Rust's external ABI dependencies are straightforward: it only needs things like malloc/free and other low-level functions. When you download Rust precompiled with rustup, the executables are built against old libc ABIs to guarantee binary back-compatibility with older distros, and they in turn generate similarly backwards-compat binaries.

In other words, the intent is that current Rust executables should work on Debian as-is, and upstream Rust would consider it a bug if that weren't the case. ABI compatibility with Python adds complexity to that, but that's independent of Rust as it also applies to C or any other compiled language.

The Rust project considers cargo to be an integral part of the Rust toolchain, and almost consider it to be part of the language itself. If Debian isn't shipping cargo they're not shipping a Rust toolchain, so the question of whether to support the rustc version they package is moot.

quark added a reviewer: mbthomas.EditedNov 29 2017, 3:34 PM
quark commandeered this revision.

Commandeering to remove it from Yadda queue. fb-hgext has been using rust-cpython already.

quark abandoned this revision.Nov 29 2017, 3:34 PM