This is an archive of the discontinued Mercurial Phabricator instance.

Differential D1411

distutils_rust: add distutils extension to compile Rust extension modules
ClosedPublic

Authored by mbthomas on Nov 14 2017, 12:39 PM.

Details

Reviewers
quark
durham
Group Reviewers
Restricted Project
Commits
rFBHGXb77a9dc92b31: distutils_rust: add distutils extension to compile Rust extension modules
Summary

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

mbthomas created this revision.Nov 14 2017, 12:39 PM
Herald added a reviewer: Restricted Project. · View Herald TranscriptNov 14 2017, 12:39 PM
quark accepted this revision.Nov 18 2017, 12:32 AM
quark added a subscriber: quark.

Maybe also add from __future__ import absolute_import stuff. But otherwise looks pretty good to me.

distutils_rust/__init__.py
56

Should this be None or False?

This revision is now accepted and ready to land.Nov 18 2017, 12:32 AM
mbthomas updated this revision to Diff 3662.Nov 20 2017, 12:21 PM
durham accepted this revision.Nov 21 2017, 12:13 PM
durham added a subscriber: durham.

freeshrugs

indygreg added a subscriber: indygreg.Nov 21 2017, 4:42 PM

Cool series! I was curious what your plans are to support Windows. That's the part that has been tripping me up in my quest to integrate Rust into the core Mercurial project. Keep in mind that official distributions of Python 2.7 are compiled with MSVC 2008 and link against its CRT. So there are CRT implications depending on how the Rust code is built.

Also, what about PyPy and other non-CPython distributions? Is it necessary to use Python extensions (as opposed to ctypes or cffi) for performance?

quark added a comment.EditedNov 22 2017, 12:38 AM
In D1411#24941, @indygreg wrote:

Also, what about PyPy and other non-CPython distributions? Is it necessary to use Python extensions (as opposed to ctypes or cffi) for performance?

From my experience, it's not only about performance but also code cleanness. I tried the C API + Cython (could replace Cython with cffi) approach. Error handling is much less pleasant (Rust Err -> C special return value -> Python exception) than just implementing Into<PyErr>.

mbthomas updated this revision to Diff 3832.Nov 24 2017, 11:53 AM
mbthomas added a comment.Nov 24 2017, 3:58 PM
In D1411#24941, @indygreg wrote:

I was curious what your plans are to support Windows. That's the part that has been tripping me up in my quest to integrate Rust into the core Mercurial project. Keep in mind that official distributions of Python 2.7 are compiled with MSVC 2008 and link against its CRT. So there are CRT implications depending on how the Rust code is built.

The plan is to support Windows with the caveat that compatibly compiled Python interpreter is used. It's fairly commonplace on the Windows platform to ship programs with their interpreter precompiled alongside the source, so we're expecting that to be acceptable for Windows users. We don't have any details yet, however.

durham added a comment.Nov 27 2017, 1:06 PM

Also, we ship our windows version of Mercurial with a Python 2.7 that has been built against MSVC 2015. Not sure if that helps.

Closed by commit rFBHGXb77a9dc92b31: distutils_rust: add distutils extension to compile Rust extension modules (authored by mbthomas). · Explain WhyNov 28 2017, 6:02 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
AMdistutils_rust/__init__.py (118 lines)

Diff 3940

distutils_rust/__init__.py

  • This file was added.
# distutils_rust.py - distutils extension for building Rust extension modules
#
# Copyright 2017 Facebook, Inc.
from __future__ import absolute_import
from distutils.core import Command
from distutils.dist import Distribution
from distutils.errors import CompileError
from distutils import log
from distutils.command.build import build
import distutils.util
import os
import shutil
import subprocess
class RustExtension(object):
"""Data for a Rust extension.
'name' is the name of the extension, and must match the 'name' in the Cargo
manifest. This will also be the name of the python module that is
produced.
'package' is the name of the python package into which the compiled
extension will be placed. If none, the extension will be placed in the
root package.
'manifest' is the path to the Cargo.toml file for the Rust project.
"""
def __init__(self, name, package=None, manifest=None):
self.name = name
self.package = package
self.manifest = manifest or 'Cargo.toml'
class BuildRustExt(Command):
description = "build Rust extensions (compile/link to build directory)"
user_options = [
('build-lib=', 'b',
"directory for compiled extension modules"),
('build-temp=', 't',
"directory for temporary files (build by-products)"),
('debug', 'g',
"compile in debug mode"),
('inplace', 'i',
"ignore build-lib and put compiled extensions into the source " +
"directory alongside your pure Python modules"),
]
boolean_options = ['debug', 'inplace']
def initialize_options(self):
self.build_lib = None
self.build_temp = None
self.debug = None
self.inplace = None
quarkUnsubmitted
Not Done

Should this be None or False?

quark: Should this be `None` or `False`?
def finalize_options(self):
self.set_undefined_options('build',
('build_lib', 'build_lib'),
('build_temp', 'build_temp'),
('debug', 'debug'),
)
def run(self):
for ext in self.distribution.rust_ext_modules:
self.build_ext(ext)
def get_temp_path(self, ext):
"""Returns the path of the temporary directory to build in."""
return os.path.join(self.build_temp,
os.path.dirname(ext.manifest),
'target')
def get_temp_output(self, ext):
"""Returns the location in the temp directory of the output file."""
platform = distutils.util.get_platform()
if platform.startswith('macosx'):
libsuffix = '.dylib'
else:
libsuffix = '.so'
return os.path.join('debug' if self.debug else 'release',
'lib' + ext.name + libsuffix)
def get_output_filename(self, ext):
"""Returns the filename of the build output."""
if self.inplace:
# the inplace option requires to find the package directory
# using the build_py command for that
build_py = self.get_finalized_command('build_py')
package_dir = build_py.get_package_dir(ext.package)
else:
package_dir = os.path.join(self.build_lib, *ext.package.split('.'))
return os.path.join(package_dir, ext.name + '.so')
def build_ext(self, ext):
log.info("building '%s' extension", ext.name)
cmd = ['cargo', 'build', '--manifest-path', ext.manifest]
if not self.debug:
cmd.append('--release')
env = os.environ.copy()
env['CARGO_TARGET_DIR'] = self.get_temp_path(ext)
rc = subprocess.call(cmd, env=env)
if rc:
raise CompileError(
"compilation of Rust extension '%s' failed" % ext.name)
src = os.path.join(self.get_temp_path(ext), self.get_temp_output(ext))
dest = self.get_output_filename(ext)
shutil.copyfile(src, dest)
shutil.copymode(src, dest)
Distribution.rust_ext_modules = ()
build.sub_commands.append(
('build_rust_ext', lambda self: bool(self.distribution.rust_ext_modules)))