This is an archive of the discontinued Mercurial Phabricator instance.

Differential D6923

setup: build extensions in parallel by default
ClosedPublic

Authored by indygreg on Sep 30 2019, 8:26 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHGf9d35f01b8b3: setup: build extensions in parallel by default
Summary

The build_ext distutils command in Python 3.5+ has a "parallel"
option that controls whether to build extensions in parallel. It
is disabled by default (None) and can be set to an integer value
for number of cores or True to indicate use all available CPU
cores.

This commit changes our build_ext command override to set
"parallel" to True unless a value has been provided by the caller.

On my machine, this makes python setup.py build_ext 1-4s faster.
It is worth noting that at this time, each individual source file
constituting the extension is still built serially. For Mercurial,
this means that we can't build faster than the slowest-to-build
extension, which is the zstd extension by a long shot. This means
that setup.py is still not very efficient at utilizing multiple
cores. But we're better than before.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Sep 30 2019, 8:26 PM
indygreg added a commit: rHGf9d35f01b8b3: setup: build extensions in parallel by default.Oct 3 2019, 11:56 PM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHGf9d35f01b8b3: setup: build extensions in parallel by default (authored by indygreg). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Msetup.py (8 lines)

Diff 16790

setup.py

negative_opt = buildextnegops negative_opt = buildextnegops
def initialize_options(self): def initialize_options(self):
self.zstd = True self.zstd = True
self.rust = True self.rust = True
return build_ext.initialize_options(self) return build_ext.initialize_options(self)
def finalize_options(self):
# Unless overridden by the end user, build extensions in parallel.
# Only influences behavior on Python 3.5+.
if getattr(self, 'parallel', None) is None:
self.parallel = True
return build_ext.finalize_options(self)
def build_extensions(self): def build_extensions(self):
ruststandalones = [e for e in self.extensions ruststandalones = [e for e in self.extensions
if isinstance(e, RustStandaloneExtension)] if isinstance(e, RustStandaloneExtension)]
self.extensions = [e for e in self.extensions self.extensions = [e for e in self.extensions
if e not in ruststandalones] if e not in ruststandalones]
# Filter out zstd if disabled via argument. # Filter out zstd if disabled via argument.
if not self.zstd: if not self.zstd:
self.extensions = [e for e in self.extensions self.extensions = [e for e in self.extensions