This is an archive of the discontinued Mercurial Phabricator instance.

Differential D8645

clonebundles: optional memory-requirement attribution
ClosedPublic

Authored by joerg.sonnenberger on Jun 21 2020, 6:40 PM.

Details

Reviewers
durin42
pulkit
Group Reviewers
hg-reviewers
Commits
rHG72feaeb510b3: clonebundles: optional memory-requirement attribution
rHG9c7ff8877592: clonebundles: optional memory-requirement attribution
Summary

The new REQUIREDRAM option allows a client to skip bundles it isn't
expected to handle well, e.g. without swapping. This allows a fallback
path to be provided e.g. using zstd level 10 instead of 22.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

durin42 accepted this revision as: durin42.Jun 25 2020, 1:30 PM
durin42 added a subscriber: durin42.

I'm a fan of this, but for default, not stable. Is there a specific reason you need a new feature on the stable branch?

joerg.sonnenberger added a comment.Jun 25 2020, 2:56 PM

No strong reason, it just was easier to start as base line.

pulkit added a subscriber: pulkit.Jul 7 2020, 2:10 PM
pulkit added inline comments.
mercurial/exchange.py
3038

need to add a constant describing what the magic 0.66 means, which can probably be configured by extensions?

joerg.sonnenberger added inline comments.Jul 7 2020, 2:35 PM
mercurial/exchange.py
3038

Just a guess how much memory we can safely use on a system without grinding to death due to swapping. It is not meant for customization, if it is necessary to manually override, the user can bump ui.available-memory appropiately.

pulkit accepted this revision.Jul 10 2020, 6:03 AM
This revision is now accepted and ready to land.Jul 10 2020, 6:03 AM

Revision Contents
Changeset List

PathPackages
Mmercurial/exchange.py (17 lines)
Mrelnotes/next (4 lines)
Mtests/test-clonebundles.t (85 lines)

Diff 21825

mercurial/exchange.py

continue continue
if b'REQUIRESNI' in entry and not sslutil.hassni: if b'REQUIRESNI' in entry and not sslutil.hassni:
repo.ui.debug( repo.ui.debug(
b'filtering %s because SNI not supported\n' % entry[b'URL'] b'filtering %s because SNI not supported\n' % entry[b'URL']
) )
continue continue
if b'REQUIREDRAM' in entry:
try:
requiredram = util.sizetoint(entry[b'REQUIREDRAM'])
except error.ParseError:
repo.ui.debug(
b'filtering %s due to a bad REQUIREDRAM attribute\n'
% entry[b'URL']
)
continue
actualram = repo.ui.estimatememory()
if actualram is not None and actualram * 0.66 < requiredram:
pulkitUnsubmitted
Not Done

need to add a constant describing what the magic 0.66 means, which can probably be configured by extensions?

pulkit: need to add a constant describing what the magic `0.66` means, which can probably be configured…
joerg.sonnenbergerAuthorUnsubmitted
Done

Just a guess how much memory we can safely use on a system without grinding to death due to swapping. It is not meant for customization, if it is necessary to manually override, the user can bump ui.available-memory appropiately.

joerg.sonnenberger: Just a guess how much memory we can safely use on a system without grinding to death due to…
repo.ui.debug(
b'filtering %s as it needs more than 2/3 of system memory\n'
% entry[b'URL']
)
continue
newentries.append(entry) newentries.append(entry)
return newentries return newentries
class clonebundleentry(object): class clonebundleentry(object):
"""Represents an item in a clone bundles manifest. """Represents an item in a clone bundles manifest.

relnotes/next

== New Features == == New Features ==
* clonebundles can be annotated with the expected memory requirements
using the `REQUIREDRAM` option. This allows clients to skip
bundles created with large zstd windows and fallback to larger, but
less demanding bundles.
== New Experimental Features == == New Experimental Features ==
* The core of some hg operations have been (and are being) * The core of some hg operations have been (and are being)
implemented in rust, for speed. `hg status` on a repository with implemented in rust, for speed. `hg status` on a repository with
300k tracked files goes from 1.8s to 0.6s for instance. 300k tracked files goes from 1.8s to 0.6s for instance.
This has currently been tested only on linux, and does not build on This has currently been tested only on linux, and does not build on
windows. See rust/README.rst in the mercurial repository for windows. See rust/README.rst in the mercurial repository for

tests/test-clonebundles.t

adding changesets adding changesets
adding manifests adding manifests
adding file changes adding file changes
added 2 changesets with 2 changes to 2 files added 2 changesets with 2 changes to 2 files
finished applying clone bundle finished applying clone bundle
searching for changes searching for changes
no changes found no changes found
2 local changesets published 2 local changesets published
$ killdaemons.py
A manifest with a gzip bundle requiring too much memory for a 16MB system and working
on a 32MB system.
$ "$PYTHON" $TESTDIR/dumbhttp.py -p $HGPORT1 --pid http.pid
$ cat http.pid >> $DAEMON_PIDS
$ hg -R server serve -d -p $HGPORT --pid-file hg.pid --accesslog access.log
$ cat hg.pid >> $DAEMON_PIDS
$ cat > server/.hg/clonebundles.manifest << EOF
> http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2 REQUIREDRAM=12MB
> EOF
$ hg clone -U --debug --config ui.available-memory=16MB http://localhost:$HGPORT gzip-too-large
using http://localhost:$HGPORT/
sending capabilities command
sending clonebundles command
filtering http://localhost:$HGPORT1/gz-a.hg as it needs more than 2/3 of system memory
no compatible clone bundles available on server; falling back to regular clone
(you may want to report this to the server operator)
query 1; heads
sending batch command
requesting all changes
sending getbundle command
bundle2-input-bundle: with-transaction
bundle2-input-part: "changegroup" (params: 1 mandatory 1 advisory) supported
adding changesets
add changeset 53245c60e682
add changeset aaff8d2ffbbf
adding manifests
adding file changes
adding bar revisions
adding foo revisions
bundle2-input-part: total payload size 920
bundle2-input-part: "listkeys" (params: 1 mandatory) supported
bundle2-input-part: "phase-heads" supported
bundle2-input-part: total payload size 24
bundle2-input-part: "cache:rev-branch-cache" (advisory) supported
bundle2-input-part: total payload size 59
bundle2-input-bundle: 4 parts total
checking for updated bookmarks
updating the branch cache
added 2 changesets with 2 changes to 2 files
new changesets 53245c60e682:aaff8d2ffbbf
calling hook changegroup.lfiles: hgext.largefiles.reposetup.checkrequireslfiles
(sent 4 HTTP requests and * bytes; received * bytes in responses) (glob)
$ hg clone -U --debug --config ui.available-memory=32MB http://localhost:$HGPORT gzip-too-large2
using http://localhost:$HGPORT/
sending capabilities command
sending clonebundles command
applying clone bundle from http://localhost:$HGPORT1/gz-a.hg
bundle2-input-bundle: 1 params with-transaction
bundle2-input-part: "changegroup" (params: 1 mandatory 1 advisory) supported
adding changesets
add changeset 53245c60e682
add changeset aaff8d2ffbbf
adding manifests
adding file changes
adding bar revisions
adding foo revisions
bundle2-input-part: total payload size 920
bundle2-input-part: "cache:rev-branch-cache" (advisory) supported
bundle2-input-part: total payload size 59
bundle2-input-bundle: 2 parts total
updating the branch cache
added 2 changesets with 2 changes to 2 files
finished applying clone bundle
query 1; heads
sending batch command
searching for changes
all remote heads known locally
no changes found
sending getbundle command
bundle2-input-bundle: with-transaction
bundle2-input-part: "listkeys" (params: 1 mandatory) supported
bundle2-input-part: "phase-heads" supported
bundle2-input-part: total payload size 24
bundle2-input-bundle: 2 parts total
checking for updated bookmarks
2 local changesets published
calling hook changegroup.lfiles: hgext.largefiles.reposetup.checkrequireslfiles
(sent 4 HTTP requests and * bytes; received * bytes in responses) (glob)
$ killdaemons.py