This is an archive of the discontinued Mercurial Phabricator instance.

Differential D833

exchange: perform stream clone with clone bundle with --uncompressed
ClosedPublic

Authored by indygreg on Sep 28 2017, 10:25 AM.

Details

Reviewers
krbullock
Group Reviewers
hg-reviewers
Commits
rHGff406f3e57b2: exchange: perform stream clone with clone bundle with --uncompressed
Summary

Previously, hg clone --uncompressed would always clone from the
origin server, even if a streaming clone bundle were available.

With this change, we invoke the clone bundle mechanism before the
stream clone mechanism, giving clone bundles the opportunity to
handle --uncompressed (which is mapped to pullop.streamclonepreferred).

The clone bundle filtering code now filters out entries that aren't
stream clones when a stream clone is requested. If a stream clone
clone bundle entry is present, it will be used. Otherwise, the client
will fall back to a server-based streaming clone.

.. feature::

`hg clone --uncompressed` uses clone bundles when possible

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Sep 28 2017, 10:25 AM
krbullock accepted this revision.Sep 30 2017, 9:56 AM
krbullock added a subscriber: krbullock.

Queued these, thanks.

This revision is now accepted and ready to land.Sep 30 2017, 9:56 AM
Closed by commit rHGff406f3e57b2: exchange: perform stream clone with clone bundle with --uncompressed (authored by indygreg). · Explain WhySep 30 2017, 9:57 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/exchange.py (24 lines)
Mtests/test-clonebundles.t (12 lines)

Diff 2195

mercurial/exchange.py

" %s") % (', '.join(sorted(missing))) " %s") % (', '.join(sorted(missing)))
raise error.Abort(msg) raise error.Abort(msg)
wlock = lock = None wlock = lock = None
try: try:
wlock = pullop.repo.wlock() wlock = pullop.repo.wlock()
lock = pullop.repo.lock() lock = pullop.repo.lock()
pullop.trmanager = transactionmanager(repo, 'pull', remote.url()) pullop.trmanager = transactionmanager(repo, 'pull', remote.url())
streamclone.maybeperformlegacystreamclone(pullop)
# This should ideally be in _pullbundle2(). However, it needs to run # This should ideally be in _pullbundle2(). However, it needs to run
# before discovery to avoid extra work. # before discovery to avoid extra work.
_maybeapplyclonebundle(pullop) _maybeapplyclonebundle(pullop)
streamclone.maybeperformlegacystreamclone(pullop)
_pulldiscovery(pullop) _pulldiscovery(pullop)
if pullop.canusebundle2: if pullop.canusebundle2:
_pullbundle2(pullop) _pullbundle2(pullop)
_pullchangeset(pullop) _pullchangeset(pullop)
_pullphase(pullop) _pullphase(pullop)
_pullbookmarks(pullop) _pullbookmarks(pullop)
_pullobsolete(pullop) _pullobsolete(pullop)
pullop.trmanager.close() pullop.trmanager.close()
pullop.clonebundleattempted = True pullop.clonebundleattempted = True
entries = parseclonebundlesmanifest(repo, res) entries = parseclonebundlesmanifest(repo, res)
if not entries: if not entries:
repo.ui.note(_('no clone bundles available on remote; ' repo.ui.note(_('no clone bundles available on remote; '
'falling back to regular clone\n')) 'falling back to regular clone\n'))
return return
entries = filterclonebundleentries(repo, entries) entries = filterclonebundleentries(
repo, entries, streamclonerequested=pullop.streamclonerequested)
if not entries: if not entries:
# There is a thundering herd concern here. However, if a server # There is a thundering herd concern here. However, if a server
# operator doesn't advertise bundles appropriate for its clients, # operator doesn't advertise bundles appropriate for its clients,
# they deserve what's coming. Furthermore, from a client's # they deserve what's coming. Furthermore, from a client's
# perspective, no automatic fallback would mean not being able to # perspective, no automatic fallback would mean not being able to
# clone! # clone!
repo.ui.warn(_('no compatible clone bundles available on server; ' repo.ui.warn(_('no compatible clone bundles available on server; '
'falling back to regular clone\n')) 'falling back to regular clone\n'))
pass pass
except error.UnsupportedBundleSpecification: except error.UnsupportedBundleSpecification:
pass pass
m.append(attrs) m.append(attrs)
return m return m
def filterclonebundleentries(repo, entries): def filterclonebundleentries(repo, entries, streamclonerequested=False):
"""Remove incompatible clone bundle manifest entries. """Remove incompatible clone bundle manifest entries.
Accepts a list of entries parsed with ``parseclonebundlesmanifest`` Accepts a list of entries parsed with ``parseclonebundlesmanifest``
and returns a new list consisting of only the entries that this client and returns a new list consisting of only the entries that this client
should be able to apply. should be able to apply.
There is no guarantee we'll be able to apply all returned entries because There is no guarantee we'll be able to apply all returned entries because
the metadata we use to filter on may be missing or wrong. the metadata we use to filter on may be missing or wrong.
""" """
newentries = [] newentries = []
for entry in entries: for entry in entries:
spec = entry.get('BUNDLESPEC') spec = entry.get('BUNDLESPEC')
if spec: if spec:
try: try:
parsebundlespec(repo, spec, strict=True) comp, version, params = parsebundlespec(repo, spec, strict=True)
# If a stream clone was requested, filter out non-streamclone
# entries.
if streamclonerequested and (comp != 'UN' or version != 's1'):
repo.ui.debug('filtering %s because not a stream clone\n' %
entry['URL'])
continue
except error.InvalidBundleSpecification as e: except error.InvalidBundleSpecification as e:
repo.ui.debug(str(e) + '\n') repo.ui.debug(str(e) + '\n')
continue continue
except error.UnsupportedBundleSpecification as e: except error.UnsupportedBundleSpecification as e:
repo.ui.debug('filtering %s because unsupported bundle ' repo.ui.debug('filtering %s because unsupported bundle '
'spec: %s\n' % (entry['URL'], str(e))) 'spec: %s\n' % (entry['URL'], str(e)))
continue continue
# If we don't have a spec and requested a stream clone, we don't know
# what the entry is so don't attempt to apply it.
elif streamclonerequested:
repo.ui.debug('filtering %s because cannot determine if a stream '
'clone bundle\n' % entry['URL'])
continue
if 'REQUIRESNI' in entry and not sslutil.hassni: if 'REQUIRESNI' in entry and not sslutil.hassni:
repo.ui.debug('filtering %s because SNI not supported\n' % repo.ui.debug('filtering %s because SNI not supported\n' %
entry['URL']) entry['URL'])
continue continue
newentries.append(entry) newentries.append(entry)

tests/test-clonebundles.t

A manifest with just a gzip bundle A manifest with just a gzip bundle
$ cat > server/.hg/clonebundles.manifest << EOF $ cat > server/.hg/clonebundles.manifest << EOF
> http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2 > http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2
> EOF > EOF
$ hg clone -U --uncompressed http://localhost:$HGPORT uncompressed-gzip $ hg clone -U --uncompressed http://localhost:$HGPORT uncompressed-gzip
no compatible clone bundles available on server; falling back to regular clone
(you may want to report this to the server operator)
streaming all changes streaming all changes
4 files to transfer, 613 bytes of data 4 files to transfer, 613 bytes of data
transferred 613 bytes in * seconds (*) (glob) transferred 613 bytes in * seconds (*) (glob)
searching for changes searching for changes
no changes found no changes found
A manifest with a stream clone but no BUNDLESPEC A manifest with a stream clone but no BUNDLESPEC
$ cat > server/.hg/clonebundles.manifest << EOF $ cat > server/.hg/clonebundles.manifest << EOF
> http://localhost:$HGPORT1/packed.hg > http://localhost:$HGPORT1/packed.hg
> EOF > EOF
$ hg clone -U --uncompressed http://localhost:$HGPORT uncompressed-no-bundlespec $ hg clone -U --uncompressed http://localhost:$HGPORT uncompressed-no-bundlespec
no compatible clone bundles available on server; falling back to regular clone
(you may want to report this to the server operator)
streaming all changes streaming all changes
4 files to transfer, 613 bytes of data 4 files to transfer, 613 bytes of data
transferred 613 bytes in * seconds (*) (glob) transferred 613 bytes in * seconds (*) (glob)
searching for changes searching for changes
no changes found no changes found
A manifest with a gzip bundle and a stream clone A manifest with a gzip bundle and a stream clone
$ cat > server/.hg/clonebundles.manifest << EOF $ cat > server/.hg/clonebundles.manifest << EOF
> http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2 > http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2
> http://localhost:$HGPORT1/packed.hg BUNDLESPEC=none-packed1 > http://localhost:$HGPORT1/packed.hg BUNDLESPEC=none-packed1
> EOF > EOF
$ hg clone -U --uncompressed http://localhost:$HGPORT uncompressed-gzip-packed $ hg clone -U --uncompressed http://localhost:$HGPORT uncompressed-gzip-packed
streaming all changes applying clone bundle from http://localhost:$HGPORT1/packed.hg
4 files to transfer, 613 bytes of data 4 files to transfer, 613 bytes of data
transferred 613 bytes in * seconds (*) (glob) transferred 613 bytes in * seconds (*) (glob)
finished applying clone bundle
searching for changes searching for changes
no changes found no changes found
A manifest with a gzip bundle and stream clone with supported requirements A manifest with a gzip bundle and stream clone with supported requirements
$ cat > server/.hg/clonebundles.manifest << EOF $ cat > server/.hg/clonebundles.manifest << EOF
> http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2 > http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2
> http://localhost:$HGPORT1/packed.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv1 > http://localhost:$HGPORT1/packed.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv1
> EOF > EOF
$ hg clone -U --uncompressed http://localhost:$HGPORT uncompressed-gzip-packed-requirements $ hg clone -U --uncompressed http://localhost:$HGPORT uncompressed-gzip-packed-requirements
streaming all changes applying clone bundle from http://localhost:$HGPORT1/packed.hg
4 files to transfer, 613 bytes of data 4 files to transfer, 613 bytes of data
transferred 613 bytes in * seconds (*) (glob) transferred 613 bytes in * seconds (*) (glob)
finished applying clone bundle
searching for changes searching for changes
no changes found no changes found
A manifest with a gzip bundle and a stream clone with unsupported requirements A manifest with a gzip bundle and a stream clone with unsupported requirements
$ cat > server/.hg/clonebundles.manifest << EOF $ cat > server/.hg/clonebundles.manifest << EOF
> http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2 > http://localhost:$HGPORT1/gz-a.hg BUNDLESPEC=gzip-v2
> http://localhost:$HGPORT1/packed.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv42 > http://localhost:$HGPORT1/packed.hg BUNDLESPEC=none-packed1;requirements%3Drevlogv42
> EOF > EOF
$ hg clone -U --uncompressed http://localhost:$HGPORT uncompressed-gzip-packed-unsupported-requirements $ hg clone -U --uncompressed http://localhost:$HGPORT uncompressed-gzip-packed-unsupported-requirements
no compatible clone bundles available on server; falling back to regular clone
(you may want to report this to the server operator)
streaming all changes streaming all changes
4 files to transfer, 613 bytes of data 4 files to transfer, 613 bytes of data
transferred 613 bytes in * seconds (*) (glob) transferred 613 bytes in * seconds (*) (glob)
searching for changes searching for changes
no changes found no changes found