This is an archive of the discontinued Mercurial Phabricator instance.

Differential D1265

treemanifest: remove sendtrees
ClosedPublic

Authored by durham on Oct 30 2017, 11:29 AM.
Tags
None
Subscribers
singhsrb

Details

Reviewers
phillco
singhsrb
Group Reviewers
Restricted Project
Commits
rFBHGX46bc44d9dbd9: treemanifest: remove sendtrees
Summary

Previously we were relying on config values to determine when to send or not send tree and flat manifests. This resulted in complicated permutations to test. Let's remove these config values and always send trees if we have them. A future patch will do the same for flat manifests.

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

durham created this revision.Oct 30 2017, 11:29 AM
Herald added a reviewer: Restricted Project. · View Herald TranscriptOct 30 2017, 11:29 AM
durham edited the summary of this revision. (Show Details)Oct 30 2017, 11:31 AM
phillco accepted this revision.Nov 1 2017, 12:28 PM
This revision is now accepted and ready to land.Nov 1 2017, 12:28 PM
singhsrb accepted this revision.EditedNov 1 2017, 2:25 PM
singhsrb added a subscriber: singhsrb.

Looks good but I just had a few general comments and questions.

tests/test-treemanifest-server.t
85

If we are sending trees, is this relevant? Or am i missing something?

147

Nice!

treemanifest/__init__.py
1107

Hmm, same logic as in pushrebase but nice to see a method for it!

durham added inline comments.Nov 1 2017, 7:02 PM
tests/test-treemanifest-server.t
85

This is the config that tells the server to reject any pushes that are flat manifests. This will be the last step in forcing our transition to tree only on the server.

treemanifest/__init__.py
1107

Yea, sharing code between extensions is a bit awkward right now. I definitely want to move all this common storage/utility stuff together at some point.

Closed by commit rFBHGX46bc44d9dbd9: treemanifest: remove sendtrees (authored by durham). · Explain WhyNov 2 2017, 3:05 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 3219

hgext3rd/pushrebase.py

heads = repo.revs('heads(%r)', revset) heads = repo.revs('heads(%r)', revset)
if len(heads) > 1: if len(heads) > 1:
raise error.Abort(_('cannot rebase divergent changesets')) raise error.Abort(_('cannot rebase divergent changesets'))
repo.ui.note(_('validated revset for rebase\n')) repo.ui.note(_('validated revset for rebase\n'))
def getrebaseparts(repo, peer, outgoing, onto, newhead): def getrebaseparts(repo, peer, outgoing, onto, newhead):
parts = [] parts = []
if (util.safehasattr(repo.manifestlog, 'datastore') and if util.safehasattr(repo.manifestlog, 'datastore'):
repo.ui.configbool("treemanifest", "sendtrees")): mfnodes = []
for node in outgoing.missing:
mfnodes.append(('', repo[node].manifestnode()))
# Only add trees if we already have them
if not repo.manifestlog.datastore.getmissing(mfnodes):
parts.append(createtreepackpart(repo, outgoing)) parts.append(createtreepackpart(repo, outgoing))
parts.append(createrebasepart(repo, peer, outgoing, onto, newhead)) parts.append(createrebasepart(repo, peer, outgoing, onto, newhead))
return parts return parts
def createtreepackpart(repo, outgoing): def createtreepackpart(repo, outgoing):
treemod = extensions.find('treemanifest') treemod = extensions.find('treemanifest')
return treemod.createtreepackpart(repo, outgoing, rebasepackparttype) return treemod.createtreepackpart(repo, outgoing, rebasepackparttype)
def createrebasepart(repo, peer, outgoing, onto, newhead): def createrebasepart(repo, peer, outgoing, onto, newhead):

tests/test-treemanifest-peertopeer.t

Pushing with treemanifest disabled does nothing Pushing with treemanifest disabled does nothing
(disable demand import so treemanifest.py is forced to load) (disable demand import so treemanifest.py is forced to load)
$ HGDEMANDIMPORT=disable hg push -q ../client2 --config extensions.treemanifest=! --config fastmanifest.usetree=False $ HGDEMANDIMPORT=disable hg push -q ../client2 --config extensions.treemanifest=! --config fastmanifest.usetree=False
$ ls ../client2/.hg/store/packs/manifests || true $ ls ../client2/.hg/store/packs/manifests || true
* No such file or directory (glob) * No such file or directory (glob)
$ hg -R ../client2 strip -q -r tip $ hg -R ../client2 strip -q -r tip
Pushing p2p with sendtrees=False does nothing
$ hg push -q ../client2
$ ls ../client2/.hg/store/packs/manifests || true
* No such file or directory (glob)
$ hg -R ../client2 strip -q -r tip
Pushing p2p puts the received packs in the local pack store Pushing p2p puts the received packs in the local pack store
$ cat >> .hg/hgrc <<EOF
> [treemanifest]
> sendtrees=True
> EOF
$ hg push -q ../client2 $ hg push -q ../client2
$ ls ../client2/.hg/store/packs/manifests $ ls ../client2/.hg/store/packs/manifests
15f45e8ca8affec27464278498594f94a3624815.histidx 15f45e8ca8affec27464278498594f94a3624815.histidx
15f45e8ca8affec27464278498594f94a3624815.histpack 15f45e8ca8affec27464278498594f94a3624815.histpack
fb1ee78215bfece34ca8e233fcf5e9fd69ec52bd.dataidx fb1ee78215bfece34ca8e233fcf5e9fd69ec52bd.dataidx
fb1ee78215bfece34ca8e233fcf5e9fd69ec52bd.datapack fb1ee78215bfece34ca8e233fcf5e9fd69ec52bd.datapack

tests/test-treemanifest-server.t

remote: rollback completed remote: rollback completed
remote: cannot push commits to a treemanifest transition server without pushrebase remote: cannot push commits to a treemanifest transition server without pushrebase
abort: push failed on remote abort: push failed on remote
[255] [255]
Test pushing only flat fails if forcetreereceive is on Test pushing only flat fails if forcetreereceive is on
$ cat >> ../master/.hg/hgrc <<EOF $ cat >> ../master/.hg/hgrc <<EOF
> [pushrebase] > [pushrebase]
> forcetreereceive=True > forcetreereceive=True
singhsrbUnsubmitted
Not Done

If we are sending trees, is this relevant? Or am i missing something?

singhsrb: If we are sending trees, is this relevant? Or am i missing something?
durhamAuthorUnsubmitted
Not Done

This is the config that tells the server to reject any pushes that are flat manifests. This will be the last step in forcing our transition to tree only on the server.

durham: This is the config that tells the server to reject any pushes that are flat manifests. This…
> EOF > EOF
$ cat >> .hg/hgrc <<EOF $ cat >> .hg/hgrc <<EOF
> [extensions] > [extensions]
> pushrebase=$TESTDIR/../hgext3rd/pushrebase.py > pushrebase=$TESTDIR/../hgext3rd/pushrebase.py
> EOF > EOF
$ mv .hg/store/packs .hg/store/packs.bak
$ hg push --to mybook $ hg push --to mybook
pushing to ssh://user@dummy/master pushing to ssh://user@dummy/master
searching for changes searching for changes
remote: error: pushes must contain tree manifests when the server has pushrebase.forcetreereceive enabled remote: error: pushes must contain tree manifests when the server has pushrebase.forcetreereceive enabled
abort: push failed on remote abort: push failed on remote
[255] [255]
$ mv .hg/store/packs.bak .hg/store/packs
Test pushing only trees with commit hooks Test pushing flat and tree
$ cat >> .hg/hgrc <<EOF
> [treemanifest]
> sendflat=False
> sendtrees=True
> EOF
$ cat >> $TESTTMP/myhook.sh <<EOF $ cat >> $TESTTMP/myhook.sh <<EOF
> set -xe > set -xe
> [[ \$(hg log -r \$HG_NODE -T '{file_adds}') == 'subdir2/z' ]] && exit 1 > [[ \$(hg log -r \$HG_NODE -T '{file_adds}') == 'subdir2/z' ]] && exit 1
> exit 2 > exit 2
> EOF > EOF
$ chmod a+x $TESTTMP/myhook.sh $ chmod a+x $TESTTMP/myhook.sh
$ cp ../master/.hg/hgrc ../master/.hg/hgrc.bak $ cp ../master/.hg/hgrc ../master/.hg/hgrc.bak
$ cat >> ../master/.hg/hgrc <<EOF $ cat >> ../master/.hg/hgrc <<EOF
> [hooks] > [hooks]
> prepushrebase.myhook=$TESTTMP/myhook.sh > prepushrebase.myhook=$TESTTMP/myhook.sh
> EOF > EOF
$ hg push --to mybook $ hg push --to mybook
pushing to ssh://user@dummy/master pushing to ssh://user@dummy/master
searching for changes searching for changes
remote: +++ hg log -r 15486e46ccf6947fbb0a0209e6ce479e7f87ffae -T '{file_adds}' remote: +++ hg log -r 15486e46ccf6947fbb0a0209e6ce479e7f87ffae -T '{file_adds}'
remote: ++ [[ subdir2/z == \s\u\b\d\i\r\2\/\z ]] remote: ++ [[ subdir2/z == \s\u\b\d\i\r\2\/\z ]]
remote: ++ exit 1 remote: ++ exit 1
remote: prepushrebase.myhook hook exited with status 1 remote: prepushrebase.myhook hook exited with status 1
abort: push failed on remote abort: push failed on remote
[255] [255]
Test pushing only trees with commit hooks
$ cat >> .hg/hgrc <<EOF
> [treemanifest]
> sendflat=False
> EOF
$ hg push --to mybook
pushing to ssh://user@dummy/master
searching for changes
remote: +++ hg log -r 15486e46ccf6947fbb0a0209e6ce479e7f87ffae -T '{file_adds}'
remote: ++ [[ subdir2/z == \s\u\b\d\i\r\2\/\z ]]
remote: ++ exit 1
remote: prepushrebase.myhook hook exited with status 1
abort: push failed on remote
[255]
$ mv ../master/.hg/hgrc.bak ../master/.hg/hgrc $ mv ../master/.hg/hgrc.bak ../master/.hg/hgrc
Test pushing only trees (no flats) with pushrebase creates trees on the server Test pushing only trees (no flats) with pushrebase creates trees on the server
$ hg push --to mybook $ hg push --to mybook
pushing to ssh://user@dummy/master pushing to ssh://user@dummy/master
searching for changes searching for changes
remote: pushing 1 changeset: remote: pushing 1 changeset:
remote: 15486e46ccf6 add subdir2/z remote: 15486e46ccf6 add subdir2/z
$ ls ../master/.hg/store/meta $ ls ../master/.hg/store/meta
subdir subdir
subdir2 subdir2
- Verify it doesn't put anything in the pack directory
singhsrbUnsubmitted
Not Done

Nice!

singhsrb: Nice!
$ ls_l ../master/.hg/store | grep pack
[1]
$ cd ../master $ cd ../master
Verify flat was updated and tree was updated, even though only tree was sent Verify flat was updated and tree was updated, even though only tree was sent
$ hg debugdata .hg/store/00manifest.i 1 $ hg debugdata .hg/store/00manifest.i 1
subdir/x\x001406e74118627694268417491f018a4a883152f0 (esc) subdir/x\x001406e74118627694268417491f018a4a883152f0 (esc)
subdir2/z\x0069a1b67522704ec122181c0890bd16e9d3e7516a (esc) subdir2/z\x0069a1b67522704ec122181c0890bd16e9d3e7516a (esc)
$ hg debugdata .hg/store/00manifesttree.i 1 $ hg debugdata .hg/store/00manifesttree.i 1

tests/test-treemanifest-treeonly.t

-r--r--r-- 211 3c68632603fbc30bd6ee720bd43c0f0940930cc8.datapack -r--r--r-- 211 3c68632603fbc30bd6ee720bd43c0f0940930cc8.datapack
-r--r--r-- 1196 54389aae9b0f11d9cc4e33901e6de68bad129e73.histidx -r--r--r-- 1196 54389aae9b0f11d9cc4e33901e6de68bad129e73.histidx
-r--r--r-- 183 54389aae9b0f11d9cc4e33901e6de68bad129e73.histpack -r--r--r-- 183 54389aae9b0f11d9cc4e33901e6de68bad129e73.histpack
Transition to tree-only client Transition to tree-only client
$ cat >> .hg/hgrc <<EOF $ cat >> .hg/hgrc <<EOF
> [treemanifest] > [treemanifest]
> treeonly=True > treeonly=True
> sendtrees=True
> sendflat=False > sendflat=False
> EOF > EOF
Make a local tree-only draft commit Make a local tree-only draft commit
$ ls_l .hg/store/packs/manifests | wc -l $ ls_l .hg/store/packs/manifests | wc -l
\s*4 (re) \s*4 (re)
$ echo t >> subdir/x $ echo t >> subdir/x
$ hg commit -qm "tree only commit" $ hg commit -qm "tree only commit"

treemanifest/__init__.py

pullprefetchcount = 0 pullprefetchcount = 0
`treemanifest.pullprefetchrevs` specifies a revset of commits who's trees should `treemanifest.pullprefetchrevs` specifies a revset of commits who's trees should
be prefetched after a pull. Defaults to None. be prefetched after a pull. Defaults to None.
[treemanifest] [treemanifest]
pullprefetchrevs = master + stable pullprefetchrevs = master + stable
Setting `treemanifest.sendtrees` to True will include tree packs in sent
bundles.
[treemanifest]
sendtrees = False
Setting `treemanifest.repackstartrev` and `treemanifest.repackendrev` causes `hg Setting `treemanifest.repackstartrev` and `treemanifest.repackendrev` causes `hg
repack --incremental` to only repack the revlog entries in the given range. The repack --incremental` to only repack the revlog entries in the given range. The
default values are 0 and len(changelog) - 1, respectively. default values are 0 and len(changelog) - 1, respectively.
[treemanifest] [treemanifest]
repackstartrev = 0 repackstartrev = 0
repackendrev = 1000 repackendrev = 1000
import time import time
cmdtable = {} cmdtable = {}
command = registrar.command(cmdtable) command = registrar.command(cmdtable)
configtable = {} configtable = {}
configitem = registrar.configitem(configtable) configitem = registrar.configitem(configtable)
configitem('treemanifest', 'sendtrees', default=False)
configitem('treemanifest', 'server', default=False) configitem('treemanifest', 'server', default=False)
PACK_CATEGORY='manifests' PACK_CATEGORY='manifests'
TREEGROUP_PARTTYPE = 'b2x:treegroup' TREEGROUP_PARTTYPE = 'b2x:treegroup'
# Temporary part type while we migrate the arguments # Temporary part type while we migrate the arguments
TREEGROUP_PARTTYPE2 = 'b2x:treegroup2' TREEGROUP_PARTTYPE2 = 'b2x:treegroup2'
RECEIVEDNODE_RECORD = 'receivednodes' RECEIVEDNODE_RECORD = 'receivednodes'
extensions.wrapfunction(bundle2, '_addpartsfromopts', _addpartsfromopts) extensions.wrapfunction(bundle2, '_addpartsfromopts', _addpartsfromopts)
extensions.wrapfunction(bundlerepo.bundlerepository, '_handlebundle2part', extensions.wrapfunction(bundlerepo.bundlerepository, '_handlebundle2part',
_handlebundle2part) _handlebundle2part)
extensions.wrapfunction(bundle2, 'getrepocaps', getrepocaps) extensions.wrapfunction(bundle2, 'getrepocaps', getrepocaps)
_registerbundle2parts() _registerbundle2parts()
def getrepocaps(orig, repo, *args, **kwargs): def getrepocaps(orig, repo, *args, **kwargs):
caps = orig(repo, *args, **kwargs) caps = orig(repo, *args, **kwargs)
if repo.ui.configbool('treemanifest', 'sendtrees'): if treeenabled(repo.ui):
caps['treemanifest'] = ('True',) caps['treemanifest'] = ('True',)
return caps return caps
def _collectmanifest(orig, repo, striprev): def _collectmanifest(orig, repo, striprev):
if repo.ui.configbool("treemanifest", "treeonly"): if repo.ui.configbool("treemanifest", "treeonly"):
return [] return []
return orig(repo, striprev) return orig(repo, striprev)
raise error.Abort( raise error.Abort(
_("unknown treegroup bundle2 part version: %s") % version) _("unknown treegroup bundle2 part version: %s") % version)
category = part.params.get('category', '') category = part.params.get('category', '')
if category != PACK_CATEGORY: if category != PACK_CATEGORY:
raise error.Abort(_("invalid treegroup pack category: %s") % raise error.Abort(_("invalid treegroup pack category: %s") %
category) category)
# Treemanifest servers don't accept tree directly. They must go through
# pushrebase, which uses it's own part type and handler.
if op.repo.svfs.treemanifestserver:
return
if part.params.get('cache', 'False') == 'True': if part.params.get('cache', 'False') == 'True':
packpath = shallowutil.getcachepackpath(repo, PACK_CATEGORY) packpath = shallowutil.getcachepackpath(repo, PACK_CATEGORY)
else: else:
packpath = shallowutil.getlocalpackpath(repo.svfs.vfs.base, packpath = shallowutil.getlocalpackpath(repo.svfs.vfs.base,
PACK_CATEGORY) PACK_CATEGORY)
receivedhistory, receiveddata = wirepack.receivepack(repo.ui, part, receivedhistory, receiveddata = wirepack.receivepack(repo.ui, part,
packpath) packpath)
@exchange.b2partsgenerator(TREEGROUP_PARTTYPE) @exchange.b2partsgenerator(TREEGROUP_PARTTYPE)
def gettreepackpart(pushop, bundler): def gettreepackpart(pushop, bundler):
# We no longer generate old tree groups # We no longer generate old tree groups
pass pass
@exchange.b2partsgenerator(TREEGROUP_PARTTYPE2) @exchange.b2partsgenerator(TREEGROUP_PARTTYPE2)
def gettreepackpart2(pushop, bundler): def gettreepackpart2(pushop, bundler):
"""add parts containing trees being pushed""" """add parts containing trees being pushed"""
if ('treepack' in pushop.stepsdone or if 'treepack' in pushop.stepsdone or not treeenabled(pushop.repo.ui):
not treeenabled(pushop.repo.ui) or
not pushop.repo.ui.configbool('treemanifest', 'sendtrees')):
return return
pushop.stepsdone.add('treepack') pushop.stepsdone.add('treepack')
# Only add trees if we have them
if _cansendtrees(pushop.repo, pushop.outgoing.missing):
part = createtreepackpart(pushop.repo, pushop.outgoing, part = createtreepackpart(pushop.repo, pushop.outgoing,
TREEGROUP_PARTTYPE2) TREEGROUP_PARTTYPE2)
bundler.addpart(part) bundler.addpart(part)
@exchange.getbundle2partsgenerator(TREEGROUP_PARTTYPE2) @exchange.getbundle2partsgenerator(TREEGROUP_PARTTYPE2)
def _getbundlechangegrouppart(bundler, repo, source, bundlecaps=None, def _getbundlechangegrouppart(bundler, repo, source, bundlecaps=None,
b2caps=None, heads=None, common=None, b2caps=None, heads=None, common=None,
**kwargs): **kwargs):
"""add parts containing trees being pulled""" """add parts containing trees being pulled"""
if ('True' not in b2caps.get('treemanifest', []) or if ('True' not in b2caps.get('treemanifest', []) or
not treeenabled(repo.ui) or not treeenabled(repo.ui) or
repo.svfs.treemanifestserver or repo.svfs.treemanifestserver or
not kwargs.get('cg', True)): not kwargs.get('cg', True)):
return return
outgoing = exchange._computeoutgoing(repo, heads, common) outgoing = exchange._computeoutgoing(repo, heads, common)
if _cansendtrees(repo, outgoing.missing):
part = createtreepackpart(repo, outgoing, TREEGROUP_PARTTYPE2) part = createtreepackpart(repo, outgoing, TREEGROUP_PARTTYPE2)
bundler.addpart(part) bundler.addpart(part)
def _cansendtrees(repo, nodes):
singhsrbUnsubmitted
Not Done

Hmm, same logic as in pushrebase but nice to see a method for it!

singhsrb: Hmm, same logic as in pushrebase but nice to see a method for it!
durhamAuthorUnsubmitted
Not Done

Yea, sharing code between extensions is a bit awkward right now. I definitely want to move all this common storage/utility stuff together at some point.

durham: Yea, sharing code between extensions is a bit awkward right now. I definitely want to move all…
mfnodes = []
for node in nodes:
mfnodes.append(('', repo[node].manifestnode()))
return not repo.manifestlog.datastore.getmissing(mfnodes)
def createtreepackpart(repo, outgoing, partname): def createtreepackpart(repo, outgoing, partname):
rootdir = '' rootdir = ''
mfnodes = [] mfnodes = []
basemfnodes = [] basemfnodes = []
directories = [] directories = []
for node in outgoing.missing: for node in outgoing.missing:
mfnode = repo[node].manifestnode() mfnode = repo[node].manifestnode()
# no-op. # no-op.
rl = treerevlog.dirlog(dir) rl = treerevlog.dirlog(dir)
rl.strip(striprev, tr) rl.strip(striprev, tr)
treerevlog.strip(striprev, tr) treerevlog.strip(striprev, tr)
def _addpartsfromopts(orig, ui, repo, bundler, source, outgoing, opts): def _addpartsfromopts(orig, ui, repo, bundler, source, outgoing, opts):
orig(ui, repo, bundler, source, outgoing, opts) orig(ui, repo, bundler, source, outgoing, opts)
if ui.configbool('treemanifest', 'sendtrees'):
# Only add trees if we have them
if _cansendtrees(repo, outgoing.missing):
part = createtreepackpart(repo, outgoing, TREEGROUP_PARTTYPE2) part = createtreepackpart(repo, outgoing, TREEGROUP_PARTTYPE2)
bundler.addpart(part) bundler.addpart(part)
def _handlebundle2part(orig, self, part): def _handlebundle2part(orig, self, part):
if part.type == TREEGROUP_PARTTYPE2: if part.type == TREEGROUP_PARTTYPE2:
tempstore = wirepack.wirepackstore(part.read()) tempstore = wirepack.wirepackstore(part.read())
# Point the bundle repo at the temp stores # Point the bundle repo at the temp stores