This is an archive of the discontinued Mercurial Phabricator instance.

Differential D1334

prefetch: merge prefetchtrees command into prefetch
AbandonedPublic

Authored by singhsrb on Nov 7 2017, 7:37 PM.
Tags
None
Subscribers
None

Details

Reviewers
None
Group Reviewers
Restricted Project
Summary

Currently,

  • hg prefetch prefetches files.
  • hg prefetchtrees prefetches trees.

This commit removes prefetchtrees and makes prefetch responsible for also
fetching the trees in case we are dealing with trees.

Test Plan

Ran all the tests.

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

singhsrb created this revision.Nov 7 2017, 7:37 PM
Herald added a reviewer: Restricted Project. · View Herald TranscriptNov 7 2017, 7:37 PM
singhsrb retitled this revision from prefetch: merge prefetchtress command into prefetch to prefetch: merge prefetchtrees command into prefetch.Nov 7 2017, 7:38 PM
singhsrb added inline comments.Nov 7 2017, 7:41 PM
tests/test-treemanifest-prefetch.t
129

I effectively removed the --base option. Let me know if that was important and I can revive it. If we do want it, does it make sense to have one for hg prefetch itself?

treemanifest/__init__.py
161

Not sure if this is the best way or place to do this. Suggestions welcome!

singhsrb planned changes to this revision.Nov 9 2017, 1:08 PM

After the discussion in D1333, this would be done differently.

singhsrb abandoned this revision.Nov 14 2017, 3:46 PM

See D1417 instead.

Revision Contents
Changeset List

Diff 3328

tests/test-treemanifest-prefetch.t

> [remotefilelog] > [remotefilelog]
> reponame = master > reponame = master
> cachepath = $CACHEDIR > cachepath = $CACHEDIR
> [fastmanifest] > [fastmanifest]
> usetree = True > usetree = True
> usecache = False > usecache = False
> EOF > EOF
Test prefetchtrees Test prefetch
$ hg prefetchtrees -r '0 + 1 + 2' $ hg prefetch -r '0 + 1 + 2'
6 trees fetched over * (glob) 6 trees fetched over * (glob)
1 files fetched over * (glob)
$ ls $CACHEDIR/master/packs/manifests $ ls $CACHEDIR/master/packs/manifests
29938257d506f677320d5abec8e34a1a9ed635fe.histidx 29938257d506f677320d5abec8e34a1a9ed635fe.histidx
29938257d506f677320d5abec8e34a1a9ed635fe.histpack 29938257d506f677320d5abec8e34a1a9ed635fe.histpack
8adc618d23082c0a5311a4bbf9ac08b9b9672471.dataidx 8adc618d23082c0a5311a4bbf9ac08b9b9672471.dataidx
8adc618d23082c0a5311a4bbf9ac08b9b9672471.datapack 8adc618d23082c0a5311a4bbf9ac08b9b9672471.datapack
$ hg debugdatapack --long $CACHEDIR/master/packs/manifests/*.dataidx $ hg debugdatapack --long $CACHEDIR/master/packs/manifests/*.dataidx
$TESTTMP/hgcache/master/packs/manifests/8adc618d23082c0a5311a4bbf9ac08b9b9672471: $TESTTMP/hgcache/master/packs/manifests/8adc618d23082c0a5311a4bbf9ac08b9b9672471:
subdir: subdir:
1 47 61 0 1 1be4ab2126dd ef362f8bbe8a 000000000000 1 47 61 0 1 1be4ab2126dd ef362f8bbe8a 000000000000
2 108 58 1 2 60a7f7acb6bb 1be4ab2126dd 000000000000 2 108 58 1 2 60a7f7acb6bb 1be4ab2126dd 000000000000
$ hg -R ../master debugindex ../master/.hg/store/00manifest.i $ hg -R ../master debugindex ../master/.hg/store/00manifest.i
rev offset length delta linkrev nodeid p1 p2 rev offset length delta linkrev nodeid p1 p2
0 0 48 -1 0 ef362f8bbe8a 000000000000 000000000000 0 0 48 -1 0 ef362f8bbe8a 000000000000 000000000000
1 48 62 0 1 1be4ab2126dd ef362f8bbe8a 000000000000 1 48 62 0 1 1be4ab2126dd ef362f8bbe8a 000000000000
2 110 59 1 2 60a7f7acb6bb 1be4ab2126dd 000000000000 2 110 59 1 2 60a7f7acb6bb 1be4ab2126dd 000000000000
Test prefetch with base node (subdir/ shouldn't show up in the pack)
singhsrbAuthorUnsubmitted
Not Done

I effectively removed the --base option. Let me know if that was important and I can revive it. If we do want it, does it make sense to have one for hg prefetch itself?

singhsrb: I effectively removed the --base option. Let me know if that was important and I can revive it.
$ rm -rf $CACHEDIR/master
$ hg prefetchtrees -r '2' --base '1'
2 trees fetched over * (glob)
$ ls $CACHEDIR/master/packs/manifests/*.dataidx
$TESTTMP/hgcache/master/packs/manifests/3fb59713808147bda39cbd97b9cd862406f5865c.dataidx
$ hg debugdatapack $CACHEDIR/master/packs/manifests/3fb59713808147bda39cbd97b9cd862406f5865c.dataidx
$TESTTMP/hgcache/master/packs/manifests/3fb59713808147bda39cbd97b9cd862406f5865c:
dir:
Node Delta Base Delta Length Blob Size
a18d21674e76 000000000000 43 (missing)
(empty name):
Node Delta Base Delta Length Blob Size
60a7f7acb6bb 000000000000 95 (missing)
Test auto prefetch during normal access Test auto prefetch during normal access
$ rm -rf $CACHEDIR/master $ rm -rf $CACHEDIR/master
|| ( exit 1 ) is needed because ls on OSX and Linux exits differently || ( exit 1 ) is needed because ls on OSX and Linux exits differently
$ ls $CACHEDIR/master/packs/manifests || ( exit 1 ) $ ls $CACHEDIR/master/packs/manifests || ( exit 1 )
*No such file or directory (glob) *No such file or directory (glob)
[1] [1]
$ hg log -r tip --stat --pager=off $ hg log -r tip --stat --pager=off
3 trees fetched over * (glob) 3 trees fetched over * (glob)
(empty name): (empty name):
Node Delta Base Delta Length Blob Size Node Delta Base Delta Length Blob Size
60a7f7acb6bb 000000000000 95 (missing) 60a7f7acb6bb 000000000000 95 (missing)
Test that auto prefetch scans up the changelog for base trees Test that auto prefetch scans up the changelog for base trees
$ rm -rf $CACHEDIR/master $ rm -rf $CACHEDIR/master
$ hg prefetchtrees -r 'tip^' $ hg prefetch -r 'tip^'
3 trees fetched over * (glob) 3 trees fetched over * (glob)
2 files fetched over * (glob)
$ rm -rf $CACHEDIR/master $ rm -rf $CACHEDIR/master
$ hg prefetchtrees -r tip $ hg prefetch -r tip
3 trees fetched over * (glob) 3 trees fetched over * (glob)
2 files fetched over * (glob)
- Only 2 of the 3 trees from tip^ are downloaded as part of --stat's fetch - Only 2 of the 3 trees from tip^ are downloaded as part of --stat's fetch
$ hg log -r tip --stat --pager=off > /dev/null $ hg log -r tip --stat --pager=off > /dev/null
2 trees fetched over * (glob) 2 trees fetched over * (glob)
2 files fetched over * (glob) 1 files fetched over * (glob)
Test auto prefetch during pull Test auto prefetch during pull
- Prefetch everything - Prefetch everything
$ echo a >> a $ echo a >> a
$ hg commit -Aqm 'draft commit that shouldnt affect prefetching trees' $ hg commit -Aqm 'draft commit that shouldnt affect prefetch'
1 files fetched over * (glob)
$ rm -rf $CACHEDIR/master $ rm -rf $CACHEDIR/master
$ hg pull --config treemanifest.pullprefetchcount=10 --traceback $ hg pull --config treemanifest.pullprefetchcount=10 --traceback
pulling from ssh://user@dummy/master pulling from ssh://user@dummy/master
searching for changes searching for changes
no changes found no changes found
prefetching trees prefetching trees
6 trees fetched over * (glob) 6 trees fetched over * (glob)
$ hg debugdatapack $CACHEDIR/master/packs/manifests/*.dataidx $ hg debugdatapack $CACHEDIR/master/packs/manifests/*.dataidx
(empty name): (empty name):
Node Delta Base Delta Length Blob Size Node Delta Base Delta Length Blob Size
60a7f7acb6bb 000000000000 95 (missing) 60a7f7acb6bb 000000000000 95 (missing)
- Prefetch commit 1 then minimally prefetch commit 2 - Prefetch commit 1 then minimally prefetch commit 2
$ rm -rf $CACHEDIR/master $ rm -rf $CACHEDIR/master
$ hg prefetchtrees -r 1 $ hg prefetch -r 1
3 trees fetched over * (glob) 3 trees fetched over * (glob)
2 files fetched over * (glob)
$ ls $CACHEDIR/master/packs/manifests/*dataidx $ ls $CACHEDIR/master/packs/manifests/*dataidx
$TESTTMP/hgcache/master/packs/manifests/148e9eb32f473ea522c591c95be0f9e772be9675.dataidx $TESTTMP/hgcache/master/packs/manifests/148e9eb32f473ea522c591c95be0f9e772be9675.dataidx
$ hg pull --config treemanifest.pullprefetchcount=1 --traceback $ hg pull --config treemanifest.pullprefetchcount=1 --traceback
pulling from ssh://user@dummy/master pulling from ssh://user@dummy/master
searching for changes searching for changes
no changes found no changes found
prefetching trees prefetching trees
2 trees fetched over * (glob) 2 trees fetched over * (glob)
3 trees fetched over * (glob) 3 trees fetched over * (glob)
z z
1 files fetched over * (glob) 1 files fetched over * (glob)
Test prefetch non-parent commits with no base node (should fetch minimal Test prefetch non-parent commits with no base node (should fetch minimal
trees - in this case 3 trees for commit 2, and 2 for commit 4 despite it having trees - in this case 3 trees for commit 2, and 2 for commit 4 despite it having
3 directories) 3 directories)
$ rm -rf $CACHEDIR/master $ rm -rf $CACHEDIR/master
$ hg prefetchtrees -r '2 + 4' $ hg prefetch -r '2 + 4'
5 trees fetched over * (glob) 5 trees fetched over * (glob)
3 files fetched over * (glob)

treemanifest/__init__.py

hg, hg,
localrepo, localrepo,
manifest, manifest,
mdiff, mdiff,
phases, phases,
registrar, registrar,
repair, repair,
revlog, revlog,
scmutil,
sshserver, sshserver,
templatekw, templatekw,
util, util,
wireproto, wireproto,
) )
from mercurial.i18n import _ from mercurial.i18n import _
from mercurial.node import bin, hex, nullid from mercurial.node import bin, hex, nullid
extensions.wrapfunction(bundlerepo.bundlerepository, '_handlebundle2part', extensions.wrapfunction(bundlerepo.bundlerepository, '_handlebundle2part',
_handlebundle2part) _handlebundle2part)
extensions.wrapfunction(bundle2, 'getrepocaps', getrepocaps) extensions.wrapfunction(bundle2, 'getrepocaps', getrepocaps)
_registerbundle2parts() _registerbundle2parts()
extensions.wrapfunction(templatekw, 'showmanifest', showmanifest) extensions.wrapfunction(templatekw, 'showmanifest', showmanifest)
templatekw.keywords['manifest'] = templatekw.showmanifest templatekw.keywords['manifest'] = templatekw.showmanifest
def wrapremotefilelog(loaded=False):
singhsrbAuthorUnsubmitted
Not Done

Not sure if this is the best way or place to do this. Suggestions welcome!

singhsrb: Not sure if this is the best way or place to do this. Suggestions welcome!
try:
remotefilelogmod = extensions.find('remotefilelog')
extensions.wrapcommand(
remotefilelogmod.cmdtable, 'prefetch', _prefetchwrapper)
except KeyError:
pass
extensions.afterloaded('remotefilelog', wrapremotefilelog)
def showmanifest(orig, **args): def showmanifest(orig, **args):
# The normal manifest template shows a rev number, which we don't have. # The normal manifest template shows a rev number, which we don't have.
# Let's just print the node. Note, this breaks the ability for us to do # Let's just print the node. Note, this breaks the ability for us to do
# '{manifest % "{node} {rev}"}' but that's fine, since we can only show # '{manifest % "{node} {rev}"}' but that's fine, since we can only show
# the node now anyway. # the node now anyway.
return hex(args[r'ctx'].manifestnode()) return hex(args[r'ctx'].manifestnode())
def getrepocaps(orig, repo, *args, **kwargs): def getrepocaps(orig, repo, *args, **kwargs):
return wrapper(origfn, *args, **kwargs) return wrapper(origfn, *args, **kwargs)
currcls.__dict__[propname].func = wrap currcls.__dict__[propname].func = wrap
break break
if currcls is object: if currcls is object:
raise AttributeError(_("%s has no property '%s'") % raise AttributeError(_("%s has no property '%s'") %
(type(currcls), propname)) (type(currcls), propname))
@command('prefetchtrees', [ def _prefetchwrapper(orig, ui, repo, *args, **opts):
('r', 'rev', '', _("revs to prefetch the trees for")), revs = scmutil.revrange(repo, opts.get('rev'))
('', 'base', '', _("revs that are assumed to already be local")),
] + commands.walkopts, _('--rev REVS PATTERN..'))
def prefetchtrees(ui, repo, *args, **opts):
revs = repo.revs(opts.get('rev'))
baserevs = []
if opts.get('base'):
baserevs = repo.revs(opts.get('base'))
mfnodes = set() mfnodes = set()
for rev in revs: for rev in revs:
mfnodes.add(repo[rev].manifestnode()) mfnodes.add(repo[rev].manifestnode())
basemfnodes = set() _prefetchtrees(repo, '', mfnodes, set(), [])
for rev in baserevs: orig(ui, repo, *args, **opts)
basemfnodes.add(repo[rev].manifestnode())
_prefetchtrees(repo, '', mfnodes, basemfnodes, [])
def _prefetchtrees(repo, rootdir, mfnodes, basemfnodes, directories): def _prefetchtrees(repo, rootdir, mfnodes, basemfnodes, directories):
# If possible, use remotefilelog's more expressive fallbackpath # If possible, use remotefilelog's more expressive fallbackpath
if util.safehasattr(repo, 'fallbackpath'): if util.safehasattr(repo, 'fallbackpath'):
fallbackpath = repo.fallbackpath fallbackpath = repo.fallbackpath
else: else:
fallbackpath = repo.ui.config('paths', 'default') fallbackpath = repo.ui.config('paths', 'default')