This is an archive of the discontinued Mercurial Phabricator instance.

Differential D5139

store: introduce _matchtrackedpath() and use it to filter store files
ClosedPublic

Authored by pulkit on Oct 17 2018, 10:53 AM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHG9aeb9e2d28a7: store: introduce _matchtrackedpath() and use it to filter store files
Summary

This patch introduces a function to filter store files on the basis of the path
which they are tracking.

The function assumes that the entries can be of two types, 'meta/*' and 'data/*'
which means it will just work on revlog based storage and not with another
storage ways.

For the 'data/*' entries, we remove the 'data/' part and '.i/.d' part from the
beginning and the end then pass that to matcher.

For the 'meta/*' entries, we remove the 'meta/' and '/00manifest.(i/d)' part from
beginning and end then call matcher.visitdir() with it to make sure all the
parent directories are also downloaded.

Since the storage filtering for narrow stream clones is implemented with this
patch, we remove the un-implemented error message, add some more tests and add
the treemanifest case to tests too.

The tests demonstrate that it works correctly.

After this patch, we have now narrow stream clones working. Narrow stream clones
are a very important feature for large repositories who have good internet
connection because they use streamclones for cloning and if they do normal
narrow clone, that takes more time then a full streamclone. Also narrow-stream
clone will drastically speed up clone timings.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

pulkit created this revision.Oct 17 2018, 10:53 AM
pulkit updated this revision to Diff 12259.Oct 19 2018, 9:01 AM
durin42 added inline comments.Nov 5 2018, 11:08 AM
mercurial/store.py
39

Please follow up with a patch that raises ProgrammingError at the end of this function - right now you just magically return False if you don't recognize the path, which feels dangerous.

Closed by commit rHG9aeb9e2d28a7: store: introduce _matchtrackedpath() and use it to filter store files (authored by pulkit). · Explain WhyNov 5 2018, 11:15 AM
This revision was automatically updated to reflect the committed changes.
yuja added a subscriber: yuja.Nov 8 2018, 7:12 AM

Can you add the following tests?

  • encoded filename differs from the original name (e.g. uppercase letter)
  • fncache disabled, but encodedstore is used
def datafiles(self, matcher=None):
    for a, b, size in super(encodedstore, self).datafiles():

+ if not _matchtrackedpath(a, matcher):
+ continue

try:
    a = decodefilename(a)

I'm pretty sure it's wrong to pass in an encoded filename to matcher.

Revision Contents
Changeset List

Diff 12233

mercurial/store.py

policy, policy,
pycompat, pycompat,
util, util,
vfs as vfsmod, vfs as vfsmod,
) )
parsers = policy.importmod(r'parsers') parsers = policy.importmod(r'parsers')
def _matchtrackedpath(path, matcher):
"""parses a fncache entry and returns whether the entry is tracking a path
matched by matcher or not.
If matcher is None, returns True"""
if matcher is None:
return True
path = decodefilename(path)
if path.startswith('data/'):
return matcher(path[len('data/'):-len('.i')])
elif path.startswith('meta/'):
return matcher.visitdir(path[len('meta/'):-len('/00manifest.i')] or '.')
durin42Unsubmitted
Not Done

Please follow up with a patch that raises ProgrammingError at the end of this function - right now you just magically return False if you don't recognize the path, which feels dangerous.

durin42: Please follow up with a patch that raises ProgrammingError at the end of this function - right…
# This avoids a collision between a file named foo and a dir named # This avoids a collision between a file named foo and a dir named
# foo.i or foo.d # foo.i or foo.d
def _encodedir(path): def _encodedir(path):
''' '''
>>> _encodedir(b'data/foo.i') >>> _encodedir(b'data/foo.i')
'data/foo.i' 'data/foo.i'
>>> _encodedir(b'data/foo.i/bla.i') >>> _encodedir(b'data/foo.i/bla.i')
'data/foo.i.hg/bla.i' 'data/foo.i.hg/bla.i'
self.createmode = _calcmode(vfs) self.createmode = _calcmode(vfs)
vfs.createmode = self.createmode vfs.createmode = self.createmode
self.rawvfs = vfs self.rawvfs = vfs
self.vfs = vfsmod.filtervfs(vfs, encodefilename) self.vfs = vfsmod.filtervfs(vfs, encodefilename)
self.opener = self.vfs self.opener = self.vfs
def datafiles(self, matcher=None): def datafiles(self, matcher=None):
for a, b, size in super(encodedstore, self).datafiles(): for a, b, size in super(encodedstore, self).datafiles():
if not _matchtrackedpath(a, matcher):
continue
try: try:
a = decodefilename(a) a = decodefilename(a)
except KeyError: except KeyError:
a = None a = None
yield a, b, size yield a, b, size
def join(self, f): def join(self, f):
return self.path + '/' + encodefilename(f) return self.path + '/' + encodefilename(f)
def join(self, f): def join(self, f):
return self.pathsep + self.encode(f) return self.pathsep + self.encode(f)
def getsize(self, path): def getsize(self, path):
return self.rawvfs.stat(path).st_size return self.rawvfs.stat(path).st_size
def datafiles(self, matcher=None): def datafiles(self, matcher=None):
for f in sorted(self.fncache): for f in sorted(self.fncache):
if not _matchtrackedpath(f, matcher):
continue
ef = self.encode(f) ef = self.encode(f)
try: try:
yield f, ef, self.getsize(ef) yield f, ef, self.getsize(ef)
except OSError as err: except OSError as err:
if err.errno != errno.ENOENT: if err.errno != errno.ENOENT:
raise raise
def copylist(self): def copylist(self):

mercurial/streamclone.py

2) A varint containing the length of the filename 2) A varint containing the length of the filename
3) A varint containing the length of file data 3) A varint containing the length of file data
4) N bytes containing the filename (the internal, store-agnostic form) 4) N bytes containing the filename (the internal, store-agnostic form)
5) N bytes containing the file data 5) N bytes containing the file data
Returns a 3-tuple of (file count, file size, data iterator). Returns a 3-tuple of (file count, file size, data iterator).
""" """
# temporarily raise error until we add storage level logic
if includes or excludes:
raise error.Abort(_("server does not support narrow stream clones"))
with repo.lock(): with repo.lock():
entries = [] entries = []
totalfilesize = 0 totalfilesize = 0
matcher = None matcher = None
if includes or excludes: if includes or excludes:
matcher = narrowspec.match(repo.root, includes, excludes) matcher = narrowspec.match(repo.root, includes, excludes)

tests/test-narrow-clone-stream.t

#testcases tree flat
Tests narrow stream clones Tests narrow stream clones
$ . "$TESTDIR/narrow-library.sh" $ . "$TESTDIR/narrow-library.sh"
#if tree
$ cat << EOF >> $HGRCPATH
> [experimental]
> treemanifest = 1
> EOF
#endif
Server setup Server setup
$ hg init master $ hg init master
$ cd master $ cd master
$ mkdir dir $ mkdir dir
$ mkdir dir/src $ mkdir dir/src
$ cd dir/src $ cd dir/src
$ for x in `$TESTDIR/seq.py 20`; do echo $x > "f$x"; hg add "f$x"; hg commit -m "Commit src $x"; done $ for x in `$TESTDIR/seq.py 20`; do echo $x > "f$x"; hg add "f$x"; hg commit -m "Commit src $x"; done
$ hg clone --narrow ssh://user@dummy/master narrow --noupdate --include "dir/src/f10" --stream $ hg clone --narrow ssh://user@dummy/master narrow --noupdate --include "dir/src/f10" --stream
streaming all changes streaming all changes
remote: abort: server does not support narrow stream clones remote: abort: server does not support narrow stream clones
abort: pull failed on remote abort: pull failed on remote
[255] [255]
Enable stream clone on the server Enable stream clone on the server
$ echo "[server]" >> master/.hg/hgrc $ echo "[experimental.server]" >> master/.hg/hgrc
$ echo "stream-narrow-clones=True" >> master/.hg/hgrc $ echo "stream-narrow-clones=True" >> master/.hg/hgrc
Cloning a specific file when stream clone is supported Cloning a specific file when stream clone is supported
$ hg clone --narrow ssh://user@dummy/master narrow --noupdate --include "dir/src/f10" --stream $ hg clone --narrow ssh://user@dummy/master narrow --noupdate --include "dir/src/f10" --stream
streaming all changes streaming all changes
remote: abort: server does not support narrow stream clones * files to transfer, * KB of data (glob)
abort: pull failed on remote transferred * KB in * seconds (* MB/sec) (glob)
[255]
$ cd narrow
$ ls
$ hg tracked
I path:dir/src/f10
Making sure we have the correct set of requirements
$ cat .hg/requires
dotencode
fncache
generaldelta
narrowhg-experimental
revlogv1
store
treemanifest (tree !)
Making sure store has the required files
$ ls .hg/store/
00changelog.i
00manifest.i
data
fncache
meta (tree !)
narrowspec
undo
undo.backupfiles
undo.phaseroots
Checking that repository has all the required data and not broken
$ hg verify
checking changesets
checking manifests
checking directory manifests (tree !)
crosschecking files in changesets and manifests
checking files
checked 40 changesets with 1 changes to 1 files