This is an archive of the discontinued Mercurial Phabricator instance.

Differential D695

gitgetmeta: enable serving only missing hg-git map data
ClosedPublic

Authored by singhsrb on Sep 12 2017, 6:50 PM.
Tags
None
Subscribers
durham

Details

Reviewers
durham
Group Reviewers
Restricted Project
Commits
rFBHGX24832600d3d2: gitgetmeta: enable serving only missing hg-git map data
Summary

The gitlookup extension allows for only requesting the complete file
from the server. This leads to download of large files (hg-git map file
specifically in this case) when only the missing chucks of the file would have
sufficed.

This commit makes changes to allow for downloading only the missing parts of
the map file. A config option 'onlymapdelta' has also been added to allow for
switching between the two different modes of operation: serving complete file
and serving only the missing chunks. The tests were subsequently updated to
cater to the new scenario and also, checking for interoperability of the two
modes.

Test Plan

Ran all the tests.

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

singhsrb created this revision.Sep 12 2017, 6:50 PM
Herald added a reviewer: Restricted Project. · View Herald TranscriptSep 12 2017, 6:50 PM
singhsrb added inline comments.Sep 12 2017, 6:58 PM
tests/test-git-getmeta.t
2

will change these to onlymapdelta.true and onlymapdelta.false to be consistent with the config being a bool.

singhsrb edited the summary of this revision. (Show Details)Sep 12 2017, 7:53 PM
durham requested changes to this revision.Sep 13 2017, 12:32 PM
durham added a subscriber: durham.

Looks good! The only real issue is the pickle stuff

hgext3rd/gitlookup.py
159

Usually missinghashes will be a small list. Maybe we should break the iteration over mapfile once we've found all the missinghashes? Might save some time.

204

Generally we avoid using pickle as a serialization format. Unpickling something can run arbitrary code, so it's a bit of a security hole to pass it over the wire. You can use json if you want something simple.

This revision now requires changes to proceed.Sep 13 2017, 12:32 PM
singhsrb added inline comments.Sep 13 2017, 12:54 PM
hgext3rd/gitlookup.py
159

I wasn't sure that will very useful because even though the missing hashes is a small list, we don't really know where we will find the hashes in the file. But I don't see any harm in returning when we finish anyways as it can potentially save time.

204

I wasn't aware of that. Thanks for the information! I will update this to use json instead.

singhsrb updated this revision to Diff 1793.Sep 13 2017, 3:10 PM
singhsrb added a comment.EditedSep 13 2017, 3:23 PM

FYI: I also added some tests and additional error checking/logging along with the suggested changes. Also, modified the pattern

if not len(listtype):

to

if not listtype:
hgext3rd/gitlookup.py
168

Fixed this to use mapfile.name instead of gitmapfile

durham accepted this revision.Sep 13 2017, 4:19 PM
This revision is now accepted and ready to land.Sep 13 2017, 4:19 PM
Closed by commit rFBHGX24832600d3d2: gitgetmeta: enable serving only missing hg-git map data (authored by singhsrb). · Explain WhySep 15 2017, 12:30 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mhgext3rd/gitlookup.py (158 lines)
Mtests/test-git-getmeta.t (176 lines)

Diff 1853

hgext3rd/gitlookup.py

ssh://server/repo ssh://server/repo
:: ::
[gitlookup] [gitlookup]
# Define the location of the map file with the mapfile config option. # Define the location of the map file with the mapfile config option.
mapfile = <location of map file> mapfile = <location of map file>
# The config option onlymapdelta controls how the server handles the hg-git
# map. A True value corresponds to serving only missing map data while False
# corresponds to serving the complete map.
onlymapdelta = False
''' '''
from mercurial import bundle2, exchange, encoding, extensions, hg from mercurial import bundle2, exchange, encoding, extensions, hg
from mercurial import localrepo, util, wireproto, error, registrar from mercurial import localrepo, util, wireproto, error, registrar
from mercurial.node import bin, nullid from mercurial.node import bin, hex, nullid
from mercurial.i18n import _ from mercurial.i18n import _
import errno, urllib import errno, json, urllib
cmdtable = {} cmdtable = {}
command = registrar.command(cmdtable) command = registrar.command(cmdtable)
def wrapwireprotocommand(command, wrapper): def wrapwireprotocommand(command, wrapper):
'''Wrap the wire proto command named `command' in table '''Wrap the wire proto command named `command' in table
Just like extensions.wrapcommand, except for wire protocol commands. Just like extensions.wrapcommand, except for wire protocol commands.
@command('gitgetmeta', [], '[SOURCE]') @command('gitgetmeta', [], '[SOURCE]')
def gitgetmeta(ui, repo, source='default'): def gitgetmeta(ui, repo, source='default'):
'''get git metadata from a server that supports fb_gitmeta''' '''get git metadata from a server that supports fb_gitmeta'''
source, branch = hg.parseurl(ui.expandpath(source)) source, branch = hg.parseurl(ui.expandpath(source))
other = hg.peer(repo, {}, source) other = hg.peer(repo, {}, source)
ui.status(_('getting git metadata from %s\n') % ui.status(_('getting git metadata from %s\n') %
util.hidepassword(source)) util.hidepassword(source))
kwargs = {'bundlecaps': exchange.caps20to10(repo)} kwargs = {'bundlecaps': exchange.caps20to10(repo)}
capsblob = bundle2.encodecaps(bundle2.getrepocaps(repo)) capsblob = bundle2.encodecaps(bundle2.getrepocaps(repo))
kwargs['bundlecaps'].add('bundle2=' + urllib.quote(capsblob)) kwargs['bundlecaps'].add('bundle2=' + urllib.quote(capsblob))
# this would ideally not be in the bundlecaps at all, but adding new kwargs # this would ideally not be in the bundlecaps at all, but adding new kwargs
# for wire transmissions is not possible as of Mercurial d19164a018a1 # for wire transmissions is not possible as of Mercurial d19164a018a1
kwargs['bundlecaps'].add('fb_gitmeta') kwargs['bundlecaps'].add('fb_gitmeta')
kwargs['heads'] = [nullid] kwargs['heads'] = [nullid]
kwargs['cg'] = False kwargs['cg'] = False
kwargs['common'] = _getcommonheads(repo)
bundle = other.getbundle('pull', **kwargs) bundle = other.getbundle('pull', **kwargs)
try: try:
op = bundle2.processbundle(repo, bundle) op = bundle2.processbundle(repo, bundle)
except error.BundleValueError as exc: except error.BundleValueError as exc:
raise error.Abort('missing support for %s' % exc) raise error.Abort('missing support for %s' % exc)
writebytes = op.records['fb:gitmeta:writebytes'] writebytes = op.records['fb:gitmeta:writebytes']
ui.status(_('wrote %d files (%d bytes)\n') % ui.status(_('wrote %d files (%d bytes)\n') %
(len(writebytes), sum(writebytes))) (len(writebytes), sum(writebytes)))
gitmetafiles = set(['git-mapfile', 'git-named-branches', 'git-tags', hgheadsfile = 'git-synced-hgheads'
'git-remote-refs']) gitmapfile = 'git-mapfile'
gitmetafiles = set(
[gitmapfile, 'git-named-branches', 'git-tags', 'git-remote-refs'])
def _getfile(repo, filename): def _getfile(repo, filename):
try: try:
return repo.vfs(filename) return repo.vfs(filename)
except (IOError, OSError) as e: except (IOError, OSError) as e:
if e.errno != errno.ENOENT: if e.errno != errno.ENOENT:
repo.ui.warn(_("warning: unable to read %s: %s\n") % (filename, e)) repo.ui.warn(_("warning: unable to read %s: %s\n") % (filename, e))
return None return None
def _getcommonheads(repo):
commonheads = []
f = _getfile(repo, hgheadsfile)
if f:
commonheads = f.readlines()
commonheads = [bin(x.strip()) for x in commonheads]
return commonheads
def _isheadmissing(repo, heads):
return not all(repo.known(heads))
def _getmissinglines(mapfile, missinghashes):
missinglines = set()
# Avoid expensive lookup through the map file if there is no missing hash.
if not missinghashes:
return missinglines
hashestofind = missinghashes.copy()
for line in mapfile:
gitsha, hgsha = line.strip().split(' ', 1)
durhamUnsubmitted
Not Done

Usually missinghashes will be a small list. Maybe we should break the iteration over mapfile once we've found all the missinghashes? Might save some time.

durham: Usually missinghashes will be a small list. Maybe we should break the iteration over mapfile…
singhsrbAuthorUnsubmitted
Not Done

I wasn't sure that will very useful because even though the missing hashes is a small list, we don't really know where we will find the hashes in the file. But I don't see any harm in returning when we finish anyways as it can potentially save time.

singhsrb: I wasn't sure that will very useful because even though the missing hashes is a small list, we…
if hgsha in hashestofind:
missinglines.add(line)
# Return the missing lines if we found all of them.
hashestofind.remove(hgsha)
if not hashestofind:
return missinglines
raise error.Abort(_('gitmeta: missing hashes in file %s') % mapfile.name)
singhsrbAuthorUnsubmitted
Not Done

Fixed this to use mapfile.name instead of gitmapfile

singhsrb: Fixed this to use `mapfile.name` instead of `gitmapfile`
class _githgmappayload(object):
def __init__(self, needfullsync, newheads, missinglines):
self.needfullsync = needfullsync
self.newheads = newheads
self.missinglines = missinglines
def _todict(self):
d = {}
d['needfullsync'] = self.needfullsync
d['newheads'] = list(self.newheads)
d['missinglines'] = list(self.missinglines)
return d
def tojson(self):
return json.dumps(self._todict())
@classmethod
def _fromdict(cls, d):
needfullsync = d['needfullsync']
newheads = set(d['newheads'])
missinglines = set(d['missinglines'])
return cls(needfullsync, newheads, missinglines)
@classmethod
def fromjson(cls, jsonstr):
d = json.loads(jsonstr)
return cls._fromdict(d)
@exchange.getbundle2partsgenerator('b2x:fb:gitmeta:githgmap')
def _getbundlegithgmappart(bundler, repo, source, bundlecaps=None, **kwargs):
'''send missing git to hg map data via bundle2'''
if 'fb_gitmeta' in bundlecaps:
# Do nothing if the config indicates serving the complete git-hg map
# file. _getbundlegitmetapart will handle serving the complete file in
# this case.
durhamUnsubmitted
Not Done

Generally we avoid using pickle as a serialization format. Unpickling something can run arbitrary code, so it's a bit of a security hole to pass it over the wire. You can use json if you want something simple.

durham: Generally we avoid using pickle as a serialization format. Unpickling something can run…
singhsrbAuthorUnsubmitted
Not Done

I wasn't aware of that. Thanks for the information! I will update this to use json instead.

singhsrb: I wasn't aware of that. Thanks for the information! I will update this to use json instead.
if not repo.ui.configbool('gitlookup', 'onlymapdelta', False):
return
mapfile = _getfile(repo, gitmapfile)
if not mapfile:
return
commonheads = kwargs['common']
# If there are missing heads, we will sync everything.
if _isheadmissing(repo, commonheads):
commonheads = []
needfullsync = (len(commonheads) == 0)
heads = repo.heads()
newheads = set(hex(head) for head in heads)
missingcommits = repo.changelog.findmissing(commonheads, heads)
missinghashes = set(hex(commit) for commit in missingcommits)
missinglines = _getmissinglines(mapfile, missinghashes)
payload = _githgmappayload(needfullsync, newheads, missinglines)
serializedpayload = payload.tojson()
part = bundle2.bundlepart(
'b2x:fb:gitmeta:githgmap',
[('filename', gitmapfile)],
data = serializedpayload
)
bundler.addpart(part)
@exchange.getbundle2partsgenerator('b2x:fb:gitmeta') @exchange.getbundle2partsgenerator('b2x:fb:gitmeta')
def _getbundlegitmetapart(bundler, repo, source, bundlecaps=None, **kwargs): def _getbundlegitmetapart(bundler, repo, source, bundlecaps=None, **kwargs):
'''send git metadata via bundle2''' '''send git metadata via bundle2'''
if 'fb_gitmeta' in bundlecaps: if 'fb_gitmeta' in bundlecaps:
for fname in sorted(gitmetafiles): filestooverwrite = gitmetafiles
# Exclude the git-hg map file if the config indicates that the server
# should only be serving the missing map data. _getbundle2partsgenerator
# will serve the missing map data in this case.
if repo.ui.configbool('gitlookup', 'onlymapdelta', False):
filestooverwrite = filestooverwrite - set([gitmapfile])
for fname in sorted(filestooverwrite):
f = _getfile(repo, fname) f = _getfile(repo, fname)
if not f: if not f:
continue continue
part = bundle2.bundlepart('b2x:fb:gitmeta', part = bundle2.bundlepart('b2x:fb:gitmeta',
[('filename', fname)], [('filename', fname)],
data=f.read()) data=f.read())
bundler.addpart(part) bundler.addpart(part)
fname = params['filename'] fname = params['filename']
if fname not in gitmetafiles: if fname not in gitmetafiles:
op.repo.ui.warn( op.repo.ui.warn(
_("warning: gitmeta: unknown file '%s' skipped\n") % fname) _("warning: gitmeta: unknown file '%s' skipped\n") % fname)
return False return False
return True return True
@bundle2.parthandler('b2x:fb:gitmeta:githgmap', ('filename',))
@bundle2.parthandler('fb:gitmeta:githgmap', ('filename',))
def bundle2getgithgmap(op, part):
params = dict(part.mandatoryparams)
if _validatepartparams(op, params):
filename = params['filename']
with op.repo.wlock():
data = _githgmappayload.fromjson(part.read())
missinglines = data.missinglines
# No need to update anything if already in sync.
if not missinglines:
return
if data.needfullsync:
newlines = missinglines
else:
mapfile = _getfile(op.repo, filename)
if mapfile:
currentlines = set(mapfile.readlines())
if currentlines & missinglines:
msg = 'warning: gitmeta: unexpected lines in .hg/%s\n'
op.repo.ui.warn(_(msg) % filename)
currentlines.update(missinglines)
newlines = currentlines
else:
raise error.Abort(
_('gitmeta: could not read from .hg/%s') % filename)
_writefile(op, filename, ''.join(newlines))
_writefile(op, hgheadsfile, '\n'.join(data.newheads))
@bundle2.parthandler('b2x:fb:gitmeta', ('filename',)) @bundle2.parthandler('b2x:fb:gitmeta', ('filename',))
@bundle2.parthandler('fb:gitmeta', ('filename',)) @bundle2.parthandler('fb:gitmeta', ('filename',))
def bundle2getgitmeta(op, part): def bundle2getgitmeta(op, part):
'''unbundle a bundle2 containing git metadata on the client''' '''unbundle a bundle2 containing git metadata on the client'''
params = dict(part.mandatoryparams) params = dict(part.mandatoryparams)
if _validatepartparams(op, params): if _validatepartparams(op, params):
filename = params['filename'] filename = params['filename']
with op.repo.wlock(): with op.repo.wlock():
data = part.read() data = part.read()
_writefile(op, filename, data) _writefile(op, filename, data)
def extsetup(ui): def extsetup(ui):
wrapwireprotocommand('lookup', remotelookup) wrapwireprotocommand('lookup', remotelookup)
extensions.wrapfunction(localrepo.localrepository, 'lookup', locallookup) extensions.wrapfunction(localrepo.localrepository, 'lookup', locallookup)

tests/test-git-getmeta.t

#testcases onlymapdelta.true onlymapdelta.false
singhsrbAuthorUnsubmitted
Not Done

will change these to onlymapdelta.true and onlymapdelta.false to be consistent with the config being a bool.

singhsrb: will change these to onlymapdelta.true and onlymapdelta.false to be consistent with the config…
$ echo "[extensions]" >> $HGRCPATH $ echo "[extensions]" >> $HGRCPATH
$ echo "gitlookup = $TESTDIR/../hgext3rd/gitlookup.py" >> $HGRCPATH $ echo "gitlookup = $TESTDIR/../hgext3rd/gitlookup.py" >> $HGRCPATH
$ echo "gitrevset = $TESTDIR/../hgext3rd/gitrevset.py" >> $HGRCPATH $ echo "gitrevset = $TESTDIR/../hgext3rd/gitrevset.py" >> $HGRCPATH
$ echo "strip = " >> $HGRCPATH $ echo "strip = " >> $HGRCPATH
$ echo "[ui]" >> $HGRCPATH $ echo "[ui]" >> $HGRCPATH
$ echo 'ssh = python "$RUNTESTDIR/dummyssh"' >> $HGRCPATH $ echo 'ssh = python "$RUNTESTDIR/dummyssh"' >> $HGRCPATH
Set up the hg-git files Set up the hg-git files
$ echo "mapfile = $TESTTMP/repo1/.hg/git-mapfile" >> hgrc $ echo "mapfile = $TESTTMP/repo1/.hg/git-mapfile" >> hgrc
Clone new repo from local repo and check that git metadata syncs properly Clone new repo from local repo and check that git metadata syncs properly
------------------------------------------------------------------------- -------------------------------------------------------------------------
$ cd ../.. $ cd ../..
$ hg clone repo1 repo2 -q $ hg clone repo1 repo2 -q
$ cd repo2 $ cd repo2
$ hg gitgetmeta -v $ hg gitgetmeta -v
getting git metadata from $TESTTMP/repo1 getting git metadata from $TESTTMP/repo1
writing .hg/git-mapfile writing .hg/git-mapfile
writing .hg/git-remote-refs writing .hg/git-remote-refs
writing .hg/git-tags writing .hg/git-tags
wrote 3 files (183 bytes) wrote 3 files (183 bytes)
$ sort .hg/git-mapfile $ sort .hg/git-mapfile
ffffffffffffffffffffffffffffffffffffffff fc5f87aa174b7d4016abf3e908fd63cc99774540 ffffffffffffffffffffffffffffffffffffffff fc5f87aa174b7d4016abf3e908fd63cc99774540
$ cat .hg/git-remote-refs $ cat .hg/git-remote-refs
ffffffffffffffffffffffffffffffffffffffff default/master ffffffffffffffffffffffffffffffffffffffff default/master
$ cat .hg/git-tags $ cat .hg/git-tags
ffffffffffffffffffffffffffffffffffffffff 0.1 ffffffffffffffffffffffffffffffffffffffff 0.1
Simulate config change from serving complete hg map to only missing delta
-------------------------------------------------------------------------
Making this change here instead of during repo setup earlier ensures that we
test scenarios where the config changes after repos have been syncing.
#if onlymapdelta.true
$ cd ../repo1/.hg
$ echo "onlymapdelta = True" >> hgrc
$ cd ..
#endif
Clone new repo from remote repo and check that git metadata syncs properly Clone new repo from remote repo and check that git metadata syncs properly
-------------------------------------------------------------------------- --------------------------------------------------------------------------
$ cd .. $ cd ..
$ hg clone ssh://user@dummy/repo1 repo3 -q $ hg clone ssh://user@dummy/repo1 repo3 -q
$ cd repo3 $ cd repo3
#if onlymapdelta.true
$ hg gitgetmeta -v
getting git metadata from ssh://user@dummy/repo1
writing .hg/git-mapfile
writing .hg/git-synced-hgheads
writing .hg/git-remote-refs
writing .hg/git-tags
wrote 4 files (223 bytes)
$ sort .hg/git-synced-hgheads
fc5f87aa174b7d4016abf3e908fd63cc99774540
#else
$ hg gitgetmeta -v $ hg gitgetmeta -v
getting git metadata from ssh://user@dummy/repo1 getting git metadata from ssh://user@dummy/repo1
writing .hg/git-mapfile writing .hg/git-mapfile
writing .hg/git-remote-refs writing .hg/git-remote-refs
writing .hg/git-tags writing .hg/git-tags
wrote 3 files (183 bytes) wrote 3 files (183 bytes)
#endif
$ sort .hg/git-mapfile $ sort .hg/git-mapfile
ffffffffffffffffffffffffffffffffffffffff fc5f87aa174b7d4016abf3e908fd63cc99774540 ffffffffffffffffffffffffffffffffffffffff fc5f87aa174b7d4016abf3e908fd63cc99774540
$ cat .hg/git-remote-refs $ cat .hg/git-remote-refs
ffffffffffffffffffffffffffffffffffffffff default/master ffffffffffffffffffffffffffffffffffffffff default/master
$ cat .hg/git-tags $ cat .hg/git-tags
ffffffffffffffffffffffffffffffffffffffff 0.1 ffffffffffffffffffffffffffffffffffffffff 0.1
Redundant sync just to see that the hg-git map file is not synced with
onlymapdelta being True
#if onlymapdelta.true
$ hg gitgetmeta -v
getting git metadata from ssh://user@dummy/repo1
writing .hg/git-remote-refs
writing .hg/git-tags
wrote 2 files (101 bytes)
$ cat .hg/git-remote-refs
ffffffffffffffffffffffffffffffffffffffff default/master
$ cat .hg/git-tags
ffffffffffffffffffffffffffffffffffffffff 0.1
#endif
Make changes upstream and check that they get reflected in clones Make changes upstream and check that they get reflected in clones
----------------------------------------------------------------- -----------------------------------------------------------------
$ cd ../repo1 $ cd ../repo1
$ echo >> a $ echo >> a
$ hg ci -m "modifying a" $ hg ci -m "modifying a"
$ hg log -r . --template "{node}\n" $ hg log -r . --template "{node}\n"
d4a59f7c570a8794e6ec20865090e7b848395b92 d4a59f7c570a8794e6ec20865090e7b848395b92
$ echo "1111111111111111111111111111111111111111 d4a59f7c570a8794e6ec20865090e7b848395b92" >> .hg/git-mapfile $ echo "1111111111111111111111111111111111111111 d4a59f7c570a8794e6ec20865090e7b848395b92" >> .hg/git-mapfile
Check local repo syncing Check local repo syncing
$ cd ../repo2 $ cd ../repo2
#if onlymapdelta.true
$ hg gitgetmeta -v
getting git metadata from $TESTTMP/repo1
writing .hg/git-mapfile
writing .hg/git-synced-hgheads
writing .hg/git-remote-refs
writing .hg/git-tags
wrote 4 files (305 bytes)
$ sort .hg/git-synced-hgheads
d4a59f7c570a8794e6ec20865090e7b848395b92
#else
$ hg gitgetmeta -v $ hg gitgetmeta -v
getting git metadata from $TESTTMP/repo1 getting git metadata from $TESTTMP/repo1
writing .hg/git-mapfile writing .hg/git-mapfile
writing .hg/git-remote-refs writing .hg/git-remote-refs
writing .hg/git-tags writing .hg/git-tags
wrote 3 files (265 bytes) wrote 3 files (265 bytes)
#endif
$ sort .hg/git-mapfile $ sort .hg/git-mapfile
1111111111111111111111111111111111111111 d4a59f7c570a8794e6ec20865090e7b848395b92 1111111111111111111111111111111111111111 d4a59f7c570a8794e6ec20865090e7b848395b92
ffffffffffffffffffffffffffffffffffffffff fc5f87aa174b7d4016abf3e908fd63cc99774540 ffffffffffffffffffffffffffffffffffffffff fc5f87aa174b7d4016abf3e908fd63cc99774540
$ cat .hg/git-remote-refs $ cat .hg/git-remote-refs
ffffffffffffffffffffffffffffffffffffffff default/master ffffffffffffffffffffffffffffffffffffffff default/master
$ hg ci -qm "creating new head with modified a" $ hg ci -qm "creating new head with modified a"
$ hg log -r . --template "{node}\n" $ hg log -r . --template "{node}\n"
8ea31c3efb6d2edb6d9fe608c29034e7e7ed5f91 8ea31c3efb6d2edb6d9fe608c29034e7e7ed5f91
$ echo "3333333333333333333333333333333333333333 8ea31c3efb6d2edb6d9fe608c29034e7e7ed5f91" >> .hg/git-mapfile $ echo "3333333333333333333333333333333333333333 8ea31c3efb6d2edb6d9fe608c29034e7e7ed5f91" >> .hg/git-mapfile
$ echo "releases/foo1 foo1" >> .hg/git-named-branches $ echo "releases/foo1 foo1" >> .hg/git-named-branches
Check remote repo syncing Check remote repo syncing
$ cd ../repo3 $ cd ../repo3
#if onlymapdelta.true
$ hg gitgetmeta -v
getting git metadata from ssh://user@dummy/repo1
writing .hg/git-mapfile
writing .hg/git-synced-hgheads
writing .hg/git-named-branches
writing .hg/git-remote-refs
writing .hg/git-tags
wrote 5 files (529 bytes)
$ sort .hg/git-synced-hgheads
8ea31c3efb6d2edb6d9fe608c29034e7e7ed5f91
c411819f7fd6036d50b17a28d3edb7aa9121985a
#else
$ hg gitgetmeta -v $ hg gitgetmeta -v
getting git metadata from ssh://user@dummy/repo1 getting git metadata from ssh://user@dummy/repo1
writing .hg/git-mapfile writing .hg/git-mapfile
writing .hg/git-named-branches writing .hg/git-named-branches
writing .hg/git-remote-refs writing .hg/git-remote-refs
writing .hg/git-tags writing .hg/git-tags
wrote 4 files (448 bytes) wrote 4 files (448 bytes)
#endif
$ sort .hg/git-mapfile $ sort .hg/git-mapfile
1111111111111111111111111111111111111111 d4a59f7c570a8794e6ec20865090e7b848395b92 1111111111111111111111111111111111111111 d4a59f7c570a8794e6ec20865090e7b848395b92
2222222222222222222222222222222222222222 c411819f7fd6036d50b17a28d3edb7aa9121985a 2222222222222222222222222222222222222222 c411819f7fd6036d50b17a28d3edb7aa9121985a
3333333333333333333333333333333333333333 8ea31c3efb6d2edb6d9fe608c29034e7e7ed5f91 3333333333333333333333333333333333333333 8ea31c3efb6d2edb6d9fe608c29034e7e7ed5f91
ffffffffffffffffffffffffffffffffffffffff fc5f87aa174b7d4016abf3e908fd63cc99774540 ffffffffffffffffffffffffffffffffffffffff fc5f87aa174b7d4016abf3e908fd63cc99774540
$ cat .hg/git-named-branches $ cat .hg/git-named-branches
$ cd ../repo1 $ cd ../repo1
$ hg strip . -q $ hg strip . -q
$ grep -v "8ea31c3efb6d2edb6d9fe608c29034e7e7ed5f91" .hg/git-mapfile > tempfile $ grep -v "8ea31c3efb6d2edb6d9fe608c29034e7e7ed5f91" .hg/git-mapfile > tempfile
$ mv tempfile .hg/git-mapfile $ mv tempfile .hg/git-mapfile
Check local repo syncing Check local repo syncing
$ cd ../repo2 $ cd ../repo2
#if onlymapdelta.true
$ hg gitgetmeta -v
getting git metadata from $TESTTMP/repo1
writing .hg/git-mapfile
writing .hg/git-synced-hgheads
writing .hg/git-named-branches
writing .hg/git-remote-refs
writing .hg/git-tags
wrote 5 files (406 bytes)
$ sort .hg/git-synced-hgheads
c411819f7fd6036d50b17a28d3edb7aa9121985a
#else
$ hg gitgetmeta -v $ hg gitgetmeta -v
getting git metadata from $TESTTMP/repo1 getting git metadata from $TESTTMP/repo1
writing .hg/git-mapfile writing .hg/git-mapfile
writing .hg/git-named-branches writing .hg/git-named-branches
writing .hg/git-remote-refs writing .hg/git-remote-refs
writing .hg/git-tags writing .hg/git-tags
wrote 4 files (366 bytes) wrote 4 files (366 bytes)
#endif
$ sort .hg/git-mapfile $ sort .hg/git-mapfile
1111111111111111111111111111111111111111 d4a59f7c570a8794e6ec20865090e7b848395b92 1111111111111111111111111111111111111111 d4a59f7c570a8794e6ec20865090e7b848395b92
2222222222222222222222222222222222222222 c411819f7fd6036d50b17a28d3edb7aa9121985a 2222222222222222222222222222222222222222 c411819f7fd6036d50b17a28d3edb7aa9121985a
ffffffffffffffffffffffffffffffffffffffff fc5f87aa174b7d4016abf3e908fd63cc99774540 ffffffffffffffffffffffffffffffffffffffff fc5f87aa174b7d4016abf3e908fd63cc99774540
$ cat .hg/git-named-branches $ cat .hg/git-named-branches
releases/foo1 foo1 releases/foo1 foo1
$ echo a >> a $ echo a >> a
$ hg ci -qm "modifying a" $ hg ci -qm "modifying a"
$ hg log -r . --template "{node}\n" $ hg log -r . --template "{node}\n"
3bfa460515b210d1e6f7e21bde166ef5c5f0d9b6 3bfa460515b210d1e6f7e21bde166ef5c5f0d9b6
$ echo "2222222222222222222222222222222222222222 3bfa460515b210d1e6f7e21bde166ef5c5f0d9b6" >> .hg/git-mapfile $ echo "2222222222222222222222222222222222222222 3bfa460515b210d1e6f7e21bde166ef5c5f0d9b6" >> .hg/git-mapfile
Check remote repo syncing Check remote repo syncing
$ cd ../repo3 $ cd ../repo3
#if onlymapdelta.true
$ hg gitgetmeta -v
getting git metadata from ssh://user@dummy/repo1
writing .hg/git-mapfile
writing .hg/git-synced-hgheads
writing .hg/git-named-branches
writing .hg/git-remote-refs
writing .hg/git-tags
wrote 5 files (324 bytes)
$ sort .hg/git-synced-hgheads
3bfa460515b210d1e6f7e21bde166ef5c5f0d9b6
#else
$ hg gitgetmeta -v $ hg gitgetmeta -v
getting git metadata from ssh://user@dummy/repo1 getting git metadata from ssh://user@dummy/repo1
writing .hg/git-mapfile writing .hg/git-mapfile
writing .hg/git-named-branches writing .hg/git-named-branches
writing .hg/git-remote-refs writing .hg/git-remote-refs
writing .hg/git-tags writing .hg/git-tags
wrote 4 files (284 bytes) wrote 4 files (284 bytes)
#endif
$ sort .hg/git-mapfile $ sort .hg/git-mapfile
2222222222222222222222222222222222222222 3bfa460515b210d1e6f7e21bde166ef5c5f0d9b6 2222222222222222222222222222222222222222 3bfa460515b210d1e6f7e21bde166ef5c5f0d9b6
ffffffffffffffffffffffffffffffffffffffff fc5f87aa174b7d4016abf3e908fd63cc99774540 ffffffffffffffffffffffffffffffffffffffff fc5f87aa174b7d4016abf3e908fd63cc99774540
$ cat .hg/git-named-branches $ cat .hg/git-named-branches
releases/foo1 foo1 releases/foo1 foo1
$ cat .hg/git-remote-refs $ cat .hg/git-remote-refs
ffffffffffffffffffffffffffffffffffffffff default/master ffffffffffffffffffffffffffffffffffffffff default/master
$ cat .hg/git-tags $ cat .hg/git-tags
ffffffffffffffffffffffffffffffffffffffff 0.1 ffffffffffffffffffffffffffffffffffffffff 0.1
Check local repo syncing Check local repo syncing
$ cd ../repo2 $ cd ../repo2
#if onlymapdelta.true
$ hg gitgetmeta -v
getting git metadata from $TESTTMP/repo1
writing .hg/git-mapfile
writing .hg/git-synced-hgheads
writing .hg/git-named-branches
writing .hg/git-remote-refs
writing .hg/git-tags
wrote 5 files (324 bytes)
$ sort .hg/git-synced-hgheads
3bfa460515b210d1e6f7e21bde166ef5c5f0d9b6
#else
$ hg gitgetmeta -v $ hg gitgetmeta -v
getting git metadata from $TESTTMP/repo1 getting git metadata from $TESTTMP/repo1
writing .hg/git-mapfile writing .hg/git-mapfile
writing .hg/git-named-branches writing .hg/git-named-branches
writing .hg/git-remote-refs writing .hg/git-remote-refs
writing .hg/git-tags writing .hg/git-tags
wrote 4 files (284 bytes) wrote 4 files (284 bytes)
#endif
$ sort .hg/git-mapfile $ sort .hg/git-mapfile
2222222222222222222222222222222222222222 3bfa460515b210d1e6f7e21bde166ef5c5f0d9b6 2222222222222222222222222222222222222222 3bfa460515b210d1e6f7e21bde166ef5c5f0d9b6
ffffffffffffffffffffffffffffffffffffffff fc5f87aa174b7d4016abf3e908fd63cc99774540 ffffffffffffffffffffffffffffffffffffffff fc5f87aa174b7d4016abf3e908fd63cc99774540
$ cat .hg/git-named-branches $ cat .hg/git-named-branches
releases/foo1 foo1 releases/foo1 foo1
$ touch b $ touch b
$ hg ci -Aqm "adding b" $ hg ci -Aqm "adding b"
$ hg log -r . --template "{node}\n" $ hg log -r . --template "{node}\n"
627ddeb6657d60a21b87c725b5c4e60d91b75f19 627ddeb6657d60a21b87c725b5c4e60d91b75f19
$ echo "3333333333333333333333333333333333333333 627ddeb6657d60a21b87c725b5c4e60d91b75f19" >> .hg/git-mapfile $ echo "3333333333333333333333333333333333333333 627ddeb6657d60a21b87c725b5c4e60d91b75f19" >> .hg/git-mapfile
Check local repo syncing Check local repo syncing
$ cd ../repo2 $ cd ../repo2
#if onlymapdelta.true
$ hg gitgetmeta -v
getting git metadata from $TESTTMP/repo1
writing .hg/git-mapfile
writing .hg/git-synced-hgheads
writing .hg/git-named-branches
writing .hg/git-remote-refs
writing .hg/git-tags
wrote 5 files (447 bytes)
$ sort .hg/git-synced-hgheads
3bfa460515b210d1e6f7e21bde166ef5c5f0d9b6
627ddeb6657d60a21b87c725b5c4e60d91b75f19
#else
$ hg gitgetmeta -v $ hg gitgetmeta -v
getting git metadata from $TESTTMP/repo1 getting git metadata from $TESTTMP/repo1
writing .hg/git-mapfile writing .hg/git-mapfile
writing .hg/git-named-branches writing .hg/git-named-branches
writing .hg/git-remote-refs writing .hg/git-remote-refs
writing .hg/git-tags writing .hg/git-tags
wrote 4 files (366 bytes) wrote 4 files (366 bytes)
#endif
$ sort .hg/git-mapfile $ sort .hg/git-mapfile
2222222222222222222222222222222222222222 3bfa460515b210d1e6f7e21bde166ef5c5f0d9b6 2222222222222222222222222222222222222222 3bfa460515b210d1e6f7e21bde166ef5c5f0d9b6
3333333333333333333333333333333333333333 627ddeb6657d60a21b87c725b5c4e60d91b75f19 3333333333333333333333333333333333333333 627ddeb6657d60a21b87c725b5c4e60d91b75f19
ffffffffffffffffffffffffffffffffffffffff fc5f87aa174b7d4016abf3e908fd63cc99774540 ffffffffffffffffffffffffffffffffffffffff fc5f87aa174b7d4016abf3e908fd63cc99774540
$ cat .hg/git-named-branches $ cat .hg/git-named-branches
releases/foo1 foo1 releases/foo1 foo1
$ cat .hg/git-remote-refs $ cat .hg/git-remote-refs
ffffffffffffffffffffffffffffffffffffffff default/master ffffffffffffffffffffffffffffffffffffffff default/master
$ cat .hg/git-tags $ cat .hg/git-tags
ffffffffffffffffffffffffffffffffffffffff 0.1 ffffffffffffffffffffffffffffffffffffffff 0.1
Check corrupted git-hg map data
-------------------------------
This is only valid when we are serving missing map data because when we serve
the complete map, we just simply serve the file without any validations on the
map data.
#if onlymapdelta.true
$ cd ../repo1
$ touch a
$ hg ci -Aqm "adding a"
$ hg log -r . --template "{node}\n"
1e2e1480acd77a0155ee53e30aab1bb4a08f9f22
Not updating the map file intentionally to simulate missing map data. Instead,
we try to sync changes and check that the syncing fails.
$ cd ../repo2
$ hg gitgetmeta -v
getting git metadata from $TESTTMP/repo1
abort: gitmeta: missing hashes in file $TESTTMP/repo1/.hg/git-mapfile
[255]
Now adding the map entries to both the repos to simulate corruption on the
client side
$ cd ../repo1
$ echo "4444444444444444444444444444444444444444 1e2e1480acd77a0155ee53e30aab1bb4a08f9f22" >> .hg/git-mapfile
$ cd ../repo2
$ echo "4444444444444444444444444444444444444444 1e2e1480acd77a0155ee53e30aab1bb4a08f9f22" >> .hg/git-mapfile
$ hg gitgetmeta -v
getting git metadata from $TESTTMP/repo1
warning: gitmeta: unexpected lines in .hg/git-mapfile
writing .hg/git-mapfile
writing .hg/git-synced-hgheads
writing .hg/git-named-branches
writing .hg/git-remote-refs
writing .hg/git-tags
wrote 5 files (529 bytes)
Strip the last commit and restore map entries to have same state as
onlymapdelta.false
$ grep -v "1e2e1480acd77a0155ee53e30aab1bb4a08f9f22" .hg/git-mapfile > tempfile
$ mv tempfile .hg/git-mapfile
$ cd ../repo1
$ hg strip -q "tip"
$ grep -v "1e2e1480acd77a0155ee53e30aab1bb4a08f9f22" .hg/git-mapfile > tempfile
$ mv tempfile .hg/git-mapfile
#endif
Check that our revset and template mappings work Check that our revset and template mappings work
------------------------------------------------ ------------------------------------------------
Clone a new repo Clone a new repo
$ cd .. $ cd ..
$ hg clone ssh://user@dummy/repo1 repo-ssh -q $ hg clone ssh://user@dummy/repo1 repo-ssh -q
$ cd repo-ssh $ cd repo-ssh