This is an archive of the discontinued Mercurial Phabricator instance.

Differential D7616

exchange: ensure all outgoing subrepo references are present before pushing
ClosedPublic

Authored by mharbison72 on Dec 12 2019, 1:54 PM.

Details

Reviewers
pulkit
Group Reviewers
hg-reviewers
Commits
rHG4b7d5d10c45d: exchange: ensure all outgoing subrepo references are present before pushing
Summary

We've run into occasional problems with people committing a repo, and then
amending or rebasing in the subrepo. That makes it so that the revision in the
parent can't be checked out, and the problem gets propagated on push. Mercurial
already tries to defend against this sort of dangling reference by pushing *all*
subrepo revisions first. This reuses the checks that trigger warnings in
hg verify to bail on the push unless using --force.

I thought about putting this on the server side, but at that point, all of the
data has been transferred, only to bail out. Additionally, SCM Manager hosts
subrepos in a location that isn't nested in the parent, so normal subrepo code
would complain that the subrepo is missing when run on the server.

Because the push command pushes subrepos before calling this exchange code, a
subrepo will be pushed before the parent is verified. Not great, but no
dangling references are exchanged, so it solves the problem. This code isn't in
the loop that pushes the subrepos because:

  1. the list of outgoing revisions is needed to limit the scope of the check
  2. the loop only accesses the current revision, and therefore can miss subrepos that were dropped in previous commits
  3. this code is called when pushing a subrepo, so the protection is recursive

I'm not sure if there's a cheap check for the list of files in the outgoing
bundle. If there is, that would provide a fast path to bypass this check for
people not using subrepos (or if no subrepo changes were made). There's
probably also room for verifying other references like tags. But since that
doesn't break checkouts, it's much less of a problem.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

mharbison72 created this revision.Dec 12 2019, 1:54 PM
pulkit accepted this revision.Dec 17 2019, 7:03 AM
This revision is now accepted and ready to land.Dec 17 2019, 7:03 AM

Revision Contents
Changeset List

PathPackages
Mmercurial/exchange.py (13 lines)
Mmercurial/subrepo.py (37 lines)
Mtests/test-amend-subrepo.t (31 lines)

Diff 18808

mercurial/exchange.py

pushop.ui.debug(msg) pushop.ui.debug(msg)
with wlock or util.nullcontextmanager(): with wlock or util.nullcontextmanager():
with lock or util.nullcontextmanager(): with lock or util.nullcontextmanager():
with pushop.trmanager or util.nullcontextmanager(): with pushop.trmanager or util.nullcontextmanager():
pushop.repo.checkpush(pushop) pushop.repo.checkpush(pushop)
_checkpublish(pushop) _checkpublish(pushop)
_pushdiscovery(pushop) _pushdiscovery(pushop)
if not pushop.force:
_checksubrepostate(pushop)
if not _forcebundle1(pushop): if not _forcebundle1(pushop):
_pushbundle2(pushop) _pushbundle2(pushop)
_pushchangeset(pushop) _pushchangeset(pushop)
_pushsyncphase(pushop) _pushsyncphase(pushop)
_pushobsolete(pushop) _pushobsolete(pushop)
_pushbookmark(pushop) _pushbookmark(pushop)
if repo.ui.configbool(b'experimental', b'remotenames'): if repo.ui.configbool(b'experimental', b'remotenames'):
def _pushdiscovery(pushop): def _pushdiscovery(pushop):
"""Run all discovery steps""" """Run all discovery steps"""
for stepname in pushdiscoveryorder: for stepname in pushdiscoveryorder:
step = pushdiscoverymapping[stepname] step = pushdiscoverymapping[stepname]
step(pushop) step(pushop)
def _checksubrepostate(pushop):
"""Ensure all outgoing referenced subrepo revisions are present locally"""
for n in pushop.outgoing.missing:
ctx = pushop.repo[n]
if b'.hgsub' in ctx.manifest() and b'.hgsubstate' in ctx.files():
for subpath in sorted(ctx.substate):
sub = ctx.sub(subpath)
sub.verify(onpush=True)
@pushdiscovery(b'changeset') @pushdiscovery(b'changeset')
def _pushdiscoverychangeset(pushop): def _pushdiscoverychangeset(pushop):
"""discover the changeset that need to be pushed""" """discover the changeset that need to be pushed"""
fci = discovery.findcommonincoming fci = discovery.findcommonincoming
if pushop.revs: if pushop.revs:
commoninc = fci( commoninc = fci(
pushop.repo, pushop.repo,
pushop.remote, pushop.remote,

mercurial/subrepo.py

def shortid(self, revid): def shortid(self, revid):
return revid return revid
def unshare(self): def unshare(self):
''' '''
convert this repository from shared to normal storage. convert this repository from shared to normal storage.
''' '''
def verify(self): def verify(self, onpush=False):
'''verify the integrity of the repository. Return 0 on success or """verify the revision of this repository that is held in `_state` is
warning, 1 on any error. present and not hidden. Return 0 on success or warning, 1 on any
''' error. In the case of ``onpush``, warnings or errors will raise an
exception if the result of pushing would be a broken remote repository.
"""
return 0 return 0
@propertycache @propertycache
def wvfs(self): def wvfs(self):
"""return vfs to access the working directory of this subrepository """return vfs to access the working directory of this subrepository
""" """
return vfsmod.vfs(self._ctx.repo().wvfs.join(self._path)) return vfsmod.vfs(self._ctx.repo().wvfs.join(self._path))
# Nothing prevents a user from sharing in a repo, and then making that a # Nothing prevents a user from sharing in a repo, and then making that a
# subrepo. Alternately, the previous unshare attempt may have failed # subrepo. Alternately, the previous unshare attempt may have failed
# part way through. So recurse whether or not this layer is shared. # part way through. So recurse whether or not this layer is shared.
if self._repo.shared(): if self._repo.shared():
self.ui.status(_(b"unsharing subrepo '%s'\n") % self._relpath) self.ui.status(_(b"unsharing subrepo '%s'\n") % self._relpath)
hg.unshare(self.ui, self._repo) hg.unshare(self.ui, self._repo)
def verify(self): def verify(self, onpush=False):
try: try:
rev = self._state[1] rev = self._state[1]
ctx = self._repo.unfiltered()[rev] ctx = self._repo.unfiltered()[rev]
if ctx.hidden(): if ctx.hidden():
# Since hidden revisions aren't pushed/pulled, it seems worth an # Since hidden revisions aren't pushed/pulled, it seems worth an
# explicit warning. # explicit warning.
ui = self._repo.ui msg = _(b"subrepo '%s' is hidden in revision %s") % (
ui.warn( self._relpath,
_(b"subrepo '%s' is hidden in revision %s\n") node.short(self._ctx.node()),
% (self._relpath, node.short(self._ctx.node()))
) )
if onpush:
raise error.Abort(msg)
else:
self._repo.ui.warn(b'%s\n' % msg)
return 0 return 0
except error.RepoLookupError: except error.RepoLookupError:
# A missing subrepo revision may be a case of needing to pull it, so # A missing subrepo revision may be a case of needing to pull it, so
# don't treat this as an error. # don't treat this as an error for `hg verify`.
self._repo.ui.warn( msg = _(b"subrepo '%s' not found in revision %s") % (
_(b"subrepo '%s' not found in revision %s\n") self._relpath,
% (self._relpath, node.short(self._ctx.node())) node.short(self._ctx.node()),
) )
if onpush:
raise error.Abort(msg)
else:
self._repo.ui.warn(b'%s\n' % msg)
return 0 return 0
@propertycache @propertycache
def wvfs(self): def wvfs(self):
"""return own wvfs for efficiency and consistency """return own wvfs for efficiency and consistency
""" """
return self._repo.wvfs return self._repo.wvfs

tests/test-amend-subrepo.t

$ hg rm .hgsub $ hg rm .hgsub
$ hg amend $ hg amend
saved backup bundle to * (glob) (obsstore-off !) saved backup bundle to * (glob) (obsstore-off !)
$ hg status --change . $ hg status --change .
M a M a
R .hgsub R .hgsub
R .hgsubstate R .hgsubstate
broken repositories will refuse to push
#if obsstore-off
$ hg up -q -C 2
#else
$ hg up -q -C 6
#endif
$ echo c >> t/b
$ hg amend -q -R t
$ hg init ../dest
$ hg init ../dest/t
$ hg init ../dest/s
$ hg push -q ../dest
abort: subrepo 't' is hidden in revision 04aa62396ec6 (obsstore-on !)
abort: subrepo 't' not found in revision 04aa62396ec6 (obsstore-off !)
[255]
... unless forced
$ hg push --force -q ../dest
$ hg verify -R ../dest
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
checked 5 changesets with 12 changes to 4 files
checking subrepo links
subrepo 't' not found in revision 04aa62396ec6
subrepo 't' not found in revision 6bce99600681
$ cd .. $ cd ..