This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4099

narrow: move .hg/narrowspec to .hg/store/narrowspec (BC)
ClosedPublic

Authored by martinvonz on Aug 3 2018, 5:24 PM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHG576eef1ab43d: narrow: move .hg/narrowspec to .hg/store/narrowspec (BC)
Summary

The narrowspec is more closely related to the store than to the
working copy. For example, if the narrowspec changes, the set of
revlogs also needs to change (the working copy may change, but that
depends on which commit is checked out). Also, when using the share
extension, the narrowspec needs to be shared along with the
store. This patch therefore moves the narrowspec into the store/
directory.

This is clearly a breaking change, but I haven't bothered trying to
fall back to reading the narrowspec from the old location (.hg/),
because there are very few users of narrow out there. (We'll add a
temporary hack to our Google-internal extension to handle the
migration.)

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

martinvonz created this revision.Aug 3 2018, 5:24 PM
martinvonz updated this revision to Diff 9874.Aug 3 2018, 8:12 PM
yuja added a subscriber: yuja.Aug 5 2018, 9:04 AM

Queued, thanks.

def savebackup(repo, backupname):

if repository.NARROW_REQUIREMENT not in repo.requirements:
    return
vfs = repo.vfs
vfs.tryunlink(backupname)
  • util.copyfile(vfs.join(FILENAME), vfs.join(backupname), hardlink=True)

+ util.copyfile(repo.svfs.join(FILENAME), vfs.join(backupname), hardlink=True)

Maybe move backup file to svfs as well?

Closed by commit rHG576eef1ab43d: narrow: move .hg/narrowspec to .hg/store/narrowspec (BC) (authored by martinvonz). · Explain WhyAug 5 2018, 9:04 AM
This revision was automatically updated to reflect the committed changes.
durin42 added a comment.Aug 7 2018, 10:23 AM

There are non-Google users of narrow at this point (recall we've gotten non-Googler patches), so please publish the temporary migration hack someplace for the benefit of the other users.

(A phabricator review that we immediately abandon would probably be fine, but we should definitely make a note in the relnotes for 4.8)

pulkit added a subscriber: pulkit.Aug 7 2018, 10:29 AM
In D4099#64266, @durin42 wrote:

There are non-Google users of narrow at this point (recall we've gotten non-Googler patches), so please publish the temporary migration hack someplace for the benefit of the other users.
(A phabricator review that we immediately abandon would probably be fine, but we should definitely make a note in the relnotes for 4.8)

Yes, we might roll-out narrow with mercurial 4.7 first, so this script will be of great help. (BTW, isn't it just a move of file?)

durin42 added a comment.Aug 7 2018, 10:32 AM
In D4099#64268, @pulkit wrote:
In D4099#64266, @durin42 wrote:

There are non-Google users of narrow at this point (recall we've gotten non-Googler patches), so please publish the temporary migration hack someplace for the benefit of the other users.
(A phabricator review that we immediately abandon would probably be fine, but we should definitely make a note in the relnotes for 4.8)

Yes, we might roll-out narrow with mercurial 4.7 first, so this script will be of great help. (BTW, isn't it just a move of file?)

It's "just" a move, but typically the way we handle these migrations at Google is we write an extension that does one of the following:

  1. Hijacks reads for the new file and reads the old file
  2. Hijacks reads for the new file if it doesn't exist and moves the new file

we run in mode 1 when we're doing the rollout and might roll back, then switch to mode 2 when we're past the rubicon on the rollout and can finalize the data migration. Once everyone has run the extension in mode 2 for a while, we reap the extension.

pulkit added a comment.Aug 7 2018, 10:35 AM
In D4099#64270, @durin42 wrote:
In D4099#64268, @pulkit wrote:
In D4099#64266, @durin42 wrote:

There are non-Google users of narrow at this point (recall we've gotten non-Googler patches), so please publish the temporary migration hack someplace for the benefit of the other users.
(A phabricator review that we immediately abandon would probably be fine, but we should definitely make a note in the relnotes for 4.8)

Yes, we might roll-out narrow with mercurial 4.7 first, so this script will be of great help. (BTW, isn't it just a move of file?)

It's "just" a move, but typically the way we handle these migrations at Google is we write an extension that does one of the following:

  1. Hijacks reads for the new file and reads the old file
  2. Hijacks reads for the new file if it doesn't exist and moves the new file

we run in mode 1 when we're doing the rollout and might roll back, then switch to mode 2 when we're past the rubicon on the rollout and can finalize the data migration. Once everyone has run the extension in mode 2 for a while, we reap the extension.

Oh, I missed that clients need to be updated in real time. It will really be great if this extension is shared. Thanks @durin42 for the explanation.

Revision Contents
Changeset List

Diff 9934

hgext/narrow/__init__.py

# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for # Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
# extensions which SHIP WITH MERCURIAL. Non-mainline extensions should # extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
# be specifying the version(s) of Mercurial they are tested with, or # be specifying the version(s) of Mercurial they are tested with, or
# leave the attribute unspecified. # leave the attribute unspecified.
testedwith = 'ships-with-hg-core' testedwith = 'ships-with-hg-core'
from mercurial import ( from mercurial import (
extensions, extensions,
hg,
localrepo, localrepo,
registrar, registrar,
repository, repository,
verify as verifymod, verify as verifymod,
) )
from . import ( from . import (
narrowbundle2, narrowbundle2,
# The verifier's matcher argument was desgined for narrowhg, so it should # The verifier's matcher argument was desgined for narrowhg, so it should
# be None from core. If another extension passes a matcher (unlikely), # be None from core. If another extension passes a matcher (unlikely),
# we'll have to fail until matchers can be composed more easily. # we'll have to fail until matchers can be composed more easily.
assert matcher is None assert matcher is None
orig(self, repo, repo.narrowmatch()) orig(self, repo, repo.narrowmatch())
def extsetup(ui): def extsetup(ui):
extensions.wrapfunction(verifymod.verifier, '__init__', _verifierinit) extensions.wrapfunction(verifymod.verifier, '__init__', _verifierinit)
extensions.wrapfunction(hg, 'postshare', narrowrepo.wrappostshare)
extensions.wrapfunction(hg, 'copystore', narrowrepo.unsharenarrowspec)
templatekeyword = narrowtemplates.templatekeyword templatekeyword = narrowtemplates.templatekeyword
revsetpredicate = narrowtemplates.revsetpredicate revsetpredicate = narrowtemplates.revsetpredicate

hgext/narrow/narrowrepo.py

# narrowrepo.py - repository which supports narrow revlogs, lazy loading # narrowrepo.py - repository which supports narrow revlogs, lazy loading
# #
# Copyright 2017 Google, Inc. # Copyright 2017 Google, Inc.
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
from __future__ import absolute_import from __future__ import absolute_import
from mercurial import (
hg,
narrowspec,
repository,
)
from . import ( from . import (
narrowdirstate, narrowdirstate,
narrowrevlog, narrowrevlog,
) )
def wrappostshare(orig, sourcerepo, destrepo, **kwargs):
orig(sourcerepo, destrepo, **kwargs)
if repository.NARROW_REQUIREMENT in sourcerepo.requirements:
with destrepo.wlock():
with destrepo.vfs('shared', 'a') as fp:
fp.write(narrowspec.FILENAME + '\n')
def unsharenarrowspec(orig, ui, repo, repopath):
if (repository.NARROW_REQUIREMENT in repo.requirements
and repo.path == repopath and repo.shared()):
srcrepo = hg.sharedreposource(repo)
with srcrepo.vfs(narrowspec.FILENAME) as f:
spec = f.read()
with repo.vfs(narrowspec.FILENAME, 'w') as f:
f.write(spec)
return orig(ui, repo, repopath)
def wraprepo(repo): def wraprepo(repo):
"""Enables narrow clone functionality on a single local repository.""" """Enables narrow clone functionality on a single local repository."""
class narrowrepository(repo.__class__): class narrowrepository(repo.__class__):
def file(self, f): def file(self, f):
fl = super(narrowrepository, self).file(f) fl = super(narrowrepository, self).file(f)
narrowrevlog.makenarrowfilelog(fl, self.narrowmatch()) narrowrevlog.makenarrowfilelog(fl, self.narrowmatch())
return fl return fl
def _makedirstate(self): def _makedirstate(self):
dirstate = super(narrowrepository, self)._makedirstate() dirstate = super(narrowrepository, self)._makedirstate()
return narrowdirstate.wrapdirstate(self, dirstate) return narrowdirstate.wrapdirstate(self, dirstate)
repo.__class__ = narrowrepository repo.__class__ = narrowrepository

mercurial/localrepo.py

return node return node
except error.LookupError: except error.LookupError:
if not self._dirstatevalidatewarned: if not self._dirstatevalidatewarned:
self._dirstatevalidatewarned = True self._dirstatevalidatewarned = True
self.ui.warn(_("warning: ignoring unknown" self.ui.warn(_("warning: ignoring unknown"
" working parent %s!\n") % short(node)) " working parent %s!\n") % short(node))
return nullid return nullid
@repofilecache(narrowspec.FILENAME) @storecache(narrowspec.FILENAME)
def narrowpats(self): def narrowpats(self):
"""matcher patterns for this repository's narrowspec """matcher patterns for this repository's narrowspec
A tuple of (includes, excludes). A tuple of (includes, excludes).
""" """
source = self source = self
if self.shared(): if self.shared():
from . import hg from . import hg
source = hg.sharedreposource(self) source = hg.sharedreposource(self)
return narrowspec.load(source) return narrowspec.load(source)
@repofilecache(narrowspec.FILENAME) @storecache(narrowspec.FILENAME)
def _narrowmatch(self): def _narrowmatch(self):
if repository.NARROW_REQUIREMENT not in self.requirements: if repository.NARROW_REQUIREMENT not in self.requirements:
return matchmod.always(self.root, '') return matchmod.always(self.root, '')
include, exclude = self.narrowpats include, exclude = self.narrowpats
return narrowspec.match(self.root, include=include, exclude=exclude) return narrowspec.match(self.root, include=include, exclude=exclude)
# TODO(martinvonz): make this property-like instead? # TODO(martinvonz): make this property-like instead?
def narrowmatch(self): def narrowmatch(self):

mercurial/narrowspec.py

return matchmod.match(root, '', [], include=include or [], return matchmod.match(root, '', [], include=include or [],
exclude=exclude or []) exclude=exclude or [])
def needsexpansion(includes): def needsexpansion(includes):
return [i for i in includes if i.startswith('include:')] return [i for i in includes if i.startswith('include:')]
def load(repo): def load(repo):
try: try:
spec = repo.vfs.read(FILENAME) spec = repo.svfs.read(FILENAME)
except IOError as e: except IOError as e:
# Treat "narrowspec does not exist" the same as "narrowspec file exists # Treat "narrowspec does not exist" the same as "narrowspec file exists
# and is empty". # and is empty".
if e.errno == errno.ENOENT: if e.errno == errno.ENOENT:
return set(), set() return set(), set()
raise raise
# maybe we should care about the profiles returned too # maybe we should care about the profiles returned too
includepats, excludepats, profiles = sparse.parseconfig(repo.ui, spec, includepats, excludepats, profiles = sparse.parseconfig(repo.ui, spec,
'narrow') 'narrow')
if profiles: if profiles:
raise error.Abort(_("including other spec files using '%include' is not" raise error.Abort(_("including other spec files using '%include' is not"
" suported in narrowspec")) " suported in narrowspec"))
return includepats, excludepats return includepats, excludepats
def save(repo, includepats, excludepats): def save(repo, includepats, excludepats):
spec = format(includepats, excludepats) spec = format(includepats, excludepats)
repo.vfs.write(FILENAME, spec) repo.svfs.write(FILENAME, spec)
def savebackup(repo, backupname): def savebackup(repo, backupname):
if repository.NARROW_REQUIREMENT not in repo.requirements: if repository.NARROW_REQUIREMENT not in repo.requirements:
return return
vfs = repo.vfs vfs = repo.vfs
vfs.tryunlink(backupname) vfs.tryunlink(backupname)
util.copyfile(vfs.join(FILENAME), vfs.join(backupname), hardlink=True) util.copyfile(repo.svfs.join(FILENAME), vfs.join(backupname), hardlink=True)
def restorebackup(repo, backupname): def restorebackup(repo, backupname):
if repository.NARROW_REQUIREMENT not in repo.requirements: if repository.NARROW_REQUIREMENT not in repo.requirements:
return return
repo.vfs.rename(backupname, FILENAME) util.rename(repo.vfs.join(backupname), repo.svfs.join(FILENAME))
def clearbackup(repo, backupname): def clearbackup(repo, backupname):
if repository.NARROW_REQUIREMENT not in repo.requirements: if repository.NARROW_REQUIREMENT not in repo.requirements:
return return
repo.vfs.unlink(backupname) repo.vfs.unlink(backupname)
def restrictpatterns(req_includes, req_excludes, repo_includes, repo_excludes): def restrictpatterns(req_includes, req_excludes, repo_includes, repo_excludes):
r""" Restricts the patterns according to repo settings, r""" Restricts the patterns according to repo settings,

mercurial/store.py

mode = vfs.stat().st_mode mode = vfs.stat().st_mode
# avoid some useless chmods # avoid some useless chmods
if (0o777 & ~util.umask) == (0o777 & mode): if (0o777 & ~util.umask) == (0o777 & mode):
mode = None mode = None
except OSError: except OSError:
mode = None mode = None
return mode return mode
_data = ('data meta 00manifest.d 00manifest.i 00changelog.d 00changelog.i' _data = ('narrowspec data meta 00manifest.d 00manifest.i'
' phaseroots obsstore') ' 00changelog.d 00changelog.i phaseroots obsstore')
def isrevlog(f, kind, st): def isrevlog(f, kind, st):
return kind == stat.S_IFREG and f[-2:] in ('.i', '.d') return kind == stat.S_IFREG and f[-2:] in ('.i', '.d')
class basicstore(object): class basicstore(object):
'''base class for local repository stores''' '''base class for local repository stores'''
def __init__(self, path, vfstype): def __init__(self, path, vfstype):
vfs = vfstype(path) vfs = vfstype(path)
ef = self.encode(f) ef = self.encode(f)
try: try:
yield f, ef, self.getsize(ef) yield f, ef, self.getsize(ef)
except OSError as err: except OSError as err:
if err.errno != errno.ENOENT: if err.errno != errno.ENOENT:
raise raise
def copylist(self): def copylist(self):
d = ('data meta dh fncache phaseroots obsstore' d = ('narrowspec data meta dh fncache phaseroots obsstore'
' 00manifest.d 00manifest.i 00changelog.d 00changelog.i') ' 00manifest.d 00manifest.i 00changelog.d 00changelog.i')
return (['requires', '00changelog.i'] + return (['requires', '00changelog.i'] +
['store/' + f for f in d.split()]) ['store/' + f for f in d.split()])
def write(self, tr): def write(self, tr):
self.fncache.write(tr) self.fncache.write(tr)
def invalidatecaches(self): def invalidatecaches(self):

tests/test-narrow-debugcommands.t

$ . "$TESTDIR/narrow-library.sh" $ . "$TESTDIR/narrow-library.sh"
$ hg init repo $ hg init repo
$ cd repo $ cd repo
$ cat << EOF > .hg/narrowspec $ cat << EOF > .hg/store/narrowspec
> [include] > [include]
> path:foo > path:foo
> [exclude] > [exclude]
> EOF > EOF
$ echo treemanifest >> .hg/requires $ echo treemanifest >> .hg/requires
$ echo narrowhg-experimental >> .hg/requires $ echo narrowhg-experimental >> .hg/requires
$ mkdir -p foo/bar $ mkdir -p foo/bar
$ echo b > foo/f $ echo b > foo/f

tests/test-narrow-pull.t

$ hg share narrow2 narrow2-share $ hg share narrow2 narrow2-share
updating working directory updating working directory
1 files updated, 0 files merged, 0 files removed, 0 files unresolved 1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd narrow2-share $ cd narrow2-share
$ hg status $ hg status
We should also be able to unshare without breaking everything: We should also be able to unshare without breaking everything:
$ hg unshare $ hg unshare
devel-warn: write with no wlock: "narrowspec" at: */hgext/narrow/narrowrepo.py:* (unsharenarrowspec) (glob)
$ hg verify $ hg verify
checking changesets checking changesets
checking manifests checking manifests
crosschecking files in changesets and manifests crosschecking files in changesets and manifests
checking files checking files
1 files, 1 changesets, 1 total revisions 1 files, 1 changesets, 1 total revisions