This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4386

manifest: proxy to revlog instance instead of inheriting
ClosedPublic

Authored by indygreg on Aug 27 2018, 12:00 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG7f5e6d3e9032: manifest: proxy to revlog instance instead of inheriting
Summary

Previously, manifestrevlog inherited revlog.revlog and therefore
exposed all its APIs. This inevitably resulted in consumers calling
low-level revlog APIs.

As part of abstracting storage, we want to formalize the interface
for manifest storage. The revlog API is much too large to define as
the interface.

Like we did for filelog, this commit divorces the manifest class
from revlog so that we can standardize on a smaller API surface.

The way I went about this commit was I broke the inheritance, ran
tests, and added proxies until all tests passed. Like filelog, there
are a handful of attributes that don't belong on the interface.
And like filelog, we'll tease these out in the future.

As part of this, we formalize an interface for manifest storage and
add checks that manifestrevlog conforms to the interface.

Adding proxies will introduce some overhead due to extra attribute
lookups and function calls. On the mozilla-unified repository:

$ hg verify
before: real 627.220 secs (user 525.870+0.000 sys 18.800+0.000)
after: real 628.930 secs (user 532.050+0.000 sys 18.320+0.000)

$ hg serve (for a clone)
before: user 223.580+0.000 sys 14.270+0.000
after: user 227.720+0.000 sys 13.920+0.000

$ hg clone
before: user 506.390+0.000 sys 29.720+0.000
after: user 513.080+0.000 sys 28.280+0.000

There appears to be some overhead here. But it appears to be 1-2%.
I think that is an appropriate price to pay for storage abstraction,
which will eventually let us have much nicer things. If the overhead
is noticed in other operations (whose CPU time isn't likely dwarfed by
fulltext resolution) or if we want to cut down on the overhead, we
could dynamically build up a type whose methods are effectively
aliased to a revlog instance's. I'm inclined to punt on that problem
for now. We may have to do it for the changelog. At which point it
could be implemented in a generic way and ported to filelog and
manifestrevlog easily enough I would think.

.. api:: manifest.manifestrevlog no longer inherits from revlog

The manifestrevlog class now wraps a revlog instance instead of
inheriting from revlog. Various attributes and methods on instances
are no longer available.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

indygreg created this revision.Aug 27 2018, 12:00 PM
martinvonz added a subscriber: martinvonz.Aug 27 2018, 12:21 PM

Adding proxies can be costly, but maybe performance only matters for the changelog? I can imagine hg verify being the command most affected by this change. Does that get even measurably slower? Of course, it's not very important that hg verify is fast; the point is that if there's no measurable impact on that command, then I can't think of anything else where it would be measurable.

indygreg planned changes to this revision.Aug 27 2018, 12:36 PM
In D4386#67436, @martinvonz wrote:

Adding proxies can be costly, but maybe performance only matters for the changelog? I can imagine hg verify being the command most affected by this change. Does that get even measurably slower? Of course, it's not very important that hg verify is fast; the point is that if there's no measurable impact on that command, then I can't think of anything else where it would be measurable.

Yes, this will add overhead. Whether that overhead is significant for any operations we care about is up in the air. I've been testing CPU utilization for hg clone on both server and client for the mozilla-unified repo. That's a good proxy for hg bundle and hg unbundle performance. I can add hg verify to my tests as well.

Honestly, for manifests I expect that I/O, decompression, and patch application to dwarf the overhead of an additional Python function call. There might be some tight loops calling into say rev() or node() that will be impacted. But many of those are later followed by something that does revision resolving, delta generation, etc and thus the regression is not noticed.

Anyway, I'll revise this commit message to include some performance numbers.

martinvonz added a comment.Aug 27 2018, 12:52 PM
In D4386#67438, @indygreg wrote:
In D4386#67436, @martinvonz wrote:

Adding proxies can be costly, but maybe performance only matters for the changelog? I can imagine hg verify being the command most affected by this change. Does that get even measurably slower? Of course, it's not very important that hg verify is fast; the point is that if there's no measurable impact on that command, then I can't think of anything else where it would be measurable.

Yes, this will add overhead. Whether that overhead is significant for any operations we care about is up in the air. I've been testing CPU utilization for hg clone on both server and client for the mozilla-unified repo. That's a good proxy for hg bundle and hg unbundle performance. I can add hg verify to my tests as well.
Honestly, for manifests I expect that I/O, decompression, and patch application to dwarf the overhead of an additional Python function call. There might be some tight loops calling into say rev() or node() that will be impacted. But many of those are later followed by something that does revision resolving, delta generation, etc and thus the regression is not noticed.

I agree. That's why I said hg verify is the only command I can imagine where the cost can possibly measurable.

Anyway, I'll revise this commit message to include some performance numbers.

Thanks.

indygreg edited the summary of this revision. (Show Details)Aug 27 2018, 1:21 PM
indygreg requested review of this revision.Aug 27 2018, 1:37 PM

I guess hg phabsend doesn't reset the Changed Planned state when a summary-only update occurs.

Closed by commit rHG7f5e6d3e9032: manifest: proxy to revlog instance instead of inheriting (authored by indygreg). · Explain WhyAug 29 2018, 1:23 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 10627

mercurial/manifest.py

def clear(self, clear_persisted_data=False): def clear(self, clear_persisted_data=False):
super(manifestfulltextcache, self).clear() super(manifestfulltextcache, self).clear()
if clear_persisted_data: if clear_persisted_data:
self._dirty = True self._dirty = True
self.write() self.write()
self._read = False self._read = False
class manifestrevlog(revlog.revlog): @interfaceutil.implementer(repository.imanifeststorage)
class manifestrevlog(object):
'''A revlog that stores manifest texts. This is responsible for caching the '''A revlog that stores manifest texts. This is responsible for caching the
full-text manifest contents. full-text manifest contents.
''' '''
def __init__(self, opener, tree='', dirlogcache=None, indexfile=None, def __init__(self, opener, tree='', dirlogcache=None, indexfile=None,
treemanifest=False): treemanifest=False):
"""Constructs a new manifest revlog """Constructs a new manifest revlog
`indexfile` - used by extensions to have two manifests at once, like `indexfile` - used by extensions to have two manifests at once, like
self._tree = tree self._tree = tree
# The dirlogcache is kept on the root manifest log # The dirlogcache is kept on the root manifest log
if tree: if tree:
self._dirlogcache = dirlogcache self._dirlogcache = dirlogcache
else: else:
self._dirlogcache = {'': self} self._dirlogcache = {'': self}
super(manifestrevlog, self).__init__(opener, indexfile, self._revlog = revlog.revlog(opener, indexfile,
# only root indexfile is cached # only root indexfile is cached
checkambig=not bool(tree), checkambig=not bool(tree),
mmaplargeindex=True) mmaplargeindex=True)
self.index = self._revlog.index
self.version = self._revlog.version
self._generaldelta = self._revlog._generaldelta
def _setupmanifestcachehooks(self, repo): def _setupmanifestcachehooks(self, repo):
"""Persist the manifestfulltextcache on lock release""" """Persist the manifestfulltextcache on lock release"""
if not util.safehasattr(repo, '_lockref'): if not util.safehasattr(repo, '_lockref'):
return return
self._fulltextcache._opener = repo.cachevfs self._fulltextcache._opener = repo.cachevfs
reporef = weakref.ref(repo) reporef = weakref.ref(repo)
manifestrevlogref = weakref.ref(self) manifestrevlogref = weakref.ref(self)
if repo._currentlock(repo._lockref) is not None: if repo._currentlock(repo._lockref) is not None:
repo._afterlock(persistmanifestcache) repo._afterlock(persistmanifestcache)
@property @property
def fulltextcache(self): def fulltextcache(self):
return self._fulltextcache return self._fulltextcache
def clearcaches(self, clear_persisted_data=False): def clearcaches(self, clear_persisted_data=False):
super(manifestrevlog, self).clearcaches() self._revlog.clearcaches()
self._fulltextcache.clear(clear_persisted_data=clear_persisted_data) self._fulltextcache.clear(clear_persisted_data=clear_persisted_data)
self._dirlogcache = {self._tree: self} self._dirlogcache = {self._tree: self}
def dirlog(self, d): def dirlog(self, d):
if d: if d:
assert self._treeondisk assert self._treeondisk
if d not in self._dirlogcache: if d not in self._dirlogcache:
mfrevlog = manifestrevlog(self.opener, d, mfrevlog = manifestrevlog(self.opener, d,
# revlog layer. # revlog layer.
_checkforbidden(added) _checkforbidden(added)
# combine the changed lists into one sorted iterator # combine the changed lists into one sorted iterator
work = heapq.merge([(x, False) for x in added], work = heapq.merge([(x, False) for x in added],
[(x, True) for x in removed]) [(x, True) for x in removed])
arraytext, deltatext = m.fastdelta(self.fulltextcache[p1], work) arraytext, deltatext = m.fastdelta(self.fulltextcache[p1], work)
cachedelta = self.rev(p1), deltatext cachedelta = self._revlog.rev(p1), deltatext
text = util.buffer(arraytext) text = util.buffer(arraytext)
n = self.addrevision(text, transaction, link, p1, p2, cachedelta) n = self._revlog.addrevision(text, transaction, link, p1, p2,
cachedelta)
else: else:
# The first parent manifest isn't already loaded, so we'll # The first parent manifest isn't already loaded, so we'll
# just encode a fulltext of the manifest and pass that # just encode a fulltext of the manifest and pass that
# through to the revlog layer, and let it handle the delta # through to the revlog layer, and let it handle the delta
# process. # process.
if self._treeondisk: if self._treeondisk:
assert readtree, "readtree must be set for treemanifest writes" assert readtree, "readtree must be set for treemanifest writes"
m1 = readtree(self._tree, p1) m1 = readtree(self._tree, p1)
m2 = readtree(self._tree, p2) m2 = readtree(self._tree, p2)
n = self._addtree(m, transaction, link, m1, m2, readtree) n = self._addtree(m, transaction, link, m1, m2, readtree)
arraytext = None arraytext = None
else: else:
text = m.text() text = m.text()
n = self.addrevision(text, transaction, link, p1, p2) n = self._revlog.addrevision(text, transaction, link, p1, p2)
arraytext = bytearray(text) arraytext = bytearray(text)
if arraytext is not None: if arraytext is not None:
self.fulltextcache[n] = arraytext self.fulltextcache[n] = arraytext
return n return n
def _addtree(self, m, transaction, link, m1, m2, readtree): def _addtree(self, m, transaction, link, m1, m2, readtree):
if self._tree != '': if self._tree != '':
# Double-check whether contents are unchanged to one parent # Double-check whether contents are unchanged to one parent
if text == m1.dirtext(): if text == m1.dirtext():
n = m1.node() n = m1.node()
elif text == m2.dirtext(): elif text == m2.dirtext():
n = m2.node() n = m2.node()
if not n: if not n:
n = self.addrevision(text, transaction, link, m1.node(), m2.node()) n = self._revlog.addrevision(text, transaction, link, m1.node(),
m2.node())
# Save nodeid so parent manifest can calculate its nodeid # Save nodeid so parent manifest can calculate its nodeid
m.setnode(n) m.setnode(n)
return n return n
def __len__(self):
return len(self._revlog)
def __iter__(self):
return self._revlog.__iter__()
def rev(self, node):
return self._revlog.rev(node)
def node(self, rev):
return self._revlog.node(rev)
def lookup(self, value):
return self._revlog.lookup(value)
def parentrevs(self, rev):
return self._revlog.parentrevs(rev)
def parents(self, node):
return self._revlog.parents(node)
def linkrev(self, rev):
return self._revlog.linkrev(rev)
def checksize(self):
return self._revlog.checksize()
def revision(self, node, _df=None, raw=False):
return self._revlog.revision(node, _df=_df, raw=raw)
def revdiff(self, rev1, rev2):
return self._revlog.revdiff(rev1, rev2)
def cmp(self, node, text):
return self._revlog.cmp(node, text)
def deltaparent(self, rev):
return self._revlog.deltaparent(rev)
def emitrevisiondeltas(self, requests):
return self._revlog.emitrevisiondeltas(requests)
def addgroup(self, deltas, linkmapper, transaction, addrevisioncb=None):
return self._revlog.addgroup(deltas, linkmapper, transaction,
addrevisioncb=addrevisioncb)
def getstrippoint(self, minlink):
return self._revlog.getstrippoint(minlink)
def strip(self, minlink, transaction):
return self._revlog.strip(minlink, transaction)
def files(self):
return self._revlog.files()
def clone(self, tr, destrevlog, **kwargs):
if not isinstance(destrevlog, manifestrevlog):
raise error.ProgrammingError('expected manifestrevlog to clone()')
return self._revlog.clone(tr, destrevlog._revlog, **kwargs)
@property
def indexfile(self):
return self._revlog.indexfile
@indexfile.setter
def indexfile(self, value):
self._revlog.indexfile = value
@property
def opener(self):
return self._revlog.opener
@opener.setter
def opener(self, value):
self._revlog.opener = value
@interfaceutil.implementer(repository.imanifestlog) @interfaceutil.implementer(repository.imanifestlog)
class manifestlog(object): class manifestlog(object):
"""A collection class representing the collection of manifest snapshots """A collection class representing the collection of manifest snapshots
referenced by commits in the repository. referenced by commits in the repository.
In this situation, 'manifest' refers to the abstract concept of a snapshot In this situation, 'manifest' refers to the abstract concept of a snapshot
of the list of files in the given commit. Consumers of the output of this of the list of files in the given commit. Consumers of the output of this
class do not care about the implementation details of the actual manifests class do not care about the implementation details of the actual manifests

mercurial/repository.py

Takes a transaction object, the changeset revision number it will Takes a transaction object, the changeset revision number it will
be associated with, its parent nodes, and lists of added and be associated with, its parent nodes, and lists of added and
removed paths. removed paths.
Returns the binary node of the created revision. Returns the binary node of the created revision.
""" """
class imanifeststorage(interfaceutil.Interface):
"""Storage interface for manifest data."""
index = interfaceutil.Attribute(
"""An ``ifilerevisionssequence`` instance.""")
indexfile = interfaceutil.Attribute(
"""Path of revlog index file.
TODO this is revlog specific and should not be exposed.
""")
opener = interfaceutil.Attribute(
"""VFS opener to use to access underlying files used for storage.
TODO this is revlog specific and should not be exposed.
""")
version = interfaceutil.Attribute(
"""Revlog version number.
TODO this is revlog specific and should not be exposed.
""")
_generaldelta = interfaceutil.Attribute(
"""Whether generaldelta storage is being used.
TODO this is revlog specific and should not be exposed.
""")
fulltextcache = interfaceutil.Attribute(
"""Dict with cache of fulltexts.
TODO this doesn't feel appropriate for the storage interface.
""")
def __len__():
"""Obtain the number of revisions stored for this manifest."""
def __iter__():
"""Iterate over revision numbers for this manifest."""
def rev(node):
"""Obtain the revision number given a binary node.
Raises ``error.LookupError`` if the node is not known.
"""
def node(rev):
"""Obtain the node value given a revision number.
Raises ``error.LookupError`` if the revision is not known.
"""
def lookup(value):
"""Attempt to resolve a value to a node.
Value can be a binary node, hex node, revision number, or a bytes
that can be converted to an integer.
Raises ``error.LookupError`` if a ndoe could not be resolved.
TODO this is only used by debug* commands and can probably be deleted
easily.
"""
def parents(node):
"""Returns a 2-tuple of parent nodes for a node.
Values will be ``nullid`` if the parent is empty.
"""
def parentrevs(rev):
"""Like parents() but operates on revision numbers."""
def linkrev(rev):
"""Obtain the changeset revision number a revision is linked to."""
def revision(node, _df=None, raw=False):
"""Obtain fulltext data for a node."""
def revdiff(rev1, rev2):
"""Obtain a delta between two revision numbers.
The returned data is the result of ``bdiff.bdiff()`` on the raw
revision data.
"""
def cmp(node, fulltext):
"""Compare fulltext to another revision.
Returns True if the fulltext is different from what is stored.
"""
def emitrevisiondeltas(requests):
"""Produce ``irevisiondelta`` from ``irevisiondeltarequest``s.
See the documentation for ``ifiledata`` for more.
"""
def addgroup(deltas, linkmapper, transaction, addrevisioncb=None):
"""Process a series of deltas for storage.
See the documentation in ``ifilemutation`` for more.
"""
def getstrippoint(minlink):
"""Find minimum revision that must be stripped to strip a linkrev.
See the documentation in ``ifilemutation`` for more.
"""
def strip(minlink, transaction):
"""Remove storage of items starting at a linkrev.
See the documentation in ``ifilemutation`` for more.
"""
def checksize():
"""Obtain the expected sizes of backing files.
TODO this is used by verify and it should not be part of the interface.
"""
def files():
"""Obtain paths that are backing storage for this manifest.
TODO this is used by verify and there should probably be a better API
for this functionality.
"""
def deltaparent(rev):
"""Obtain the revision that a revision is delta'd against.
TODO delta encoding is an implementation detail of storage and should
not be exposed to the storage interface.
"""
def clone(tr, dest, **kwargs):
"""Clone this instance to another."""
def clearcaches(clear_persisted_data=False):
"""Clear any caches associated with this instance."""
def dirlog(d):
"""Obtain a manifest storage instance for a tree."""
def add(m, transaction, link, p1, p2, added, removed, readtree=None):
"""Add a revision to storage.
``m`` is an object conforming to ``imanifestdict``.
``link`` is the linkrev revision number.
``p1`` and ``p2`` are the parent revision numbers.
``added`` and ``removed`` are iterables of added and removed paths,
respectively.
"""
class imanifestlog(interfaceutil.Interface): class imanifestlog(interfaceutil.Interface):
"""Interface representing a collection of manifest snapshots. """Interface representing a collection of manifest snapshots.
Represents the root manifest in a repository. Represents the root manifest in a repository.
Also serves as a means to access nested tree manifests and to cache Also serves as a means to access nested tree manifests and to cache
tree manifests. tree manifests.
""" """

tests/test-check-interfaces.py

manifest.manifestctx) manifest.manifestctx)
ziverify.verifyClass(repository.imanifestrevisionwritable, ziverify.verifyClass(repository.imanifestrevisionwritable,
manifest.memmanifestctx) manifest.memmanifestctx)
ziverify.verifyClass(repository.imanifestrevisionstored, ziverify.verifyClass(repository.imanifestrevisionstored,
manifest.treemanifestctx) manifest.treemanifestctx)
ziverify.verifyClass(repository.imanifestrevisionwritable, ziverify.verifyClass(repository.imanifestrevisionwritable,
manifest.memtreemanifestctx) manifest.memtreemanifestctx)
ziverify.verifyClass(repository.imanifestlog, manifest.manifestlog) ziverify.verifyClass(repository.imanifestlog, manifest.manifestlog)
ziverify.verifyClass(repository.imanifeststorage, manifest.manifestrevlog)
vfs = vfsmod.vfs(b'.') vfs = vfsmod.vfs(b'.')
fl = filelog.filelog(vfs, b'dummy.i') fl = filelog.filelog(vfs, b'dummy.i')
checkzobject(fl, allowextra=True) checkzobject(fl, allowextra=True)
# Conforms to imanifestlog. # Conforms to imanifestlog.
ml = manifest.manifestlog(vfs, repo) ml = manifest.manifestlog(vfs, repo)
checkzobject(ml) checkzobject(ml)
checkzobject(repo.manifestlog) checkzobject(repo.manifestlog)
# Conforms to imanifestrevision. # Conforms to imanifestrevision.
mctx = ml[repo[0].manifestnode()] mctx = ml[repo[0].manifestnode()]
checkzobject(mctx) checkzobject(mctx)
# Conforms to imanifestrevisionwritable. # Conforms to imanifestrevisionwritable.
checkzobject(mctx.new()) checkzobject(mctx.new())
checkzobject(mctx.copy()) checkzobject(mctx.copy())
# Conforms to imanifestdict. # Conforms to imanifestdict.
checkzobject(mctx.read()) checkzobject(mctx.read())
mrl = manifest.manifestrevlog(vfs)
checkzobject(mrl)
ziverify.verifyClass(repository.irevisiondelta, ziverify.verifyClass(repository.irevisiondelta,
revlog.revlogrevisiondelta) revlog.revlogrevisiondelta)
ziverify.verifyClass(repository.irevisiondeltarequest, ziverify.verifyClass(repository.irevisiondeltarequest,
changegroup.revisiondeltarequest) changegroup.revisiondeltarequest)
rd = revlog.revlogrevisiondelta( rd = revlog.revlogrevisiondelta(
node=b'', node=b'',
p1node=b'', p1node=b'',