This is an archive of the discontinued Mercurial Phabricator instance.

Differential D3154

filelog: wrap revlog instead of inheriting it (API)
ClosedPublic

Authored by indygreg on Apr 5 2018, 8:51 PM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHG1541e1a8e87d: filelog: wrap revlog instead of inheriting it (API)
Summary

The revlog base class exposes a ton of methods. Inheriting the
revlog class for filelog will make it difficult to expose a
clean interface. There will be abstraction violations.

This commit breaks the inheritance of revlog by the filelog
class. Filelog instances now contain a reference to a revlog
instance. Various properties and methods are now proxied to
that instance.

There is precedence for doing this: manifestlog does something
similar. Although, manifestlog has a cleaner interface than
filelog. We'll get there with filelog...

The new filelog class exposes a handful of extra properties and
methods that aren't part of the declared filelog interface.
Every extra item was added in order to get a test to pass. The
set of tests that failed without these extra proxies has
significant overlap with the set of tests that don't work with
the simple store repo. There should be no surprise there.

Hopefully the hardest part about this commit to review are the
changes to bundlerepo and unionrepo. Both repository types
define a custom revlog or revlog-like class and then have a
custom filelog that inherits from both filelog and their custom
revlog. This code has been changed so the filelog types don't
inherit from revlog. Instead, they replace the revlog instance
on the created filelog. This is super hacky. I plan to fix this
in a future commit by parameterizing filelog.init.

Because Python function call overhead is a thing, this change
could impact performance by introducing a nearly empty proxy
function for various methods and properties. I would gladly
measure the performance impact of it, but I'm not sure what
operations have tight loops over filelog attribute lookups
or function calls. I know some of the DAG traversal code can
be sensitive about the performance of e.g. parentrevs(). However,
many of these functions are implemented on the revlog class and
therefore have direct access to self.parentrevs() and aren't
going through a proxy.

.. api::

filelog.filelog is now a standalone class and doesn't inherit
from revlog. Instead, it wraps a revlog instance at self._revlog.
This change was made in an attempt to formalize storage APIs and
prevent revlog implementation details leaking through to callers.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Apr 5 2018, 8:51 PM
indygreg added a comment.Apr 5 2018, 8:55 PM

I want to draw attention to this patch for the potential performance concerns.

Like I said in the commit message, I'd love to know what the performance impact is. I'm just not sure which revlog/filelog attributes/methods are called in tight loops outside of revlog/filelog itself.

If the proxy methods are an issue, it /might/ be faster to use __new__ to hack up each instance so its self.__dict__[x] contained pointers to revlog functions bound as methods to the self._revlog instance. Super hacky. That's why I'd like evidence that performance is a real concern before implementing it.

durin42 accepted this revision.Apr 6 2018, 9:27 PM
This revision is now accepted and ready to land.Apr 6 2018, 9:27 PM
durin42 added a comment.Apr 6 2018, 9:48 PM
test-unionrepo.t:
+      t = self.revision(node)
+    File "/tmp/hgtests.wmoy8H/install/lib/python/mercurial/filelog.py", line 93, in revision
+      return self._revlog.revision(node, _df=_df, raw=raw)
+  TypeError: revision() got an unexpected keyword argument '_df'
+  [1]
durin42 requested changes to this revision.Apr 6 2018, 9:48 PM
This revision now requires changes to proceed.Apr 6 2018, 9:48 PM
indygreg added a comment.Apr 7 2018, 1:16 AM
In D3154#51010, @durin42 wrote:
test-unionrepo.t:
+      t = self.revision(node)
+    File "/tmp/hgtests.wmoy8H/install/lib/python/mercurial/filelog.py", line 93, in revision
+      return self._revlog.revision(node, _df=_df, raw=raw)
+  TypeError: revision() got an unexpected keyword argument '_df'
+  [1]

I remember this error and I thought I fixed it! But alas I can reproduce locally. I must have fixed it on a different class.

indygreg updated this revision to Diff 7860.Apr 7 2018, 1:40 AM
durin42 accepted this revision.Apr 9 2018, 6:30 PM
This revision is now accepted and ready to land.Apr 9 2018, 6:30 PM
Closed by commit rHG1541e1a8e87d: filelog: wrap revlog instead of inheriting it (API) (authored by indygreg). · Explain WhyApr 9 2018, 6:37 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/bundlerepo.py (6 lines)
Mmercurial/filelog.py (194 lines)
Mmercurial/unionrepo.py (10 lines)

Diff 7860

mercurial/bundlerepo.py

def dirlog(self, d): def dirlog(self, d):
if d in self._dirlogstarts: if d in self._dirlogstarts:
self.bundle.seek(self._dirlogstarts[d]) self.bundle.seek(self._dirlogstarts[d])
return bundlemanifest( return bundlemanifest(
self.opener, self.bundle, self._linkmapper, self.opener, self.bundle, self._linkmapper,
self._dirlogstarts, dir=d) self._dirlogstarts, dir=d)
return super(bundlemanifest, self).dirlog(d) return super(bundlemanifest, self).dirlog(d)
class bundlefilelog(bundlerevlog, filelog.filelog): class bundlefilelog(filelog.filelog):
def __init__(self, opener, path, cgunpacker, linkmapper): def __init__(self, opener, path, cgunpacker, linkmapper):
filelog.filelog.__init__(self, opener, path) filelog.filelog.__init__(self, opener, path)
bundlerevlog.__init__(self, opener, self.indexfile, cgunpacker, self._revlog = bundlerevlog(opener, self.indexfile,
linkmapper) cgunpacker, linkmapper)
def baserevision(self, nodeorrev): def baserevision(self, nodeorrev):
return filelog.filelog.revision(self, nodeorrev, raw=True) return filelog.filelog.revision(self, nodeorrev, raw=True)
class bundlepeer(localrepo.localpeer): class bundlepeer(localrepo.localpeer):
def canpush(self): def canpush(self):
return False return False

mercurial/filelog.py

# filelog.py - file history class for mercurial # filelog.py - file history class for mercurial
# #
# Copyright 2005-2007 Matt Mackall <mpm@selenic.com> # Copyright 2005-2007 Matt Mackall <mpm@selenic.com>
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
from __future__ import absolute_import from __future__ import absolute_import
from .thirdparty.zope import ( from .thirdparty.zope import (
interface as zi, interface as zi,
) )
from . import ( from . import (
error,
repository, repository,
revlog, revlog,
) )
@zi.implementer(repository.ifilestorage) @zi.implementer(repository.ifilestorage)
class filelog(revlog.revlog): class filelog(object):
def __init__(self, opener, path): def __init__(self, opener, path):
super(filelog, self).__init__(opener, self._revlog = revlog.revlog(opener,
"/".join(("data", path + ".i")), '/'.join(('data', path + '.i')),
censorable=True) censorable=True)
# full name of the user visible file, relative to the repository root # full name of the user visible file, relative to the repository root
self.filename = path self.filename = path
self.index = self._revlog.index
self.version = self._revlog.version
self.storedeltachains = self._revlog.storedeltachains
self._generaldelta = self._revlog._generaldelta
def __len__(self):
return len(self._revlog)
def __iter__(self):
return self._revlog.__iter__()
def revs(self, start=0, stop=None):
return self._revlog.revs(start=start, stop=stop)
def parents(self, node):
return self._revlog.parents(node)
def parentrevs(self, rev):
return self._revlog.parentrevs(rev)
def rev(self, node):
return self._revlog.rev(node)
def node(self, rev):
return self._revlog.node(rev)
def lookup(self, node):
return self._revlog.lookup(node)
def linkrev(self, rev):
return self._revlog.linkrev(rev)
def flags(self, rev):
return self._revlog.flags(rev)
def commonancestorsheads(self, node1, node2):
return self._revlog.commonancestorsheads(node1, node2)
def descendants(self, revs):
return self._revlog.descendants(revs)
def headrevs(self):
return self._revlog.headrevs()
def heads(self, start=None, stop=None):
return self._revlog.heads(start, stop)
def children(self, node):
return self._revlog.children(node)
def deltaparent(self, rev):
return self._revlog.deltaparent(rev)
def candelta(self, baserev, rev):
return self._revlog.candelta(baserev, rev)
def iscensored(self, rev):
return self._revlog.iscensored(rev)
def rawsize(self, rev):
return self._revlog.rawsize(rev)
def checkhash(self, text, node, p1=None, p2=None, rev=None):
return self._revlog.checkhash(text, node, p1=p1, p2=p2, rev=rev)
def revision(self, node, _df=None, raw=False):
return self._revlog.revision(node, _df=_df, raw=raw)
def revdiff(self, rev1, rev2):
return self._revlog.revdiff(rev1, rev2)
def addrevision(self, revisiondata, transaction, linkrev, p1, p2,
node=None, flags=revlog.REVIDX_DEFAULT_FLAGS,
cachedelta=None):
return self._revlog.addrevision(revisiondata, transaction, linkrev,
p1, p2, node=node, flags=flags,
cachedelta=cachedelta)
def addgroup(self, deltas, linkmapper, transaction, addrevisioncb=None):
return self._revlog.addgroup(deltas, linkmapper, transaction,
addrevisioncb=addrevisioncb)
def getstrippoint(self, minlink):
return self._revlog.getstrippoint(minlink)
def strip(self, minlink, transaction):
return self._revlog.strip(minlink, transaction)
def files(self):
return self._revlog.files()
def checksize(self):
return self._revlog.checksize()
def read(self, node): def read(self, node):
t = self.revision(node) t = self.revision(node)
if not t.startswith('\1\n'): if not t.startswith('\1\n'):
return t return t
s = t.index('\1\n', 2) s = t.index('\1\n', 2)
return t[s + 2:] return t[s + 2:]
# for revisions with renames, we have to go the slow way # for revisions with renames, we have to go the slow way
node = self.node(rev) node = self.node(rev)
if self.renamed(node): if self.renamed(node):
return len(self.read(node)) return len(self.read(node))
if self.iscensored(rev): if self.iscensored(rev):
return 0 return 0
# XXX if self.read(node).startswith("\1\n"), this returns (size+4) # XXX if self.read(node).startswith("\1\n"), this returns (size+4)
return super(filelog, self).size(rev) return self._revlog.size(rev)
def cmp(self, node, text): def cmp(self, node, text):
"""compare text with a given file revision """compare text with a given file revision
returns True if text is different than what is stored. returns True if text is different than what is stored.
""" """
t = text t = text
if text.startswith('\1\n'): if text.startswith('\1\n'):
t = '\1\n\1\n' + text t = '\1\n\1\n' + text
samehashes = not super(filelog, self).cmp(node, t) samehashes = not self._revlog.cmp(node, t)
if samehashes: if samehashes:
return False return False
# censored files compare against the empty file # censored files compare against the empty file
if self.iscensored(self.rev(node)): if self.iscensored(self.rev(node)):
return text != '' return text != ''
# renaming a file produces a different hash, even if the data # renaming a file produces a different hash, even if the data
# remains unchanged. Check if it's the case (slow): # remains unchanged. Check if it's the case (slow):
if self.renamed(node): if self.renamed(node):
t2 = self.read(node) t2 = self.read(node)
return t2 != text return t2 != text
return True return True
@property
def filename(self):
return self._revlog.filename
@filename.setter
def filename(self, value):
self._revlog.filename = value
# TODO these aren't part of the interface and aren't internal methods.
# Callers should be fixed to not use them.
@property
def indexfile(self):
return self._revlog.indexfile
@indexfile.setter
def indexfile(self, value):
self._revlog.indexfile = value
@property
def datafile(self):
return self._revlog.datafile
@property
def opener(self):
return self._revlog.opener
@property
def _lazydeltabase(self):
return self._revlog._lazydeltabase
@_lazydeltabase.setter
def _lazydeltabase(self, value):
self._revlog._lazydeltabase = value
@property
def _aggressivemergedeltas(self):
return self._revlog._aggressivemergedeltas
@_aggressivemergedeltas.setter
def _aggressivemergedeltas(self, value):
self._revlog._aggressivemergedeltas = value
@property
def _inline(self):
return self._revlog._inline
@property
def _withsparseread(self):
return getattr(self._revlog, '_withsparseread', False)
@property
def _srmingapsize(self):
return self._revlog._srmingapsize
@property
def _srdensitythreshold(self):
return self._revlog._srdensitythreshold
def _deltachain(self, rev, stoprev=None):
return self._revlog._deltachain(rev, stoprev)
def chainbase(self, rev):
return self._revlog.chainbase(rev)
def chainlen(self, rev):
return self._revlog.chainlen(rev)
def clone(self, tr, destrevlog, **kwargs):
if not isinstance(destrevlog, filelog):
raise error.ProgrammingError('expected filelog to clone()')
return self._revlog.clone(tr, destrevlog._revlog, **kwargs)
def start(self, rev):
return self._revlog.start(rev)
def end(self, rev):
return self._revlog.end(rev)
def length(self, rev):
return self._revlog.length(rev)
def compress(self, data):
return self._revlog.compress(data)
def _addrevision(self, *args, **kwargs):
return self._revlog._addrevision(*args, **kwargs)

mercurial/unionrepo.py

return self.revlog2.revdiff( return self.revlog2.revdiff(
self.revlog2.rev(self.node(rev1)), self.revlog2.rev(self.node(rev1)),
self.revlog2.rev(self.node(rev2))) self.revlog2.rev(self.node(rev2)))
elif rev1 <= self.repotiprev and rev2 <= self.repotiprev: elif rev1 <= self.repotiprev and rev2 <= self.repotiprev:
return self.baserevdiff(rev1, rev2) return self.baserevdiff(rev1, rev2)
return mdiff.textdiff(self.revision(rev1), self.revision(rev2)) return mdiff.textdiff(self.revision(rev1), self.revision(rev2))
def revision(self, nodeorrev, raw=False): def revision(self, nodeorrev, _df=None, raw=False):
"""return an uncompressed revision of a given node or revision """return an uncompressed revision of a given node or revision
number. number.
""" """
if isinstance(nodeorrev, int): if isinstance(nodeorrev, int):
rev = nodeorrev rev = nodeorrev
node = self.node(rev) node = self.node(rev)
else: else:
node = nodeorrev node = nodeorrev
linkmapper) linkmapper)
def baserevision(self, nodeorrev): def baserevision(self, nodeorrev):
return manifest.manifestrevlog.revision(self, nodeorrev) return manifest.manifestrevlog.revision(self, nodeorrev)
def baserevdiff(self, rev1, rev2): def baserevdiff(self, rev1, rev2):
return manifest.manifestrevlog.revdiff(self, rev1, rev2) return manifest.manifestrevlog.revdiff(self, rev1, rev2)
class unionfilelog(unionrevlog, filelog.filelog): class unionfilelog(filelog.filelog):
def __init__(self, opener, path, opener2, linkmapper, repo): def __init__(self, opener, path, opener2, linkmapper, repo):
filelog.filelog.__init__(self, opener, path) filelog.filelog.__init__(self, opener, path)
filelog2 = filelog.filelog(opener2, path) filelog2 = filelog.filelog(opener2, path)
unionrevlog.__init__(self, opener, self.indexfile, filelog2, self._revlog = unionrevlog(opener, self.indexfile,
linkmapper) filelog2._revlog, linkmapper)
self._repo = repo self._repo = repo
self.repotiprev = self._revlog.repotiprev
self.revlog2 = self._revlog.revlog2
def baserevision(self, nodeorrev): def baserevision(self, nodeorrev):
return filelog.filelog.revision(self, nodeorrev) return filelog.filelog.revision(self, nodeorrev)
def baserevdiff(self, rev1, rev2): def baserevdiff(self, rev1, rev2):
return filelog.filelog.revdiff(self, rev1, rev2) return filelog.filelog.revdiff(self, rev1, rev2)
def iscensored(self, rev): def iscensored(self, rev):