( )⚙ D6151 branchmap: remove the dict interface from the branchcache class (API)

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D6151

branchmap: remove the dict interface from the branchcache class (API)
ClosedPublic

Authored by pulkit on Mar 19 2019, 7:30 AM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG624d6683c705: branchmap: remove the dict interface from the branchcache class (API)
Summary

The current branchmap computation involves reading the whole branchmap from
disk, validating all the nodes even if they are not required. This leads to a
lot of time on repos which have large branchmap or a lot of branches. On large
repos, this can validate around 1000's of nodes.

On some operations, like finding whether a branch exists or not, we don't need
to validate all the nodes. Or updating heads for a single branch.

Before this patch, branchcache class was having dict interface and it was hard
to keep track of reads.

This patch removes the dict interface. Upcoming patches will implement lazy
loading and validation of data and implement better API's.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

pulkit created this revision.Mar 19 2019, 7:30 AM
Closed by commit rHG624d6683c705: branchmap: remove the dict interface from the branchcache class (API) (authored by pulkit). · Explain WhyMar 23 2019, 11:39 AM
This revision was automatically updated to reflect the committed changes.
yuja added a subscriber: yuja.Mar 23 2019, 9:10 PM
def copy(self):
    """return an deep copy of the branchcache object"""
  • return type(self)(
  • self, self.tipnode, self.tiprev, self.filteredhash,

+ return branchcache(
+ self.entries, self.tipnode, self.tiprev, self.filteredhash,

self._closednodes)

In order to support subclassing, a branchcache type has to be resolved
dynamically. IIRC, that's the point of type(self) use here.

Revision Contents
Changeset List

Diff 14583

mercurial/branchmap.py

self._per_filter[candidate] = cache self._per_filter[candidate] = cache
cache.write(rview) cache.write(rview)
return return
def clear(self): def clear(self):
self._per_filter.clear() self._per_filter.clear()
class branchcache(dict): class branchcache(object):
"""A dict like object that hold branches heads cache. """A dict like object that hold branches heads cache.
This cache is used to avoid costly computations to determine all the This cache is used to avoid costly computations to determine all the
branch heads of a repo. branch heads of a repo.
The cache is serialized on disk in the following format: The cache is serialized on disk in the following format:
<tip hex node> <tip rev number> [optional filtered repo hex hash] <tip hex node> <tip rev number> [optional filtered repo hex hash]
<branch head hex node> <open/closed state> <branch name> <branch head hex node> <open/closed state> <branch name>
<branch head hex node> <open/closed state> <branch name> <branch head hex node> <open/closed state> <branch name>
... ...
The first line is used to check if the cache is still valid. If the The first line is used to check if the cache is still valid. If the
branch cache is for a filtered repo view, an optional third hash is branch cache is for a filtered repo view, an optional third hash is
included that hashes the hashes of all filtered revisions. included that hashes the hashes of all filtered revisions.
The open/closed state is represented by a single letter 'o' or 'c'. The open/closed state is represented by a single letter 'o' or 'c'.
This field can be used to avoid changelog reads when determining if a This field can be used to avoid changelog reads when determining if a
branch head closes a branch or not. branch head closes a branch or not.
""" """
def __init__(self, entries=(), tipnode=nullid, tiprev=nullrev, def __init__(self, entries=(), tipnode=nullid, tiprev=nullrev,
filteredhash=None, closednodes=None): filteredhash=None, closednodes=None):
super(branchcache, self).__init__(entries)
self.tipnode = tipnode self.tipnode = tipnode
self.tiprev = tiprev self.tiprev = tiprev
self.filteredhash = filteredhash self.filteredhash = filteredhash
# closednodes is a set of nodes that close their branch. If the branch # closednodes is a set of nodes that close their branch. If the branch
# cache has been updated, it may contain nodes that are no longer # cache has been updated, it may contain nodes that are no longer
# heads. # heads.
if closednodes is None: if closednodes is None:
self._closednodes = set() self._closednodes = set()
else: else:
self._closednodes = closednodes self._closednodes = closednodes
self.entries = dict(entries)
def __iter__(self):
return iter(self.entries)
def __setitem__(self, key, value):
self.entries[key] = value
def __getitem__(self, key):
return self.entries[key]
def setdefault(self, *args):
return self.entries.setdefault(*args)
def iteritems(self):
return self.entries.iteritems()
def itervalues(self):
return self.entries.itervalues()
@classmethod @classmethod
def fromfile(cls, repo): def fromfile(cls, repo):
f = None f = None
try: try:
f = repo.cachevfs(cls._filename(repo)) f = repo.cachevfs(cls._filename(repo))
lineiter = iter(f) lineiter = iter(f)
cachekey = next(lineiter).rstrip('\n').split(" ", 2) cachekey = next(lineiter).rstrip('\n').split(" ", 2)
return heads return heads
def iterbranches(self): def iterbranches(self):
for bn, heads in self.iteritems(): for bn, heads in self.iteritems():
yield (bn, heads) + self._branchtip(heads) yield (bn, heads) + self._branchtip(heads)
def copy(self): def copy(self):
"""return an deep copy of the branchcache object""" """return an deep copy of the branchcache object"""
return type(self)( return branchcache(
self, self.tipnode, self.tiprev, self.filteredhash, self.entries, self.tipnode, self.tiprev, self.filteredhash,
self._closednodes) self._closednodes)
def write(self, repo): def write(self, repo):
try: try:
f = repo.cachevfs(self._filename(repo), "w", atomictemp=True) f = repo.cachevfs(self._filename(repo), "w", atomictemp=True)
cachekey = [hex(self.tipnode), '%d' % self.tiprev] cachekey = [hex(self.tipnode), '%d' % self.tiprev]
if self.filteredhash is not None: if self.filteredhash is not None:
cachekey.append(hex(self.filteredhash)) cachekey.append(hex(self.filteredhash))
f.write(" ".join(cachekey) + '\n') f.write(" ".join(cachekey) + '\n')
nodecount = 0 nodecount = 0
for label, nodes in sorted(self.iteritems()): for label, nodes in sorted(self.iteritems()):
label = encoding.fromlocal(label) label = encoding.fromlocal(label)
for node in nodes: for node in nodes:
nodecount += 1 nodecount += 1
if node in self._closednodes: if node in self._closednodes:
state = 'c' state = 'c'
else: else:
state = 'o' state = 'o'
f.write("%s %s %s\n" % (hex(node), state, label)) f.write("%s %s %s\n" % (hex(node), state, label))
f.close() f.close()
repo.ui.log('branchcache', repo.ui.log('branchcache',
'wrote %s branch cache with %d labels and %d nodes\n', 'wrote %s branch cache with %d labels and %d nodes\n',
repo.filtername, len(self), nodecount) repo.filtername, len(self.entries), nodecount)
except (IOError, OSError, error.Abort) as inst: except (IOError, OSError, error.Abort) as inst:
# Abort may be raised by read only opener, so log and continue # Abort may be raised by read only opener, so log and continue
repo.ui.debug("couldn't write branch cache: %s\n" % repo.ui.debug("couldn't write branch cache: %s\n" %
stringutil.forcebytestr(inst)) stringutil.forcebytestr(inst))
def update(self, repo, revgen): def update(self, repo, revgen):
"""Given a branchhead cache, self, that may have extra nodes or be """Given a branchhead cache, self, that may have extra nodes or be
missing heads, and a generator of nodes that are strictly a superset of missing heads, and a generator of nodes that are strictly a superset of
if tiprev > self.tiprev: if tiprev > self.tiprev:
self.tipnode = cl.node(tiprev) self.tipnode = cl.node(tiprev)
self.tiprev = tiprev self.tiprev = tiprev
if not self.validfor(repo): if not self.validfor(repo):
# cache key are not valid anymore # cache key are not valid anymore
self.tipnode = nullid self.tipnode = nullid
self.tiprev = nullrev self.tiprev = nullrev
for heads in self.values(): for heads in self.itervalues():
tiprev = max(cl.rev(node) for node in heads) tiprev = max(cl.rev(node) for node in heads)
if tiprev > self.tiprev: if tiprev > self.tiprev:
self.tipnode = cl.node(tiprev) self.tipnode = cl.node(tiprev)
self.tiprev = tiprev self.tiprev = tiprev
self.filteredhash = scmutil.filteredhash(repo, self.tiprev) self.filteredhash = scmutil.filteredhash(repo, self.tiprev)
duration = util.timer() - starttime duration = util.timer() - starttime
repo.ui.log('branchcache', 'updated %s branch cache in %.4f seconds\n', repo.ui.log('branchcache', 'updated %s branch cache in %.4f seconds\n',

mercurial/localrepo.py

raise error.RepoLookupError(_("unknown branch '%s'") % branch) raise error.RepoLookupError(_("unknown branch '%s'") % branch)
else: else:
pass pass
def lookup(self, key): def lookup(self, key):
return scmutil.revsymbol(self, key).node() return scmutil.revsymbol(self, key).node()
def lookupbranch(self, key): def lookupbranch(self, key):
if key in self.branchmap(): if key in self.branchmap().entries:
return key return key
return scmutil.revsymbol(self, key).branch() return scmutil.revsymbol(self, key).branch()
def known(self, nodes): def known(self, nodes):
cl = self.changelog cl = self.changelog
nm = cl.nodemap nm = cl.nodemap
filtered = cl.filteredrevs filtered = cl.filteredrevs
Heads are returned in topological order, from newest to oldest. Heads are returned in topological order, from newest to oldest.
If branch is None, use the dirstate branch. If branch is None, use the dirstate branch.
If start is not None, return only heads reachable from start. If start is not None, return only heads reachable from start.
If closed is True, return heads that are marked as closed as well. If closed is True, return heads that are marked as closed as well.
''' '''
if branch is None: if branch is None:
branch = self[None].branch() branch = self[None].branch()
branches = self.branchmap() branches = self.branchmap()
if branch not in branches: if branch not in branches.entries:
return [] return []
# the cache returns heads ordered lowest to highest # the cache returns heads ordered lowest to highest
bheads = list(reversed(branches.branchheads(branch, closed=closed))) bheads = list(reversed(branches.branchheads(branch, closed=closed)))
if start is not None: if start is not None:
# filter out the heads that cannot be reached from startrev # filter out the heads that cannot be reached from startrev
fbheads = set(self.changelog.nodesbetween([start], bheads)[2]) fbheads = set(self.changelog.nodesbetween([start], bheads)[2])
bheads = [h for h in bheads if h in fbheads] bheads = [h for h in bheads if h in fbheads]
return bheads return bheads