( )⚙ D4606 narrow: when writing treemanifests, skip inspecting directories outside narrow

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4606

narrow: when writing treemanifests, skip inspecting directories outside narrow
ClosedPublic

Authored by spectral on Sep 16 2018, 12:44 AM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG24870f1be088: narrow: when writing treemanifests, skip inspecting directories outside narrow
Summary

This provides significant speed benefits when narrow and treemanifests are in
use, see the timing numbers below. Note that like previously, differences of <5%
are considered noise.

The below timing numbers are in the same style as previously (example:
ee7ee0c516ca). 'before' is 9db85644, and does not include that example commit's
improvements.

diff --git:
repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u   |   |   | 1.327 s +-  0.051 s    | 1.296 s +-  0.009 s   |  97.7%
m-u   |   | x | 1.310 s +-  0.020 s    | 1.295 s +-  0.015 s   |  98.9%
m-u   | x |   | 1.295 s +-  0.018 s    | 1.296 s +-  0.007 s   | 100.1%
m-u   | x | x | 83.5 ms +-   0.8 ms    | 84.1 ms +-   0.8 ms   | 100.7%
l-d-r |   |   | 205.1 ms +-   3.5 ms   | 205.0 ms +-   3.8 ms  | 100.0%
l-d-r |   | x | 194.2 ms +-   5.6 ms   | 192.3 ms +-   4.3 ms  |  99.0%
l-d-r | x |   | 99.1 ms +-   2.2 ms    | 97.8 ms +-   0.9 ms   |  98.7%
l-d-r | x | x | 66.2 ms +-   1.0 ms    | 67.2 ms +-   2.7 ms   | 101.5%

diff -c . --git:
repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u   |   |   | 233.9 ms +-   1.9 ms   | 235.6 ms +-   5.1 ms  | 100.7%
m-u   |   | x | 151.4 ms +-   1.2 ms   | 152.2 ms +-   2.0 ms  | 100.5%
m-u   | x |   | 234.8 ms +-   2.7 ms   | 235.0 ms +-   2.7 ms  | 100.1%
m-u   | x | x | 127.8 ms +-   2.1 ms   | 126.0 ms +-   1.1 ms  |  98.6%
l-d-r |   |   | 82.5 ms +-   1.6 ms    | 82.3 ms +-   2.0 ms   |  99.8%
l-d-r |   | x | 3.742 s +-  0.017 s    | 3.819 s +-  0.208 s   | 102.1%
l-d-r | x |   | 84.4 ms +-   1.5 ms    | 83.2 ms +-   1.0 ms   |  98.6%
l-d-r | x | x | 751.2 ms +-   5.0 ms   | 755.8 ms +-  12.9 ms  | 100.6%

rebase -r . --keep -d .^^:
repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u   |   |   | 5.519 s +-  0.038 s    | 5.526 s +-  0.057 s   | 100.1%
m-u   |   | x | 5.588 s +-  0.048 s    | 5.607 s +-  0.061 s   | 100.3%
m-u   | x |   | 5.520 s +-  0.044 s    | 5.546 s +-  0.059 s   | 100.5%
m-u   | x | x | 586.6 ms +-  12.8 ms   | 554.9 ms +-  21.2 ms  |  94.6% <--
l-d-r |   |   | 629.8 ms +-   5.5 ms   | 627.4 ms +-   6.6 ms  |  99.6%
l-d-r |   | x | 6.165 s +-  0.058 s    | 6.255 s +-  0.303 s   | 101.5%
l-d-r | x |   | 270.2 ms +-   2.3 ms   | 271.4 ms +-   2.7 ms  | 100.4%
l-d-r | x | x | 4.700 s +-  0.025 s    | 1.651 s +-  0.016 s   |  35.1% <--

status --change . --copies:
repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u   |   |   | 215.4 ms +-   2.3 ms   | 216.5 ms +-   4.2 ms  | 100.5%
m-u   |   | x | 132.9 ms +-   1.2 ms   | 132.0 ms +-   1.4 ms  |  99.3%
m-u   | x |   | 217.0 ms +-   1.9 ms   | 215.4 ms +-   1.9 ms  |  99.3%
m-u   | x | x | 108.6 ms +-   1.0 ms   | 108.2 ms +-   1.5 ms  |  99.6%
l-d-r |   |   | 80.0 ms +-   1.3 ms    | 80.5 ms +-   1.1 ms   | 100.6%
l-d-r |   | x | 3.916 s +-  0.187 s    | 3.966 s +-  0.236 s   | 101.3%
l-d-r | x |   | 84.4 ms +-   3.1 ms    | 83.9 ms +-   1.1 ms   |  99.4%
l-d-r | x | x | 758.0 ms +-   8.2 ms   | 753.5 ms +-   5.0 ms  |  99.4%

status --copies:
repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u   |   |   | 1.905 s +-  0.025 s    | 1.910 s +-  0.044 s   | 100.3%
m-u   |   | x | 1.892 s +-  0.009 s    | 1.895 s +-  0.012 s   | 100.2%
m-u   | x |   | 1.891 s +-  0.012 s    | 1.902 s +-  0.018 s   | 100.6%
m-u   | x | x | 93.3 ms +-   0.9 ms    | 93.4 ms +-   0.8 ms   | 100.1%
l-d-r |   |   | 570.7 ms +-   7.8 ms   | 571.9 ms +-  18.5 ms  | 100.2%
l-d-r |   | x | 561.5 ms +-   5.2 ms   | 562.9 ms +-   6.1 ms  | 100.2%
l-d-r | x |   | 171.7 ms +-   2.6 ms   | 171.9 ms +-   1.2 ms  | 100.1%
l-d-r | x | x | 142.7 ms +-   2.0 ms   | 140.3 ms +-   1.0 ms  |  98.3%

update $rev^; ~/src/hg/hg{hg}/hg update $rev:
repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
------+---+---+------------------------+-----------------------+------------
m-u   |   |   | 3.126 s +-  0.016 s    | 3.128 s +-  0.015 s   | 100.1%
m-u   |   | x | 3.014 s +-  0.068 s    | 3.008 s +-  0.031 s   |  99.8%
m-u   | x |   | 3.143 s +-  0.037 s    | 3.184 s +-  0.086 s   | 101.3%
m-u   | x | x | 308.0 ms +-   1.8 ms   | 308.1 ms +-   5.7 ms  | 100.0%
l-d-r |   |   | 430.8 ms +-   4.5 ms   | 436.4 ms +-   8.7 ms  | 101.3%
l-d-r |   | x | 9.676 s +-  0.127 s    | 9.945 s +-  0.272 s   | 102.8%
l-d-r | x |   | 254.2 ms +-   3.3 ms   | 255.7 ms +-   3.1 ms  | 100.6%
l-d-r | x | x | 1.571 s +-  0.030 s    | 1.555 s +-  0.014 s   |  99.0%

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

spectral created this revision.Sep 16 2018, 12:44 AM
yuja added a subscriber: yuja.Sep 16 2018, 9:48 PM

Looks good to me. One nit.

  • a/mercurial/manifest.py

+++ b/mercurial/manifest.py
@@ -1203,7 +1203,7 @@

    s._dirty = False
self._loadfunc = _load_for_read
  • def writesubtrees(self, m1, m2, writesubtree):

+ def writesubtrees(self, m1, m2, writesubtree, match=None):

Can you remove the unsupported default value match=None from inner functions?

  • def add(self, m, transaction, link, p1, p2, added, removed, readtree=None):

+ def add(self, m, transaction, link, p1, p2, added, removed, readtree=None,
+ match=None):

if p1 in self.fulltextcache and util.safehasattr(m, 'fastdelta'):
    # If our first parent is in the manifest cache, we can
    # compute a delta here using properties we know about the

@@ -1471,7 +1481,8 @@

assert readtree, "readtree must be set for treemanifest writes"
m1 = readtree(self.tree, p1)
m2 = readtree(self.tree, p2)
  • n = self._addtree(m, transaction, link, m1, m2, readtree)

+ n = self._addtree(m, transaction, link, m1, m2, readtree,
+ match=match)

Perhaps, assert match or matchmod.always() can be added.

martinvonz added a subscriber: martinvonz.Sep 17 2018, 11:55 AM
martinvonz added inline comments.
mercurial/localrepo.py
2134–2135

Note that that the merge will fail with "...which is not *yet* supported" (emphasis added). It's probably going to be a long time before we support that case, because it will probably involve indicating the in the mergestate which side of the merge to indicate which side of the merge to choose for these files outside the narrowspec. But that's far enough away that it shouldn't block this patch. I'm mentioning it here just so others know what my plans for that were.

spectral updated this revision to Diff 11112.Sep 17 2018, 2:31 PM
spectral updated this revision to Diff 11113.Sep 17 2018, 2:53 PM
spectral added a comment.Sep 17 2018, 2:55 PM
In D4606#70323, @yuja wrote:

Looks good to me. One nit.

  • a/mercurial/manifest.py

+++ b/mercurial/manifest.py
@@ -1203,7 +1203,7 @@

    s._dirty = False
self._loadfunc = _load_for_read
  • def writesubtrees(self, m1, m2, writesubtree):

+ def writesubtrees(self, m1, m2, writesubtree, match=None):

Can you remove the unsupported default value match=None from inner functions?

Done.

  • def add(self, m, transaction, link, p1, p2, added, removed, readtree=None):

+ def add(self, m, transaction, link, p1, p2, added, removed, readtree=None,
+ match=None):

if p1 in self.fulltextcache and util.safehasattr(m, 'fastdelta'):
    # If our first parent is in the manifest cache, we can
    # compute a delta here using properties we know about the

@@ -1471,7 +1481,8 @@

assert readtree, "readtree must be set for treemanifest writes"
m1 = readtree(self.tree, p1)
m2 = readtree(self.tree, p2)
  • n = self._addtree(m, transaction, link, m1, m2, readtree)

+ n = self._addtree(m, transaction, link, m1, m2, readtree,
+ match=match)

Perhaps, assert match or matchmod.always() can be added.

Added, I hadn't realized that I'd accidentally not handled match=None... Turns out only manifestctx.write() calls this add() method, and only localrepo calls that write() method :)

Closed by commit rHG24870f1be088: narrow: when writing treemanifests, skip inspecting directories outside narrow (authored by spectral). · Explain WhySep 18 2018, 9:47 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/localrepo.py (9 lines)
Mmercurial/manifest.py (36 lines)
Mmercurial/repository.py (15 lines)

Diff 11112

mercurial/localrepo.py

# try hard to detect unmodified manifest entry so that the # try hard to detect unmodified manifest entry so that the
# exact same commit can be reproduced later on convert. # exact same commit can be reproduced later on convert.
md = m1.diff(m, scmutil.matchfiles(self, ctx.files())) md = m1.diff(m, scmutil.matchfiles(self, ctx.files()))
if not files and md: if not files and md:
self.ui.debug('not reusing manifest (no file change in ' self.ui.debug('not reusing manifest (no file change in '
'changelog, but manifest differs)\n') 'changelog, but manifest differs)\n')
if files or md: if files or md:
self.ui.note(_("committing manifest\n")) self.ui.note(_("committing manifest\n"))
# we're using narrowmatch here since it's already applied at
# other stages (such as dirstate.walk), so we're already
# ignoring things outside of narrowspec in most cases. The
# one case where we might have files outside the narrowspec
# at this point is merges, and we already error out in the
martinvonzUnsubmitted
Not Done

Note that that the merge will fail with "...which is not *yet* supported" (emphasis added). It's probably going to be a long time before we support that case, because it will probably involve indicating the in the mergestate which side of the merge to indicate which side of the merge to choose for these files outside the narrowspec. But that's far enough away that it shouldn't block this patch. I'm mentioning it here just so others know what my plans for that were.

martinvonz: Note that that the merge will fail with "...which is not *yet* supported" (emphasis added).
# case where the merge has files outside of the narrowspec,
# so this is safe.
mn = mctx.write(trp, linkrev, mn = mctx.write(trp, linkrev,
p1.manifestnode(), p2.manifestnode(), p1.manifestnode(), p2.manifestnode(),
added, drop) added, drop, match=self.narrowmatch())
else: else:
self.ui.debug('reusing manifest form p1 (listed files ' self.ui.debug('reusing manifest form p1 (listed files '
'actually unchanged)\n') 'actually unchanged)\n')
mn = p1.manifestnode() mn = p1.manifestnode()
else: else:
self.ui.debug('reusing manifest from p1 (no file change)\n') self.ui.debug('reusing manifest from p1 (no file change)\n')
mn = p1.manifestnode() mn = p1.manifestnode()
files = [] files = []

mercurial/manifest.py

return _text(sorted(dirs + files + lazydirs)) return _text(sorted(dirs + files + lazydirs))
def read(self, gettext, readsubtree): def read(self, gettext, readsubtree):
def _load_for_read(s): def _load_for_read(s):
s.parse(gettext(), readsubtree) s.parse(gettext(), readsubtree)
s._dirty = False s._dirty = False
self._loadfunc = _load_for_read self._loadfunc = _load_for_read
def writesubtrees(self, m1, m2, writesubtree): def writesubtrees(self, m1, m2, writesubtree, match=None):
self._load() # for consistency; should never have any effect here self._load() # for consistency; should never have any effect here
m1._load() m1._load()
m2._load() m2._load()
emptytree = treemanifest() emptytree = treemanifest()
def getnode(m, d): def getnode(m, d):
ld = m._lazydirs.get(d) ld = m._lazydirs.get(d)
if ld: if ld:
return ld[1] return ld[1]
return m._dirs.get(d, emptytree)._node return m._dirs.get(d, emptytree)._node
# we should have always loaded everything by the time we get here for
# `self`, but possibly not in `m1` or `m2`.
assert not self._lazydirs
# let's skip investigating things that `match` says we do not need.
visit = match.visitchildrenset(self._dir[:-1] or '.')
if visit == 'this' or visit == 'all':
visit = None
for d, subm in self._dirs.iteritems(): for d, subm in self._dirs.iteritems():
if visit and d[:-1] not in visit:
continue
subp1 = getnode(m1, d) subp1 = getnode(m1, d)
subp2 = getnode(m2, d) subp2 = getnode(m2, d)
if subp1 == nullid: if subp1 == nullid:
subp1, subp2 = subp2, subp1 subp1, subp2 = subp2, subp1
writesubtree(subm, subp1, subp2) writesubtree(subm, subp1, subp2, match=match)
def walksubtrees(self, matcher=None): def walksubtrees(self, matcher=None):
"""Returns an iterator of the subtrees of this manifest, including this """Returns an iterator of the subtrees of this manifest, including this
manifest itself. manifest itself.
If `matcher` is provided, it only returns subtrees that match. If `matcher` is provided, it only returns subtrees that match.
""" """
if matcher and not matcher.visitdir(self._dir[:-1] or '.'): if matcher and not matcher.visitdir(self._dir[:-1] or '.'):
assert self._treeondisk assert self._treeondisk
if d not in self._dirlogcache: if d not in self._dirlogcache:
mfrevlog = manifestrevlog(self.opener, d, mfrevlog = manifestrevlog(self.opener, d,
self._dirlogcache, self._dirlogcache,
treemanifest=self._treeondisk) treemanifest=self._treeondisk)
self._dirlogcache[d] = mfrevlog self._dirlogcache[d] = mfrevlog
return self._dirlogcache[d] return self._dirlogcache[d]
def add(self, m, transaction, link, p1, p2, added, removed, readtree=None): def add(self, m, transaction, link, p1, p2, added, removed, readtree=None,
match=None):
if p1 in self.fulltextcache and util.safehasattr(m, 'fastdelta'): if p1 in self.fulltextcache and util.safehasattr(m, 'fastdelta'):
# If our first parent is in the manifest cache, we can # If our first parent is in the manifest cache, we can
# compute a delta here using properties we know about the # compute a delta here using properties we know about the
# manifest up-front, which may save time later for the # manifest up-front, which may save time later for the
# revlog layer. # revlog layer.
_checkforbidden(added) _checkforbidden(added)
# combine the changed lists into one sorted iterator # combine the changed lists into one sorted iterator
# The first parent manifest isn't already loaded, so we'll # The first parent manifest isn't already loaded, so we'll
# just encode a fulltext of the manifest and pass that # just encode a fulltext of the manifest and pass that
# through to the revlog layer, and let it handle the delta # through to the revlog layer, and let it handle the delta
# process. # process.
if self._treeondisk: if self._treeondisk:
assert readtree, "readtree must be set for treemanifest writes" assert readtree, "readtree must be set for treemanifest writes"
m1 = readtree(self.tree, p1) m1 = readtree(self.tree, p1)
m2 = readtree(self.tree, p2) m2 = readtree(self.tree, p2)
n = self._addtree(m, transaction, link, m1, m2, readtree) n = self._addtree(m, transaction, link, m1, m2, readtree,
match=match)
arraytext = None arraytext = None
else: else:
text = m.text() text = m.text()
n = self._revlog.addrevision(text, transaction, link, p1, p2) n = self._revlog.addrevision(text, transaction, link, p1, p2)
arraytext = bytearray(text) arraytext = bytearray(text)
if arraytext is not None: if arraytext is not None:
self.fulltextcache[n] = arraytext self.fulltextcache[n] = arraytext
return n return n
def _addtree(self, m, transaction, link, m1, m2, readtree): def _addtree(self, m, transaction, link, m1, m2, readtree, match):
# If the manifest is unchanged compared to one parent, # If the manifest is unchanged compared to one parent,
# don't write a new revision # don't write a new revision
if self.tree != '' and (m.unmodifiedsince(m1) or m.unmodifiedsince( if self.tree != '' and (m.unmodifiedsince(m1) or m.unmodifiedsince(
m2)): m2)):
return m.node() return m.node()
def writesubtree(subm, subp1, subp2): def writesubtree(subm, subp1, subp2, match):
sublog = self.dirlog(subm.dir()) sublog = self.dirlog(subm.dir())
sublog.add(subm, transaction, link, subp1, subp2, None, None, sublog.add(subm, transaction, link, subp1, subp2, None, None,
readtree=readtree) readtree=readtree, match=match)
m.writesubtrees(m1, m2, writesubtree) m.writesubtrees(m1, m2, writesubtree, match=match)
text = m.dirtext() text = m.dirtext()
n = None n = None
if self.tree != '': if self.tree != '':
# Double-check whether contents are unchanged to one parent # Double-check whether contents are unchanged to one parent
if text == m1.dirtext(): if text == m1.dirtext():
n = m1.node() n = m1.node()
elif text == m2.dirtext(): elif text == m2.dirtext():
n = m2.node() n = m2.node()
def copy(self): def copy(self):
memmf = memmanifestctx(self._manifestlog) memmf = memmanifestctx(self._manifestlog)
memmf._manifestdict = self.read().copy() memmf._manifestdict = self.read().copy()
return memmf return memmf
def read(self): def read(self):
return self._manifestdict return self._manifestdict
def write(self, transaction, link, p1, p2, added, removed): def write(self, transaction, link, p1, p2, added, removed, match=None):
return self._storage().add(self._manifestdict, transaction, link, return self._storage().add(self._manifestdict, transaction, link,
p1, p2, added, removed) p1, p2, added, removed, match=match)
@interfaceutil.implementer(repository.imanifestrevisionstored) @interfaceutil.implementer(repository.imanifestrevisionstored)
class manifestctx(object): class manifestctx(object):
"""A class representing a single revision of a manifest, including its """A class representing a single revision of a manifest, including its
contents, its parent revs, and its linkrev. contents, its parent revs, and its linkrev.
""" """
def __init__(self, manifestlog, node): def __init__(self, manifestlog, node):
self._manifestlog = manifestlog self._manifestlog = manifestlog
def copy(self): def copy(self):
memmf = memtreemanifestctx(self._manifestlog, dir=self._dir) memmf = memtreemanifestctx(self._manifestlog, dir=self._dir)
memmf._treemanifest = self._treemanifest.copy() memmf._treemanifest = self._treemanifest.copy()
return memmf return memmf
def read(self): def read(self):
return self._treemanifest return self._treemanifest
def write(self, transaction, link, p1, p2, added, removed): def write(self, transaction, link, p1, p2, added, removed, match=None):
def readtree(dir, node): def readtree(dir, node):
return self._manifestlog.get(dir, node).read() return self._manifestlog.get(dir, node).read()
return self._storage().add(self._treemanifest, transaction, link, return self._storage().add(self._treemanifest, transaction, link,
p1, p2, added, removed, readtree=readtree) p1, p2, added, removed, readtree=readtree,
match=match)
@interfaceutil.implementer(repository.imanifestrevisionstored) @interfaceutil.implementer(repository.imanifestrevisionstored)
class treemanifestctx(object): class treemanifestctx(object):
def __init__(self, manifestlog, dir, node): def __init__(self, manifestlog, dir, node):
self._manifestlog = manifestlog self._manifestlog = manifestlog
self._dir = dir self._dir = dir
self._data = None self._data = None

mercurial/repository.py

"""Calls self.read().find(key)``. """Calls self.read().find(key)``.
Returns a 2-tuple of ``(node, flags)`` or raises ``KeyError``. Returns a 2-tuple of ``(node, flags)`` or raises ``KeyError``.
""" """
class imanifestrevisionwritable(imanifestrevisionbase): class imanifestrevisionwritable(imanifestrevisionbase):
"""Interface representing a manifest revision that can be committed.""" """Interface representing a manifest revision that can be committed."""
def write(transaction, linkrev, p1node, p2node, added, removed): def write(transaction, linkrev, p1node, p2node, added, removed, match=None):
"""Add this revision to storage. """Add this revision to storage.
Takes a transaction object, the changeset revision number it will Takes a transaction object, the changeset revision number it will
be associated with, its parent nodes, and lists of added and be associated with, its parent nodes, and lists of added and
removed paths. removed paths.
If match is provided, storage can choose not to inspect or write out
items that do not match. Storage is still required to be able to provide
the full manifest in the future for any directories written (these
manifests should not be "narrowed on disk").
Returns the binary node of the created revision. Returns the binary node of the created revision.
""" """
class imanifeststorage(interfaceutil.Interface): class imanifeststorage(interfaceutil.Interface):
"""Storage interface for manifest data.""" """Storage interface for manifest data."""
tree = interfaceutil.Attribute( tree = interfaceutil.Attribute(
"""The path to the directory this manifest tracks. """The path to the directory this manifest tracks.
"""Clone this instance to another.""" """Clone this instance to another."""
def clearcaches(clear_persisted_data=False): def clearcaches(clear_persisted_data=False):
"""Clear any caches associated with this instance.""" """Clear any caches associated with this instance."""
def dirlog(d): def dirlog(d):
"""Obtain a manifest storage instance for a tree.""" """Obtain a manifest storage instance for a tree."""
def add(m, transaction, link, p1, p2, added, removed, readtree=None): def add(m, transaction, link, p1, p2, added, removed, readtree=None,
match=None):
"""Add a revision to storage. """Add a revision to storage.
``m`` is an object conforming to ``imanifestdict``. ``m`` is an object conforming to ``imanifestdict``.
``link`` is the linkrev revision number. ``link`` is the linkrev revision number.
``p1`` and ``p2`` are the parent revision numbers. ``p1`` and ``p2`` are the parent revision numbers.
``added`` and ``removed`` are iterables of added and removed paths, ``added`` and ``removed`` are iterables of added and removed paths,
respectively. respectively.
``match`` is a matcher that can be used to hint to storage that not all
paths must be inspected; this is an optimization and can be safely
ignored. Note that the storage must still be able to reproduce a full
manifest including files that did not match.
""" """
class imanifestlog(interfaceutil.Interface): class imanifestlog(interfaceutil.Interface):
"""Interface representing a collection of manifest snapshots. """Interface representing a collection of manifest snapshots.
Represents the root manifest in a repository. Represents the root manifest in a repository.
Also serves as a means to access nested tree manifests and to cache Also serves as a means to access nested tree manifests and to cache