This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4747

revlog: add method for obtaining storage info (API)
ClosedPublic

Authored by indygreg on Sep 26 2018, 2:28 PM.

Details

Reviewers
pulkit
Group Reviewers
hg-reviewers
Commits
rHG14e500b58263: revlog: add method for obtaining storage info (API)
Summary

We currently have a handful of methods on the file and manifest
storage interfaces for obtaining metadata about storage. e.g.
files() is used to obtain the files backing storage. rawsize()
is to quickly compute the size of tracked revisions without resolving
their fulltext.

Code in upgrade and stream clone make heavy use of these methods.

The existing APIs are generic and don't necessarily have the
specialization that we need going forward. For example, files()
doesn't distinguish between exclusive storage and shared storage.
This makes stream clone difficult to implement when e.g. there may
be a single file backing storage for multiple tracked paths. It
also makes reporting difficult, as we don't know how many bytes are
actually used by storage since we can't easily identify shared files.

This commit implements a new method for obtaining storage metadata.

It is designed to accept arguments specifying what metadata to request
and to return a dict with those fields populated. We /could/ make
each of these attributes a separate method. But this is a specialized
API and I'm trying to avoid method bloat on the interfaces. There is
also the possibility that certain callers will want to obtain multiple
fields in different combinations and some backends may have performance
issues obtaining all that data via separate method calls.

Simple storage integration tests have been added. For now, we assume
fields can't be "None" (ignoring the interface documentation). We can
revisit this later.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Sep 26 2018, 2:28 PM
pulkit accepted this revision.Sep 26 2018, 3:19 PM
Closed by commit rHG14e500b58263: revlog: add method for obtaining storage info (API) (authored by indygreg). · Explain WhySep 26 2018, 3:32 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 11402

mercurial/filelog.py

t2 = self.read(node) t2 = self.read(node)
return t2 != text return t2 != text
return True return True
def verifyintegrity(self, state): def verifyintegrity(self, state):
return self._revlog.verifyintegrity(state) return self._revlog.verifyintegrity(state)
def storageinfo(self, exclusivefiles=False, sharedfiles=False,
revisionscount=False, trackedsize=False,
storedsize=False):
return self._revlog.storageinfo(
exclusivefiles=exclusivefiles, sharedfiles=sharedfiles,
revisionscount=revisionscount, trackedsize=trackedsize,
storedsize=storedsize)
# TODO these aren't part of the interface and aren't internal methods. # TODO these aren't part of the interface and aren't internal methods.
# Callers should be fixed to not use them. # Callers should be fixed to not use them.
# Used by bundlefilelog, unionfilelog. # Used by bundlefilelog, unionfilelog.
@property @property
def indexfile(self): def indexfile(self):
return self._revlog.indexfile return self._revlog.indexfile

mercurial/manifest.py

return self._revlog.files() return self._revlog.files()
def clone(self, tr, destrevlog, **kwargs): def clone(self, tr, destrevlog, **kwargs):
if not isinstance(destrevlog, manifestrevlog): if not isinstance(destrevlog, manifestrevlog):
raise error.ProgrammingError('expected manifestrevlog to clone()') raise error.ProgrammingError('expected manifestrevlog to clone()')
return self._revlog.clone(tr, destrevlog._revlog, **kwargs) return self._revlog.clone(tr, destrevlog._revlog, **kwargs)
def storageinfo(self, exclusivefiles=False, sharedfiles=False,
revisionscount=False, trackedsize=False,
storedsize=False):
return self._revlog.storageinfo(
exclusivefiles=exclusivefiles, sharedfiles=sharedfiles,
revisionscount=revisionscount, trackedsize=trackedsize,
storedsize=storedsize)
@property @property
def indexfile(self): def indexfile(self):
return self._revlog.indexfile return self._revlog.indexfile
@indexfile.setter @indexfile.setter
def indexfile(self, value): def indexfile(self, value):
self._revlog.indexfile = value self._revlog.indexfile = value

mercurial/repository.py

def files(): def files():
"""Obtain paths that are backing storage for this file. """Obtain paths that are backing storage for this file.
TODO this is used heavily by verify code and there should probably TODO this is used heavily by verify code and there should probably
be a better API for that. be a better API for that.
""" """
def storageinfo(exclusivefiles=False, sharedfiles=False,
revisionscount=False, trackedsize=False,
storedsize=False):
"""Obtain information about storage for this file's data.
Returns a dict describing storage for this tracked path. The keys
in the dict map to arguments of the same. The arguments are bools
indicating whether to calculate and obtain that data.
exclusivefiles
Iterable of (vfs, path) describing files that are exclusively
used to back storage for this tracked path.
sharedfiles
Iterable of (vfs, path) describing files that are used to back
storage for this tracked path. Those files may also provide storage
for other stored entities.
revisionscount
Number of revisions available for retrieval.
trackedsize
Total size in bytes of all tracked revisions. This is a sum of the
length of the fulltext of all revisions.
storedsize
Total size in bytes used to store data for all tracked revisions.
This is commonly less than ``trackedsize`` due to internal usage
of deltas rather than fulltext revisions.
Not all storage backends may support all queries are have a reasonable
value to use. In that case, the value should be set to ``None`` and
callers are expected to handle this special value.
"""
def verifyintegrity(state): def verifyintegrity(state):
"""Verifies the integrity of file storage. """Verifies the integrity of file storage.
``state`` is a dict holding state of the verifier process. It can be ``state`` is a dict holding state of the verifier process. It can be
used to communicate data between invocations of multiple storage used to communicate data between invocations of multiple storage
primitives. primitives.
If individual revisions cannot have their revision content resolved, If individual revisions cannot have their revision content resolved,
treemanifets. treemanifets.
``match`` is a matcher that can be used to hint to storage that not all ``match`` is a matcher that can be used to hint to storage that not all
paths must be inspected; this is an optimization and can be safely paths must be inspected; this is an optimization and can be safely
ignored. Note that the storage must still be able to reproduce a full ignored. Note that the storage must still be able to reproduce a full
manifest including files that did not match. manifest including files that did not match.
""" """
def storageinfo(exclusivefiles=False, sharedfiles=False,
revisionscount=False, trackedsize=False,
storedsize=False):
"""Obtain information about storage for this manifest's data.
See ``ifilestorage.storageinfo()`` for a description of this method.
This one behaves the same way, except for manifest data.
"""
class imanifestlog(interfaceutil.Interface): class imanifestlog(interfaceutil.Interface):
"""Interface representing a collection of manifest snapshots. """Interface representing a collection of manifest snapshots.
Represents the root manifest in a repository. Represents the root manifest in a repository.
Also serves as a means to access nested tree manifests and to cache Also serves as a means to access nested tree manifests and to cache
tree manifests. tree manifests.
""" """

mercurial/revlog.py

yield revlogproblem(error=_('censored file data'), yield revlogproblem(error=_('censored file data'),
node=node) node=node)
state['skipread'].add(node) state['skipread'].add(node)
except Exception as e: except Exception as e:
yield revlogproblem( yield revlogproblem(
error=_('unpacking %s: %s') % (short(node), e), error=_('unpacking %s: %s') % (short(node), e),
node=node) node=node)
state['skipread'].add(node) state['skipread'].add(node)
def storageinfo(self, exclusivefiles=False, sharedfiles=False,
revisionscount=False, trackedsize=False,
storedsize=False):
d = {}
if exclusivefiles:
d['exclusivefiles'] = [(self.opener, self.indexfile)]
if not self._inline:
d['exclusivefiles'].append((self.opener, self.datafile))
if sharedfiles:
d['sharedfiles'] = []
if revisionscount:
d['revisionscount'] = len(self)
if trackedsize:
d['trackedsize'] = sum(map(self.rawsize, iter(self)))
if storedsize:
d['storedsize'] = sum(self.opener.stat(path).st_size
for path in self.files())
return d

mercurial/testing/storage.py

All file storage backends for data should conform to the tests in this All file storage backends for data should conform to the tests in this
class. class.
Use ``makeifiledatatests()`` to create an instance of this type. Use ``makeifiledatatests()`` to create an instance of this type.
""" """
def testempty(self): def testempty(self):
f = self._makefilefn() f = self._makefilefn()
self.assertEqual(f.storageinfo(), {})
self.assertEqual(f.storageinfo(revisionscount=True, trackedsize=True),
{'revisionscount': 0, 'trackedsize': 0})
self.assertEqual(f.rawsize(nullrev), 0) self.assertEqual(f.rawsize(nullrev), 0)
for i in range(-5, 5): for i in range(-5, 5):
if i == nullrev: if i == nullrev:
continue continue
with self.assertRaises(IndexError): with self.assertRaises(IndexError):
f.rawsize(i) f.rawsize(i)
def testsinglerevision(self): def testsinglerevision(self):
fulltext = b'initial' fulltext = b'initial'
f = self._makefilefn() f = self._makefilefn()
with self._maketransactionfn() as tr: with self._maketransactionfn() as tr:
node = f.add(fulltext, None, tr, 0, nullid, nullid) node = f.add(fulltext, None, tr, 0, nullid, nullid)
self.assertEqual(f.storageinfo(), {})
self.assertEqual(f.storageinfo(revisionscount=True, trackedsize=True),
{'revisionscount': 1, 'trackedsize': len(fulltext)})
self.assertEqual(f.rawsize(0), len(fulltext)) self.assertEqual(f.rawsize(0), len(fulltext))
with self.assertRaises(IndexError): with self.assertRaises(IndexError):
f.rawsize(1) f.rawsize(1)
self.assertEqual(f.size(0), len(fulltext)) self.assertEqual(f.size(0), len(fulltext))
with self.assertRaises(IndexError): with self.assertRaises(IndexError):
fulltext2 = b'y' + fulltext0 + b'z' fulltext2 = b'y' + fulltext0 + b'z'
f = self._makefilefn() f = self._makefilefn()
with self._maketransactionfn() as tr: with self._maketransactionfn() as tr:
node0 = f.add(fulltext0, None, tr, 0, nullid, nullid) node0 = f.add(fulltext0, None, tr, 0, nullid, nullid)
node1 = f.add(fulltext1, None, tr, 1, node0, nullid) node1 = f.add(fulltext1, None, tr, 1, node0, nullid)
node2 = f.add(fulltext2, None, tr, 3, node1, nullid) node2 = f.add(fulltext2, None, tr, 3, node1, nullid)
self.assertEqual(f.storageinfo(), {})
self.assertEqual(
f.storageinfo(revisionscount=True, trackedsize=True),
{
'revisionscount': 3,
'trackedsize': len(fulltext0) + len(fulltext1) + len(fulltext2),
})
self.assertEqual(f.rawsize(0), len(fulltext0)) self.assertEqual(f.rawsize(0), len(fulltext0))
self.assertEqual(f.rawsize(1), len(fulltext1)) self.assertEqual(f.rawsize(1), len(fulltext1))
self.assertEqual(f.rawsize(2), len(fulltext2)) self.assertEqual(f.rawsize(2), len(fulltext2))
with self.assertRaises(IndexError): with self.assertRaises(IndexError):
f.rawsize(3) f.rawsize(3)
self.assertEqual(f.size(0), len(fulltext0)) self.assertEqual(f.size(0), len(fulltext0))