This is an archive of the discontinued Mercurial Phabricator instance.

Differential D9811

tagcache: distinguish between invalid and missing entries
ClosedPublic

Authored by mharbison72 on Jan 17 2021, 4:54 PM.

Details

Reviewers
Alphare
Group Reviewers
hg-reviewers
Commits
rHG5aac1a1a5beb: tagcache: distinguish between invalid and missing entries
rHG7fbca460cbfb: tagcache: distinguish between invalid and missing entries
Summary

The TortoiseHg repo has typically not had a newly applied tag accessible by name
for recent releases, for unknown reasons. Deleting and rebuilding the tag cache
doesn't fix it, though deleting the cache and running hg log -r $new_tag does.
Eventually the situation does sort itself out for new clones from the server.
In an effort to figure out what the issue is, Pierre-Yves David suggested
listing these entries in the debug output more specifically.

This isn't complete yet- the second test change that says "missing" is more like
"invalid", since it was truncated. The problem there is the code that reads the
raw array truncates any partial records and then fills it with 0xFF, which
signifies that it is missing. As a side note, that means the check for the
length when validating an existing entry never fails.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

mharbison72 created this revision.Jan 17 2021, 4:54 PM
Alphare accepted this revision.Jan 22 2021, 5:25 AM
Alphare added a subscriber: Alphare.

I'm always uncomfortable with a function that returns a value, None and False... but I suppose that for performance reasons we can't really do better. Where are your Rust enums when you need them? ;)

mharbison72 added a commit: rHG7fbca460cbfb: tagcache: distinguish between invalid and missing entries.Feb 11 2021, 11:34 AM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG7fbca460cbfb: tagcache: distinguish between invalid and missing entries (authored by mharbison72). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/bundle2.py (2 lines)
Mmercurial/debugcommands.py (8 lines)
Mmercurial/tags.py (8 lines)
Mtests/test-tags.t (36 lines)

Diff 25553

mercurial/bundle2.py

# We don't bother using a generator to produce output data because # We don't bother using a generator to produce output data because
# a) we only have 40 bytes per head and even esoteric numbers of heads # a) we only have 40 bytes per head and even esoteric numbers of heads
# consume little memory (1M heads is 40MB) b) we don't want to send the # consume little memory (1M heads is 40MB) b) we don't want to send the
# part if we don't have entries and knowing if we have entries requires # part if we don't have entries and knowing if we have entries requires
# cache lookups. # cache lookups.
for node in outgoing.ancestorsof: for node in outgoing.ancestorsof:
# Don't compute missing, as this may slow down serving. # Don't compute missing, as this may slow down serving.
fnode = cache.getfnode(node, computemissing=False) fnode = cache.getfnode(node, computemissing=False)
if fnode is not None: if fnode:
chunks.extend([node, fnode]) chunks.extend([node, fnode])
if chunks: if chunks:
bundler.newpart(b'hgtagsfnodes', data=b''.join(chunks)) bundler.newpart(b'hgtagsfnodes', data=b''.join(chunks))
def addpartrevbranchcache(repo, bundler, outgoing): def addpartrevbranchcache(repo, bundler, outgoing):
# we include the rev branch cache for the bundle changeset # we include the rev branch cache for the bundle changeset

mercurial/debugcommands.py

@command(b'debugtagscache', []) @command(b'debugtagscache', [])
def debugtagscache(ui, repo): def debugtagscache(ui, repo):
"""display the contents of .hg/cache/hgtagsfnodes1""" """display the contents of .hg/cache/hgtagsfnodes1"""
cache = tagsmod.hgtagsfnodescache(repo.unfiltered()) cache = tagsmod.hgtagsfnodescache(repo.unfiltered())
for r in repo: for r in repo:
node = repo[r].node() node = repo[r].node()
tagsnode = cache.getfnode(node, computemissing=False) tagsnode = cache.getfnode(node, computemissing=False)
tagsnodedisplay = hex(tagsnode) if tagsnode else b'missing/invalid' if tagsnode:
tagsnodedisplay = hex(tagsnode)
elif tagsnode is False:
tagsnodedisplay = b'invalid'
else:
tagsnodedisplay = b'missing'
ui.write(b'%d %s %s\n' % (r, hex(node), tagsnodedisplay)) ui.write(b'%d %s %s\n' % (r, hex(node), tagsnodedisplay))
@command( @command(
b'debugtemplate', b'debugtemplate',
[ [
(b'r', b'rev', [], _(b'apply template on changesets'), _(b'REV')), (b'r', b'rev', [], _(b'apply template on changesets'), _(b'REV')),
(b'D', b'define', [], _(b'define template keyword'), _(b'KEY=VALUE')), (b'D', b'define', [], _(b'define template keyword'), _(b'KEY=VALUE')),

mercurial/tags.py

# slightly less evil than copying a potentially large array slice. # slightly less evil than copying a potentially large array slice.
for i in range(rawlen - rawlentokeep): for i in range(rawlen - rawlentokeep):
self._raw.pop() self._raw.pop()
rawlen = len(self._raw) rawlen = len(self._raw)
self._dirtyoffset = rawlen self._dirtyoffset = rawlen
if rawlen < wantedlen: if rawlen < wantedlen:
if self._dirtyoffset is None: if self._dirtyoffset is None:
self._dirtyoffset = rawlen self._dirtyoffset = rawlen
# TODO: zero fill entire record, because it's invalid not missing?
self._raw.extend(b'\xff' * (wantedlen - rawlen)) self._raw.extend(b'\xff' * (wantedlen - rawlen))
def getfnode(self, node, computemissing=True): def getfnode(self, node, computemissing=True):
"""Obtain the filenode of the .hgtags file at a specified revision. """Obtain the filenode of the .hgtags file at a specified revision.
If the value is in the cache, the entry will be validated and returned. If the value is in the cache, the entry will be validated and returned.
Otherwise, the filenode will be computed and returned unless Otherwise, the filenode will be computed and returned unless
"computemissing" is False, in which case None will be returned without "computemissing" is False. In that case, None will be returned if
the entry is missing or False if the entry is invalid without
any potentially expensive computation being performed. any potentially expensive computation being performed.
If an .hgtags does not exist at the specified revision, nullid is If an .hgtags does not exist at the specified revision, nullid is
returned. returned.
""" """
if node == nullid: if node == nullid:
return nullid return nullid
self.hitcount += 1 self.hitcount += 1
return record[4:] return record[4:]
# Fall through. # Fall through.
# If we get here, the entry is either missing or invalid. # If we get here, the entry is either missing or invalid.
if not computemissing: if not computemissing:
if record != _fnodesmissingrec:
return False
return None return None
fnode = None fnode = None
cl = self._repo.changelog cl = self._repo.changelog
p1rev, p2rev = cl._uncheckedparentrevs(rev) p1rev, p2rev = cl._uncheckedparentrevs(rev)
p1node = cl.node(p1rev) p1node = cl.node(p1rev)
p1fnode = self.getfnode(p1node, computemissing=False) p1fnode = self.getfnode(p1node, computemissing=False)
if p2rev != nullrev: if p2rev != nullrev:
# There is some no-merge changeset where p1 is null and p2 is set # There is some no-merge changeset where p1 is null and p2 is set
# Processing them as merge is just slower, but still gives a good # Processing them as merge is just slower, but still gives a good
# result. # result.
p2node = cl.node(p1rev) p2node = cl.node(p1rev)
p2fnode = self.getfnode(p2node, computemissing=False) p2fnode = self.getfnode(p2node, computemissing=False)
if p1fnode != p2fnode: if p1fnode != p2fnode:
# we cannot rely on readfast because we don't know against what # we cannot rely on readfast because we don't know against what
# parent the readfast delta is computed # parent the readfast delta is computed
p1fnode = None p1fnode = None
if p1fnode is not None: if p1fnode:
mctx = ctx.manifestctx() mctx = ctx.manifestctx()
fnode = mctx.readfast().get(b'.hgtags') fnode = mctx.readfast().get(b'.hgtags')
if fnode is None: if fnode is None:
fnode = p1fnode fnode = p1fnode
if fnode is None: if fnode is None:
# Populate missing entry. # Populate missing entry.
try: try:
fnode = ctx.filenode(b'.hgtags') fnode = ctx.filenode(b'.hgtags')

tests/test-tags.t

$ fnodescacheexists $ fnodescacheexists
fnodes cache exists fnodes cache exists
$ f --size --hexdump .hg/cache/hgtagsfnodes1 $ f --size --hexdump .hg/cache/hgtagsfnodes1
.hg/cache/hgtagsfnodes1: size=48 .hg/cache/hgtagsfnodes1: size=48
0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................| 0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0010: ff ff ff ff ff ff ff ff b9 15 46 36 26 b7 b4 a7 |..........F6&...| 0010: ff ff ff ff ff ff ff ff b9 15 46 36 26 b7 b4 a7 |..........F6&...|
0020: 73 e0 9e e3 c5 2f 51 0e 19 e0 5e 1f f9 66 d8 59 |s..../Q...^..f.Y| 0020: 73 e0 9e e3 c5 2f 51 0e 19 e0 5e 1f f9 66 d8 59 |s..../Q...^..f.Y|
$ hg debugtagscache $ hg debugtagscache
0 acb14030fe0a21b60322c440ad2d20cf7685a376 missing/invalid 0 acb14030fe0a21b60322c440ad2d20cf7685a376 missing
1 b9154636be938d3d431e75a7c906504a079bfe07 26b7b4a773e09ee3c52f510e19e05e1ff966d859 1 b9154636be938d3d431e75a7c906504a079bfe07 26b7b4a773e09ee3c52f510e19e05e1ff966d859
Repeat with cold tag cache: Repeat with cold tag cache:
$ rm -f .hg/cache/tags2-visible .hg/cache/hgtagsfnodes1 $ rm -f .hg/cache/tags2-visible .hg/cache/hgtagsfnodes1
$ hg identify $ hg identify
b9154636be93 tip b9154636be93 tip
$ rm -f .hg/cache/tags2-visible $ rm -f .hg/cache/tags2-visible
>>> import os >>> import os
>>> with open(".hg/cache/hgtagsfnodes1", "ab+") as fp: >>> with open(".hg/cache/hgtagsfnodes1", "ab+") as fp:
... fp.seek(-10, os.SEEK_END) and None ... fp.seek(-10, os.SEEK_END) and None
... fp.truncate() and None ... fp.truncate() and None
$ hg debugtagscache | tail -2 $ hg debugtagscache | tail -2
4 0c192d7d5e6b78a714de54a2e9627952a877e25a 0c04f2a8af31de17fab7422878ee5a2dadbc943d 4 0c192d7d5e6b78a714de54a2e9627952a877e25a 0c04f2a8af31de17fab7422878ee5a2dadbc943d
5 8dbfe60eff306a54259cfe007db9e330e7ecf866 missing/invalid 5 8dbfe60eff306a54259cfe007db9e330e7ecf866 missing
$ hg tags $ hg tags
tip 5:8dbfe60eff30 tip 5:8dbfe60eff30
bar 1:78391a272241 bar 1:78391a272241
$ hg debugtagscache | tail -2 $ hg debugtagscache | tail -2
4 0c192d7d5e6b78a714de54a2e9627952a877e25a 0c04f2a8af31de17fab7422878ee5a2dadbc943d 4 0c192d7d5e6b78a714de54a2e9627952a877e25a 0c04f2a8af31de17fab7422878ee5a2dadbc943d
5 8dbfe60eff306a54259cfe007db9e330e7ecf866 0c04f2a8af31de17fab7422878ee5a2dadbc943d 5 8dbfe60eff306a54259cfe007db9e330e7ecf866 0c04f2a8af31de17fab7422878ee5a2dadbc943d
If the 4 bytes of node hash for a record don't match an existing node, the entry
is flagged as invalid.
>>> import os
>>> with open(".hg/cache/hgtagsfnodes1", "rb+") as fp:
... fp.seek(-24, os.SEEK_END) and None
... fp.write(b'\xde\xad') and None
$ f --size --hexdump .hg/cache/hgtagsfnodes1
.hg/cache/hgtagsfnodes1: size=144
0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
0030: 7a 94 12 77 0c 04 f2 a8 af 31 de 17 fa b7 42 28 |z..w.....1....B(|
0040: 78 ee 5a 2d ad bc 94 3d 6f a4 50 21 7d 3b 71 8c |x.Z-...=o.P!};q.|
0050: 96 4e f3 7b 89 e5 50 eb da fd 57 89 e7 6c e1 b0 |.N.{..P...W..l..|
0060: 0c 19 2d 7d 0c 04 f2 a8 af 31 de 17 fa b7 42 28 |..-}.....1....B(|
0070: 78 ee 5a 2d ad bc 94 3d de ad e6 0e 0c 04 f2 a8 |x.Z-...=........|
0080: af 31 de 17 fa b7 42 28 78 ee 5a 2d ad bc 94 3d |.1....B(x.Z-...=|
$ hg debugtagscache | tail -2
4 0c192d7d5e6b78a714de54a2e9627952a877e25a 0c04f2a8af31de17fab7422878ee5a2dadbc943d
5 8dbfe60eff306a54259cfe007db9e330e7ecf866 invalid
$ hg tags
tip 5:8dbfe60eff30
bar 1:78391a272241
#if unix-permissions no-root #if unix-permissions no-root
Errors writing to .hgtags fnodes cache are silently ignored Errors writing to .hgtags fnodes cache are silently ignored
$ echo dummy2 > foo $ echo dummy2 > foo
$ hg commit -m throwaway2 $ hg commit -m throwaway2
$ chmod a-w .hg/cache/hgtagsfnodes1 $ chmod a-w .hg/cache/hgtagsfnodes1
$ rm -f .hg/cache/tags2-visible $ rm -f .hg/cache/tags2-visible
$ hg tags $ hg tags
tip 6:b968051b5cf3 tip 6:b968051b5cf3
bar 1:78391a272241 bar 1:78391a272241
$ hg blackbox -l 6 $ hg blackbox -l 6
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> couldn't write cache/hgtagsfnodes1: [Errno *] * (glob) 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> couldn't write cache/hgtagsfnodes1: [Errno *] * (glob)
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> 3/4 cache hits/lookups in * seconds (glob) 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> 2/4 cache hits/lookups in * seconds (glob)
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing .hg/cache/tags2-visible with 1 tags 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing .hg/cache/tags2-visible with 1 tags
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags exited 0 after * seconds (glob) 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags exited 0 after * seconds (glob)
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> blackbox -l 6 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> blackbox -l 6
$ chmod a+w .hg/cache/hgtagsfnodes1 $ chmod a+w .hg/cache/hgtagsfnodes1
$ rm -f .hg/cache/tags2-visible $ rm -f .hg/cache/tags2-visible
$ hg tags $ hg tags
tip 6:b968051b5cf3 tip 6:b968051b5cf3
bar 1:78391a272241 bar 1:78391a272241
$ hg blackbox -l 6 $ hg blackbox -l 6
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing 24 bytes to cache/hgtagsfnodes1 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing 24 bytes to cache/hgtagsfnodes1
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> 3/4 cache hits/lookups in * seconds (glob) 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> 2/4 cache hits/lookups in * seconds (glob)
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing .hg/cache/tags2-visible with 1 tags 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing .hg/cache/tags2-visible with 1 tags
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags exited 0 after * seconds (glob) 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags exited 0 after * seconds (glob)
1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> blackbox -l 6 1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> blackbox -l 6
$ f --size .hg/cache/hgtagsfnodes1 $ f --size .hg/cache/hgtagsfnodes1
.hg/cache/hgtagsfnodes1: size=168 .hg/cache/hgtagsfnodes1: size=168
$ hg -q --config extensions.strip= strip -r 6 --no-backup $ hg -q --config extensions.strip= strip -r 6 --no-backup