This is an archive of the discontinued Mercurial Phabricator instance.

Differential D9811

tagcache: distinguish between invalid and missing entries
ClosedPublic

Authored by mharbison72 on Jan 17 2021, 4:54 PM.

Download Raw Diff

Details

Reviewers

Alphare

Group Reviewers

hg-reviewers

Commits

rHG5aac1a1a5beb: tagcache: distinguish between invalid and missing entries
rHG7fbca460cbfb: tagcache: distinguish between invalid and missing entries

Summary

The TortoiseHg repo has typically not had a newly applied tag accessible by name
for recent releases, for unknown reasons. Deleting and rebuilding the tag cache
doesn't fix it, though deleting the cache and running hg log -r $new_tag does.
Eventually the situation does sort itself out for new clones from the server.
In an effort to figure out what the issue is, Pierre-Yves David suggested
listing these entries in the debug output more specifically.

This isn't complete yet- the second test change that says "missing" is more like
"invalid", since it was truncated. The problem there is the code that reads the
raw array truncates any partial records and then fills it with 0xFF, which
signifies that it is missing. As a side note, that means the check for the
length when validating an existing entry never fails.

Diff Detail

Repository

rHG Mercurial

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

mharbison72 created this revision.Jan 17 2021, 4:54 PM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptJan 17 2021, 4:54 PM

Herald added a subscriber: mercurial-patches. · View Herald Transcript

mharbison72 added a child revision: D9812: tests: demonstrate a case where a corrupt tag cache causes an abort.Jan 17 2021, 4:54 PM

I'm always uncomfortable with a function that returns a value, None and False... but I suppose that for performance reasons we can't really do better. Where are your Rust enums when you need them? ;)

mharbison72 added a commit: rHG7fbca460cbfb: tagcache: distinguish between invalid and missing entries.Feb 11 2021, 11:34 AM

This revision was not accepted when it landed; it landed in state Needs Review.

Closed by commit rHG7fbca460cbfb: tagcache: distinguish between invalid and missing entries (authored by mharbison72). · Explain Why

This revision was automatically updated to reflect the committed changes.

mharbison72 added a commit: rHG5aac1a1a5beb: tagcache: distinguish between invalid and missing entries.Feb 12 2021, 12:04 AM

pulkit removed a child revision: D9812: tests: demonstrate a case where a corrupt tag cache causes an abort.Feb 17 2021, 2:44 PM

Revision Contents
Changeset List

			Path	Packages
M			mercurial/bundle2.py (2 lines)
M			mercurial/debugcommands.py (8 lines)
M			mercurial/tags.py (8 lines)
M			tests/test-tags.t (36 lines)

Diff 25553

mercurial/bundle2.py

	# We don't bother using a generator to produce output data because			# We don't bother using a generator to produce output data because
	# a) we only have 40 bytes per head and even esoteric numbers of heads			# a) we only have 40 bytes per head and even esoteric numbers of heads
	# consume little memory (1M heads is 40MB) b) we don't want to send the			# consume little memory (1M heads is 40MB) b) we don't want to send the
	# part if we don't have entries and knowing if we have entries requires			# part if we don't have entries and knowing if we have entries requires
	# cache lookups.			# cache lookups.
	for node in outgoing.ancestorsof:			for node in outgoing.ancestorsof:
	# Don't compute missing, as this may slow down serving.			# Don't compute missing, as this may slow down serving.
	fnode = cache.getfnode(node, computemissing=False)			fnode = cache.getfnode(node, computemissing=False)
	if fnode is not None:			if fnode:
	chunks.extend([node, fnode])			chunks.extend([node, fnode])

	if chunks:			if chunks:
	bundler.newpart(b'hgtagsfnodes', data=b''.join(chunks))			bundler.newpart(b'hgtagsfnodes', data=b''.join(chunks))


	def addpartrevbranchcache(repo, bundler, outgoing):			def addpartrevbranchcache(repo, bundler, outgoing):
	# we include the rev branch cache for the bundle changeset			# we include the rev branch cache for the bundle changeset

mercurial/debugcommands.py


	@command(b'debugtagscache', [])			@command(b'debugtagscache', [])
	def debugtagscache(ui, repo):			def debugtagscache(ui, repo):
	"""display the contents of .hg/cache/hgtagsfnodes1"""			"""display the contents of .hg/cache/hgtagsfnodes1"""
	cache = tagsmod.hgtagsfnodescache(repo.unfiltered())			cache = tagsmod.hgtagsfnodescache(repo.unfiltered())
	for r in repo:			for r in repo:
	node = repo[r].node()			node = repo[r].node()
	tagsnode = cache.getfnode(node, computemissing=False)			tagsnode = cache.getfnode(node, computemissing=False)
	tagsnodedisplay = hex(tagsnode) if tagsnode else b'missing/invalid'			if tagsnode:
				tagsnodedisplay = hex(tagsnode)
				elif tagsnode is False:
				tagsnodedisplay = b'invalid'
				else:
				tagsnodedisplay = b'missing'

	ui.write(b'%d %s %s\n' % (r, hex(node), tagsnodedisplay))			ui.write(b'%d %s %s\n' % (r, hex(node), tagsnodedisplay))


	@command(			@command(
	b'debugtemplate',			b'debugtemplate',
	[			[
	(b'r', b'rev', [], _(b'apply template on changesets'), _(b'REV')),			(b'r', b'rev', [], _(b'apply template on changesets'), _(b'REV')),
	(b'D', b'define', [], _(b'define template keyword'), _(b'KEY=VALUE')),			(b'D', b'define', [], _(b'define template keyword'), _(b'KEY=VALUE')),

mercurial/tags.py

	# slightly less evil than copying a potentially large array slice.			# slightly less evil than copying a potentially large array slice.
	for i in range(rawlen - rawlentokeep):			for i in range(rawlen - rawlentokeep):
	self._raw.pop()			self._raw.pop()
	rawlen = len(self._raw)			rawlen = len(self._raw)
	self._dirtyoffset = rawlen			self._dirtyoffset = rawlen
	if rawlen < wantedlen:			if rawlen < wantedlen:
	if self._dirtyoffset is None:			if self._dirtyoffset is None:
	self._dirtyoffset = rawlen			self._dirtyoffset = rawlen
				# TODO: zero fill entire record, because it's invalid not missing?
	self._raw.extend(b'\xff' * (wantedlen - rawlen))			self._raw.extend(b'\xff' * (wantedlen - rawlen))

	def getfnode(self, node, computemissing=True):			def getfnode(self, node, computemissing=True):
	"""Obtain the filenode of the .hgtags file at a specified revision.			"""Obtain the filenode of the .hgtags file at a specified revision.

	If the value is in the cache, the entry will be validated and returned.			If the value is in the cache, the entry will be validated and returned.
	Otherwise, the filenode will be computed and returned unless			Otherwise, the filenode will be computed and returned unless
	"computemissing" is False, in which case None will be returned without			"computemissing" is False. In that case, None will be returned if
				the entry is missing or False if the entry is invalid without
	any potentially expensive computation being performed.			any potentially expensive computation being performed.

	If an .hgtags does not exist at the specified revision, nullid is			If an .hgtags does not exist at the specified revision, nullid is
	returned.			returned.
	"""			"""
	if node == nullid:			if node == nullid:
	return nullid			return nullid

	self.hitcount += 1			self.hitcount += 1
	return record[4:]			return record[4:]

	# Fall through.			# Fall through.

	# If we get here, the entry is either missing or invalid.			# If we get here, the entry is either missing or invalid.

	if not computemissing:			if not computemissing:
				if record != _fnodesmissingrec:
				return False
	return None			return None

	fnode = None			fnode = None
	cl = self._repo.changelog			cl = self._repo.changelog
	p1rev, p2rev = cl._uncheckedparentrevs(rev)			p1rev, p2rev = cl._uncheckedparentrevs(rev)
	p1node = cl.node(p1rev)			p1node = cl.node(p1rev)
	p1fnode = self.getfnode(p1node, computemissing=False)			p1fnode = self.getfnode(p1node, computemissing=False)
	if p2rev != nullrev:			if p2rev != nullrev:
	# There is some no-merge changeset where p1 is null and p2 is set			# There is some no-merge changeset where p1 is null and p2 is set
	# Processing them as merge is just slower, but still gives a good			# Processing them as merge is just slower, but still gives a good
	# result.			# result.
	p2node = cl.node(p1rev)			p2node = cl.node(p1rev)
	p2fnode = self.getfnode(p2node, computemissing=False)			p2fnode = self.getfnode(p2node, computemissing=False)
	if p1fnode != p2fnode:			if p1fnode != p2fnode:
	# we cannot rely on readfast because we don't know against what			# we cannot rely on readfast because we don't know against what
	# parent the readfast delta is computed			# parent the readfast delta is computed
	p1fnode = None			p1fnode = None
	if p1fnode is not None:			if p1fnode:
	mctx = ctx.manifestctx()			mctx = ctx.manifestctx()
	fnode = mctx.readfast().get(b'.hgtags')			fnode = mctx.readfast().get(b'.hgtags')
	if fnode is None:			if fnode is None:
	fnode = p1fnode			fnode = p1fnode
	if fnode is None:			if fnode is None:
	# Populate missing entry.			# Populate missing entry.
	try:			try:
	fnode = ctx.filenode(b'.hgtags')			fnode = ctx.filenode(b'.hgtags')

tests/test-tags.t

	$ fnodescacheexists			$ fnodescacheexists
	fnodes cache exists			fnodes cache exists
	$ f --size --hexdump .hg/cache/hgtagsfnodes1			$ f --size --hexdump .hg/cache/hgtagsfnodes1
	.hg/cache/hgtagsfnodes1: size=48			.hg/cache/hgtagsfnodes1: size=48
	0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff \|................\|			0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff \|................\|
	0010: ff ff ff ff ff ff ff ff b9 15 46 36 26 b7 b4 a7 \|..........F6&...\|			0010: ff ff ff ff ff ff ff ff b9 15 46 36 26 b7 b4 a7 \|..........F6&...\|
	0020: 73 e0 9e e3 c5 2f 51 0e 19 e0 5e 1f f9 66 d8 59 \|s..../Q...^..f.Y\|			0020: 73 e0 9e e3 c5 2f 51 0e 19 e0 5e 1f f9 66 d8 59 \|s..../Q...^..f.Y\|
	$ hg debugtagscache			$ hg debugtagscache
	0 acb14030fe0a21b60322c440ad2d20cf7685a376 missing/invalid			0 acb14030fe0a21b60322c440ad2d20cf7685a376 missing
	1 b9154636be938d3d431e75a7c906504a079bfe07 26b7b4a773e09ee3c52f510e19e05e1ff966d859			1 b9154636be938d3d431e75a7c906504a079bfe07 26b7b4a773e09ee3c52f510e19e05e1ff966d859

	Repeat with cold tag cache:			Repeat with cold tag cache:

	$ rm -f .hg/cache/tags2-visible .hg/cache/hgtagsfnodes1			$ rm -f .hg/cache/tags2-visible .hg/cache/hgtagsfnodes1
	$ hg identify			$ hg identify
	b9154636be93 tip			b9154636be93 tip

	$ rm -f .hg/cache/tags2-visible			$ rm -f .hg/cache/tags2-visible
	>>> import os			>>> import os
	>>> with open(".hg/cache/hgtagsfnodes1", "ab+") as fp:			>>> with open(".hg/cache/hgtagsfnodes1", "ab+") as fp:
	... fp.seek(-10, os.SEEK_END) and None			... fp.seek(-10, os.SEEK_END) and None
	... fp.truncate() and None			... fp.truncate() and None

	$ hg debugtagscache \| tail -2			$ hg debugtagscache \| tail -2
	4 0c192d7d5e6b78a714de54a2e9627952a877e25a 0c04f2a8af31de17fab7422878ee5a2dadbc943d			4 0c192d7d5e6b78a714de54a2e9627952a877e25a 0c04f2a8af31de17fab7422878ee5a2dadbc943d
	5 8dbfe60eff306a54259cfe007db9e330e7ecf866 missing/invalid			5 8dbfe60eff306a54259cfe007db9e330e7ecf866 missing
	$ hg tags			$ hg tags
	tip 5:8dbfe60eff30			tip 5:8dbfe60eff30
	bar 1:78391a272241			bar 1:78391a272241
	$ hg debugtagscache \| tail -2			$ hg debugtagscache \| tail -2
	4 0c192d7d5e6b78a714de54a2e9627952a877e25a 0c04f2a8af31de17fab7422878ee5a2dadbc943d			4 0c192d7d5e6b78a714de54a2e9627952a877e25a 0c04f2a8af31de17fab7422878ee5a2dadbc943d
	5 8dbfe60eff306a54259cfe007db9e330e7ecf866 0c04f2a8af31de17fab7422878ee5a2dadbc943d			5 8dbfe60eff306a54259cfe007db9e330e7ecf866 0c04f2a8af31de17fab7422878ee5a2dadbc943d

				If the 4 bytes of node hash for a record don't match an existing node, the entry
				is flagged as invalid.

				>>> import os
				>>> with open(".hg/cache/hgtagsfnodes1", "rb+") as fp:
				... fp.seek(-24, os.SEEK_END) and None
				... fp.write(b'\xde\xad') and None

				$ f --size --hexdump .hg/cache/hgtagsfnodes1
				.hg/cache/hgtagsfnodes1: size=144
				0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff \|................\|
				0010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff \|................\|
				0020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff \|................\|
				0030: 7a 94 12 77 0c 04 f2 a8 af 31 de 17 fa b7 42 28 \|z..w.....1....B(\|
				0040: 78 ee 5a 2d ad bc 94 3d 6f a4 50 21 7d 3b 71 8c \|x.Z-...=o.P!};q.\|
				0050: 96 4e f3 7b 89 e5 50 eb da fd 57 89 e7 6c e1 b0 \|.N.{..P...W..l..\|
				0060: 0c 19 2d 7d 0c 04 f2 a8 af 31 de 17 fa b7 42 28 \|..-}.....1....B(\|
				0070: 78 ee 5a 2d ad bc 94 3d de ad e6 0e 0c 04 f2 a8 \|x.Z-...=........\|
				0080: af 31 de 17 fa b7 42 28 78 ee 5a 2d ad bc 94 3d \|.1....B(x.Z-...=\|

				$ hg debugtagscache \| tail -2
				4 0c192d7d5e6b78a714de54a2e9627952a877e25a 0c04f2a8af31de17fab7422878ee5a2dadbc943d
				5 8dbfe60eff306a54259cfe007db9e330e7ecf866 invalid

				$ hg tags
				tip 5:8dbfe60eff30
				bar 1:78391a272241

	#if unix-permissions no-root			#if unix-permissions no-root
	Errors writing to .hgtags fnodes cache are silently ignored			Errors writing to .hgtags fnodes cache are silently ignored

	$ echo dummy2 > foo			$ echo dummy2 > foo
	$ hg commit -m throwaway2			$ hg commit -m throwaway2

	$ chmod a-w .hg/cache/hgtagsfnodes1			$ chmod a-w .hg/cache/hgtagsfnodes1
	$ rm -f .hg/cache/tags2-visible			$ rm -f .hg/cache/tags2-visible

	$ hg tags			$ hg tags
	tip 6:b968051b5cf3			tip 6:b968051b5cf3
	bar 1:78391a272241			bar 1:78391a272241

	$ hg blackbox -l 6			$ hg blackbox -l 6
	1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags			1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags
	1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> couldn't write cache/hgtagsfnodes1: [Errno ] (glob)			1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> couldn't write cache/hgtagsfnodes1: [Errno ] (glob)
	1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> 3/4 cache hits/lookups in * seconds (glob)			1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> 2/4 cache hits/lookups in * seconds (glob)
	1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing .hg/cache/tags2-visible with 1 tags			1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing .hg/cache/tags2-visible with 1 tags
	1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags exited 0 after * seconds (glob)			1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags exited 0 after * seconds (glob)
	1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> blackbox -l 6			1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> blackbox -l 6

	$ chmod a+w .hg/cache/hgtagsfnodes1			$ chmod a+w .hg/cache/hgtagsfnodes1

	$ rm -f .hg/cache/tags2-visible			$ rm -f .hg/cache/tags2-visible
	$ hg tags			$ hg tags
	tip 6:b968051b5cf3			tip 6:b968051b5cf3
	bar 1:78391a272241			bar 1:78391a272241

	$ hg blackbox -l 6			$ hg blackbox -l 6
	1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags			1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags
	1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing 24 bytes to cache/hgtagsfnodes1			1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing 24 bytes to cache/hgtagsfnodes1
	1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> 3/4 cache hits/lookups in * seconds (glob)			1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> 2/4 cache hits/lookups in * seconds (glob)
	1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing .hg/cache/tags2-visible with 1 tags			1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> writing .hg/cache/tags2-visible with 1 tags
	1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags exited 0 after * seconds (glob)			1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> tags exited 0 after * seconds (glob)
	1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> blackbox -l 6			1970/01/01 00:00:00 bob @b968051b5cf3f624b771779c6d5f84f1d4c3fb5d (5000)> blackbox -l 6

	$ f --size .hg/cache/hgtagsfnodes1			$ f --size .hg/cache/hgtagsfnodes1
	.hg/cache/hgtagsfnodes1: size=168			.hg/cache/hgtagsfnodes1: size=168

	$ hg -q --config extensions.strip= strip -r 6 --no-backup			$ hg -q --config extensions.strip= strip -r 6 --no-backup

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	25089		Jan 17 2021, 4:54 PM	★	★
Diff 2	25553	rHG7fbca460cbfbb861b7572b1650e9a6996864e4d3	Dec 24 2020, 11:21 AM	★	★