This is an archive of the discontinued Mercurial Phabricator instance.

Differential D11271

issue6528: add a config option to control the fixing on the fly
ClosedPublic

Authored by marmoute on Aug 7 2021, 12:58 PM.

Download Raw Diff

Details

Reviewers

pulkit

Group Reviewers

hg-reviewers

Commits

rHG2813d406b036: issue6528: add a config option to control the fixing on the fly

Summary

This will allow people who know to be safe to avoid any performance overhead
(and other potential issue).

Diff Detail

Repository

rHG Mercurial

Branch

stable

Lint

No Linters Available

Unit

No Unit Test Coverage

Event Timeline

marmoute created this revision.Aug 7 2021, 12:58 PM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptAug 7 2021, 12:58 PM

Herald added a subscriber: mercurial-patches. · View Herald Transcript

✅ refresh by Heptapod after a successful CI run (🐙 💚)
⚠ This patch is intended for stable ⚠

Note to other reviewers: I'm on vacation so I probably won't be queuing patches too much, I wanted to check up on the rest of my series.

tests/test-issue6528.t
593	Those comments say the opposite of what they're trying to

marmoute planned changes to this revision.Aug 8 2021, 4:08 PM

marmoute added inline comments.

tests/test-issue6528.t
593	The copy-paste devil strike again. Thanks for spotting this, update on the way.

✅ refresh by Heptapod after a successful CI run (🐙 💚)
⚠ This patch is intended for stable ⚠

pulkit accepted this revision.Aug 10 2021, 8:19 AM

This revision is now accepted and ready to land.Aug 10 2021, 8:19 AM

Can we add details to releasenotes about debugcommand and config option introduced for the bug fix?

marmoute added a commit: rHG2813d406b036: issue6528: add a config option to control the fixing on the fly.Aug 10 2021, 9:33 AM

Closed by commit rHG2813d406b036: issue6528: add a config option to control the fixing on the fly (authored by marmoute). · Explain Why

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

			Path	Packages
M			mercurial/configitems.py (5 lines)
M			mercurial/filelog.py (5 lines)
M			mercurial/helptext/config.txt (15 lines)
M			mercurial/localrepo.py (3 lines)
M			tests/test-issue6528.t (95 lines)

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	29850		Aug 7 2021, 12:58 PM	★	★
Diff 2	29862		Aug 8 2021, 5:22 AM	★	★
Diff 3	29864		Aug 8 2021, 6:59 PM	★	★
Diff 4	29883	rHG2813d406b03607cdb8c06cb04c44efcc9a79d9a2	Aug 7 2021, 10:51 AM	★	★

Commit	Parents	Author	Summary	Date
351b80852e02	f41e63391d70	Pierre-Yves David		Aug 7 2021, 10:51 AM

Status	Author	Revision
Closed	marmoute	D11271 issue6528: add a config option to control the fixing on the fly
Closed	marmoute	D11270 issue6528: also filter delta on the fly when applying a changegroup
Closed	marmoute	D11269 filelog: open the writing context a bit earlier in `addgroup`
Closed	marmoute	D11268 issue6528: implement _is_revision_affected_fast using callback
Closed	marmoute	D11267 issue6528: implement _is_revision_affected using callback
Closed	Alphare	D11263 debugcommands: add a `--paranoid` option to `debug-repair-issue-6528`
Closed	Alphare	D11262 repair: improve performance of detection of revisions affected by issue6528
Closed	Alphare	D11239 debugcommands: introduce a debug command to repair repos affected by issue6528

Diff 29850

mercurial/configitems.py

	experimental=True,			experimental=True,
	)			)
	coreconfigitem(			coreconfigitem(
	b'storage',			b'storage',
	b'revlog.optimize-delta-parent-choice',			b'revlog.optimize-delta-parent-choice',
	default=True,			default=True,
	alias=[(b'format', b'aggressivemergedeltas')],			alias=[(b'format', b'aggressivemergedeltas')],
	)			)
				coreconfigitem(
				b'storage',
				b'revlog.issue6528.fix-incoming',
				default=True,
				)
	# experimental as long as rust is experimental (or a C version is implemented)			# experimental as long as rust is experimental (or a C version is implemented)
	coreconfigitem(			coreconfigitem(
	b'storage',			b'storage',
	b'revlog.persistent-nodemap.mmap',			b'revlog.persistent-nodemap.mmap',
	default=True,			default=True,
	)			)
	# experimental as long as format.use-persistent-nodemap is.			# experimental as long as format.use-persistent-nodemap is.
	coreconfigitem(			coreconfigitem(

mercurial/filelog.py

	target=(revlog_constants.KIND_FILELOG, path),			target=(revlog_constants.KIND_FILELOG, path),
	radix=b'/'.join((b'data', path)),			radix=b'/'.join((b'data', path)),
	censorable=True,			censorable=True,
	)			)
	# Full name of the user visible file, relative to the repository root.			# Full name of the user visible file, relative to the repository root.
	# Used by LFS.			# Used by LFS.
	self._revlog.filename = path			self._revlog.filename = path
	self.nullid = self._revlog.nullid			self.nullid = self._revlog.nullid
				opts = opener.options
				self._fix_issue6528 = opts.get(b'issue6528.fix-incoming', True)

	def __len__(self):			def __len__(self):
	return len(self._revlog)			return len(self._revlog)

	def __iter__(self):			def __iter__(self):
	return self._revlog.__iter__()			return self._revlog.__iter__()

	def hasnode(self, node):			def hasnode(self, node):
	_(			_(
	b'revlog storage does not support missing '			b'revlog storage does not support missing '
	b'parents write mode'			b'parents write mode'
	)			)
	)			)

	with self._revlog._writing(transaction):			with self._revlog._writing(transaction):

				if self._fix_issue6528:
	deltas = rewrite.filter_delta_issue6528(self._revlog, deltas)			deltas = rewrite.filter_delta_issue6528(self._revlog, deltas)

	return self._revlog.addgroup(			return self._revlog.addgroup(
	deltas,			deltas,
	linkmapper,			linkmapper,
	transaction,			transaction,
	addrevisioncb=addrevisioncb,			addrevisioncb=addrevisioncb,
	duplicaterevisioncb=duplicaterevisioncb,			duplicaterevisioncb=duplicaterevisioncb,
	)			)

mercurial/helptext/config.txt

	(default: True)			(default: True)

	``storage``			``storage``
	-----------			-----------

	Control the strategy Mercurial uses internally to store history. Options in this			Control the strategy Mercurial uses internally to store history. Options in this
	category impact performance and repository size.			category impact performance and repository size.

				``revlog.issue6528.fix-incoming``
				Version 5.8 of Mercurial had a bug leading to altering the parent of file
				revision with copy information (or any other metadata) on exchange. This
				leads to the copy metadata to be overlooked by various internal logic. The
				issue was fixed in Mercurial 5.8.1.
				(See https://bz.mercurial-scm.org/show_bug.cgi?id=6528 for details)

				As a result Mercurial is now checking and fixing incoming file revisions to
				make sure there parents are in the right order. This behavior can be
				disabled by setting this option to `no`. This apply to revisions added
				through push, pull, clone and unbundle.

				To fix affected revisions that already exist within the repository, one can
				use :hg:`debug-repair-issue-6528`.

	``revlog.optimize-delta-parent-choice``			``revlog.optimize-delta-parent-choice``
	When storing a merge revision, both parents will be equally considered as			When storing a merge revision, both parents will be equally considered as
	a possible delta base. This results in better delta selection and improved			a possible delta base. This results in better delta selection and improved
	revlog compression. This option is enabled by default.			revlog compression. This option is enabled by default.

	Turning this option off can result in large increase of repository size for			Turning this option off can result in large increase of repository size for
	repository with many merges.			repository with many merges.

mercurial/localrepo.py

	if chunkcachesize is not None:			if chunkcachesize is not None:
	options[b'chunkcachesize'] = chunkcachesize			options[b'chunkcachesize'] = chunkcachesize

	deltabothparents = ui.configbool(			deltabothparents = ui.configbool(
	b'storage', b'revlog.optimize-delta-parent-choice'			b'storage', b'revlog.optimize-delta-parent-choice'
	)			)
	options[b'deltabothparents'] = deltabothparents			options[b'deltabothparents'] = deltabothparents

				issue6528 = ui.configbool(b'storage', b'revlog.issue6528.fix-incoming')
				options[b'issue6528.fix-incoming'] = issue6528

	lazydelta = ui.configbool(b'storage', b'revlog.reuse-external-delta')			lazydelta = ui.configbool(b'storage', b'revlog.reuse-external-delta')
	lazydeltabase = False			lazydeltabase = False
	if lazydelta:			if lazydelta:
	lazydeltabase = ui.configbool(			lazydeltabase = ui.configbool(
	b'storage', b'revlog.reuse-external-delta-parent'			b'storage', b'revlog.reuse-external-delta-parent'
	)			)
	if lazydeltabase is None:			if lazydeltabase is None:
	lazydeltabase = not scmutil.gddeltaconfig(ui)			lazydeltabase = not scmutil.gddeltaconfig(ui)

tests/test-issue6528.t

	2 files updated, 0 files merged, 0 files removed, 0 files unresolved			2 files updated, 0 files merged, 0 files removed, 0 files unresolved

	And that the repair command does not find anything to fix			And that the repair command does not find anything to fix

	$ hg debug-repair-issue6528			$ hg debug-repair-issue6528
	no affected revisions were found			no affected revisions were found

	$ cd ..			$ cd ..

				A config option can disable the fixing of the bad bundle on the fly
				-------------------------------------------------------------------



				from a v1 bundle
				~~~~~~~~~~~~~~~~

				$ hg debugbundle --spec "$TESTDIR"/bundles/issue6528.hg-v1
				bzip2-v1

				$ hg init unbundle-v1-no-fix
				$ cd unbundle-v1-no-fix

				$ hg unbundle "$TESTDIR"/bundles/issue6528.hg-v1 --config storage.revlog.issue6528.fix-incoming=no
				adding changesets
				adding manifests
				adding file changes
				added 8 changesets with 12 changes to 4 files
				new changesets f5a5a568022f:3beabb508514 (8 drafts)
				(run 'hg update' to get a working copy)

				Check that revision were fixed on the fly

				$ hg debugrevlogindex b.txt
				rev linkrev nodeid p1 p2
				0 2 05b806ebe5ea 000000000000 000000000000
				1 3 a58b36ad6b65 05b806ebe5ea 000000000000
				2 6 216a5fe8b8ed 000000000000 000000000000
				3 7 ea4f2f2463cc 216a5fe8b8ed 000000000000

				$ hg debugrevlogindex D.txt
				rev linkrev nodeid p1 p2
				0 6 2a8d3833f2fb 000000000000 000000000000
				1 7 2a80419dfc31 2a8d3833f2fb 000000000000

				That we don't see the symptoms of the bug

				$ hg up -- -1
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved

				And that the repair command does not find anything to fix

				$ hg debug-repair-issue6528 --dry-run
				found affected revision 1 for filelog 'data/D.txt.i'
				found affected revision 1 for filelog 'data/b.txt.i'
				found affected revision 3 for filelog 'data/b.txt.i'

				$ cd ..

				from a v2 bundle
				~~~~~~~~~~~~~~~~

				$ hg debugbundle --spec "$TESTDIR"/bundles/issue6528.hg-v2
				bzip2-v2

				$ hg init unbundle-v2-no-fix
				$ cd unbundle-v2-no-fix

				$ hg unbundle "$TESTDIR"/bundles/issue6528.hg-v2 --config storage.revlog.issue6528.fix-incoming=no
				adding changesets
				adding manifests
				adding file changes
				added 8 changesets with 12 changes to 4 files
				new changesets f5a5a568022f:3beabb508514 (8 drafts)
				(run 'hg update' to get a working copy)

				Check that revision were fixed on the fly
				AlphareUnsubmitted Not Done Those comments say the opposite of what they're trying to Alphare: Those comments say the opposite of what they're trying to
				marmouteAuthorUnsubmitted Done The copy-paste devil strike again. Thanks for spotting this, update on the way. marmoute: The copy-paste devil strike again. Thanks for spotting this, update on the way.

				$ hg debugrevlogindex b.txt
				rev linkrev nodeid p1 p2
				0 2 05b806ebe5ea 000000000000 000000000000
				1 3 a58b36ad6b65 05b806ebe5ea 000000000000
				2 6 216a5fe8b8ed 000000000000 000000000000
				3 7 ea4f2f2463cc 216a5fe8b8ed 000000000000

				$ hg debugrevlogindex D.txt
				rev linkrev nodeid p1 p2
				0 6 2a8d3833f2fb 000000000000 000000000000
				1 7 2a80419dfc31 2a8d3833f2fb 000000000000

				That we don't see the symptoms of the bug

				$ hg up -- -1
				2 files updated, 0 files merged, 0 files removed, 0 files unresolved

				And that the repair command does not find anything to fix

				$ hg debug-repair-issue6528 --dry-run
				found affected revision 1 for filelog 'data/D.txt.i'
				found affected revision 1 for filelog 'data/b.txt.i'
				found affected revision 3 for filelog 'data/b.txt.i'

				$ cd ..