This is an archive of the discontinued Mercurial Phabricator instance.

Differential D9487

upgrade: start moving the "to be happening" data in a dedicated object
ClosedPublic

Authored by marmoute on Dec 1 2020, 5:12 PM.

Download Raw Diff

Details

Reviewers

pulkit

Group Reviewers

hg-reviewers

Commits

rHGc407513a44a3: upgrade: start moving the "to be happening" data in a dedicated object
rHG74923cb88537: upgrade: start moving the "to be happening" data in a dedicated object

Summary

The upgrade code has a lot of logic to determine which action needs to be
performed depending of various element (sometimes depending from each other). It
would be nice to have a consistent object representing this. That could be
cleanly passed and avoid some logic duplication.

So we create this object as a start.

Diff Detail

Repository

rHG Mercurial

Branch

default

Lint

No Linters Available

Unit

No Unit Test Coverage

Event Timeline

marmoute created this revision.Dec 1 2020, 5:12 PM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptDec 1 2020, 5:12 PM

Herald added a subscriber: mercurial-patches. · View Herald Transcript

baymax updated this revision to Diff 24038.Dec 4 2020, 6:48 PM

pulkit requested changes to this revision.Dec 6 2020, 6:27 AM

pulkit added a subscriber: pulkit.

pulkit added inline comments.

mercurial/upgrade.py
184	Seems like incomplete statement.

This revision now requires changes to proceed.Dec 6 2020, 6:27 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

pulkit accepted this revision.Dec 7 2020, 3:48 AM

This revision is now accepted and ready to land.Dec 7 2020, 3:48 AM

marmoute added a commit: rHG74923cb88537: upgrade: start moving the "to be happening" data in a dedicated object.Dec 7 2020, 4:36 AM

Closed by commit rHG74923cb88537: upgrade: start moving the "to be happening" data in a dedicated object (authored by marmoute). · Explain Why

This revision was automatically updated to reflect the committed changes.

marmoute added a commit: rHGc407513a44a3: upgrade: start moving the "to be happening" data in a dedicated object.Dec 7 2020, 1:19 PM

Revision Contents
Changeset List

			Path	Packages
M			mercurial/upgrade.py (11 lines)
M			mercurial/upgrade_utils/actions.py (9 lines)
M			mercurial/upgrade_utils/engine.py (22 lines)

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	23915		Dec 1 2020, 5:12 PM	★	★
Diff 2	24038		Dec 4 2020, 6:48 PM	★	★
Diff 3	24071		Dec 6 2020, 10:09 AM	★	★
Diff 4	24074	rHG74923cb88537fafa6a8165500e18ab74abd5946b	Dec 1 2020, 4:37 PM	★	★

Commit	Parents	Author	Summary	Date
f7cec7b0260c	e70128bd33ab	Pierre-Yves David		Dec 1 2020, 4:37 PM

Status	Author	Revision
Closed	marmoute	D9475 upgrade: move optimisation to something more declarative
Closed	marmoute	D9474 upgrade: capitalize the `deficiency` constant
Closed	marmoute	D9473 upgrade: capitalize the `deficiency` constant
Closed	Alphare	D9421 copies-rust: extract the processing of a ChangedFiles in its own function
Closed	Alphare	D9420 copies-rust: move the parent token to an enum
Closed	indygreg	D9281 cext: add .pyi files for C extensions
Closed	mharbison72	D9356 phabricator: allow local revisions to be specified with `phabupdate`
Closed	Alphare	D9419 copies: properly copies parent dictionary before updating it
Closed	marmoute	D9487 upgrade: start moving the "to be happening" data in a dedicated object
Closed	marmoute	D9486 upgrade: gather code about requirement checking together
Closed	marmoute	D9485 upgrade: extract the checking of target requirements change
Closed	marmoute	D9484 upgrade: drop an outdated comment
Closed	marmoute	D9483 upgrade: gather code about source checking together
Closed	marmoute	D9482 upgrade: move requirements checking in a dedicated function
Closed	marmoute	D9477 upgrade: split definition and management of the actions from the main code
Closed	marmoute	D9476 upgrade: split actual upgrade code away from the main module
Closed	marmoute	D9469 upgrade: display the list of processed revlog before proceeding
Closed	SimonSapin	D9519 rhg: add a test with persistent-nodemap
Closed	SimonSapin	D9490 rust: use NodePrefix::from_hex instead of hex::decode directly
Closed	SimonSapin	D9479 rhg: allow specifying a changeset ID prefix
Closed	martinvonz	D9514 tests: update test-releasenotes-formatting.t with new exit codes

Diff 23915

mercurial/upgrade.py

	if not revlogs:			if not revlogs:
	ui.write((b'no revlogs to process\n'))			ui.write((b'no revlogs to process\n'))
	else:			else:
	ui.write((b'processed revlogs:\n'))			ui.write((b'processed revlogs:\n'))
	for r in sorted(revlogs):			for r in sorted(revlogs):
	ui.write((b' - %s\n' % r))			ui.write((b' - %s\n' % r))
	ui.write((b'\n'))			ui.write((b'\n'))

				upgrade_op = upgrade_actions.UpgradeOperation(
				newreqs,
				[a.name for a in actions],
				revlogs,
				)
				upgrade_op
				pulkitUnsubmitted Not Done Seems like incomplete statement. pulkit: Seems like incomplete statement.

	if not run:			if not run:
	fromconfig = []			fromconfig = []
	onlydefault = []			onlydefault = []

	for d in deficiencies:			for d in deficiencies:
	if d.fromconfig(repo):			if d.fromconfig(repo):
	fromconfig.append(d)			fromconfig.append(d)
	elif d.default:			elif d.default:

	# Else we're in the run=true case.			# Else we're in the run=true case.
	ui.write(_(b'upgrade will perform the following actions:\n\n'))			ui.write(_(b'upgrade will perform the following actions:\n\n'))
	printrequirements()			printrequirements()
	printoptimisations()			printoptimisations()
	printupgradeactions()			printupgradeactions()
	print_affected_revlogs()			print_affected_revlogs()

	upgradeactions = [a.name for a in actions]

	ui.status(_(b'beginning upgrade...\n'))			ui.status(_(b'beginning upgrade...\n'))
	with repo.wlock(), repo.lock():			with repo.wlock(), repo.lock():
	ui.status(_(b'repository locked and read-only\n'))			ui.status(_(b'repository locked and read-only\n'))
	# Our strategy for upgrading the repository is to create a new,			# Our strategy for upgrading the repository is to create a new,
	# temporary repository, write data to it, then do a swap of the			# temporary repository, write data to it, then do a swap of the
	# data. There are less heavyweight ways to do this, but it is easier			# data. There are less heavyweight ways to do this, but it is easier
	# to create a new repo object than to instantiate all the components			# to create a new repo object than to instantiate all the components
	# (like the store) separately.			# (like the store) separately.
	)			)

	# clone ui without using ui.copy because repo.ui is protected			# clone ui without using ui.copy because repo.ui is protected
	repoui = repo.ui.__class__(repo.ui)			repoui = repo.ui.__class__(repo.ui)
	dstrepo = hg.repository(repoui, path=tmppath, create=True)			dstrepo = hg.repository(repoui, path=tmppath, create=True)

	with dstrepo.wlock(), dstrepo.lock():			with dstrepo.wlock(), dstrepo.lock():
	backuppath = upgrade_engine.upgrade(			backuppath = upgrade_engine.upgrade(
	ui, repo, dstrepo, newreqs, upgradeactions, revlogs=revlogs			ui, repo, dstrepo, upgrade_op
	)			)
	if not (backup or backuppath is None):			if not (backup or backuppath is None):
	ui.status(			ui.status(
	_(b'removing old repository content%s\n') % backuppath			_(b'removing old repository content%s\n') % backuppath
	)			)
	repo.vfs.rmtree(backuppath, forcibly=True)			repo.vfs.rmtree(backuppath, forcibly=True)
	backuppath = None			backuppath = None

mercurial/upgrade_utils/actions.py

	newactions.append(d)			newactions.append(d)

	# FUTURE consider adding some optimizations here for certain transitions.			# FUTURE consider adding some optimizations here for certain transitions.
	# e.g. adding generaldelta could schedule parent redeltas.			# e.g. adding generaldelta could schedule parent redeltas.

	return newactions			return newactions


				class UpgradeOperation(object):
				"""represent the work to be done during an upgrade"""

				def __init__(self, requirements, actions, revlogs_to_process):
				self.requirements = requirements
				self.actions = actions
				self.revlogs_to_process = revlogs_to_process


	### Code checking if a repository can got through the upgrade process at all. #			### Code checking if a repository can got through the upgrade process at all. #


	def requiredsourcerequirements(repo):			def requiredsourcerequirements(repo):
	"""Obtain requirements required to be present to upgrade a repo.			"""Obtain requirements required to be present to upgrade a repo.

	An upgrade will not be allowed if the repository doesn't have the			An upgrade will not be allowed if the repository doesn't have the
	requirements returned by this function.			requirements returned by this function.

mercurial/upgrade_utils/engine.py

	def _finishdatamigration(ui, srcrepo, dstrepo, requirements):			def _finishdatamigration(ui, srcrepo, dstrepo, requirements):
	"""Hook point for extensions to perform additional actions during upgrade.			"""Hook point for extensions to perform additional actions during upgrade.

	This function is called after revlogs and store files have been copied but			This function is called after revlogs and store files have been copied but
	before the new store is swapped into the original location.			before the new store is swapped into the original location.
	"""			"""


	def upgrade(			def upgrade(ui, srcrepo, dstrepo, upgrade_op):
	ui, srcrepo, dstrepo, requirements, actions, revlogs=UPGRADE_ALL_REVLOGS
	):
	"""Do the low-level work of upgrading a repository.			"""Do the low-level work of upgrading a repository.

	The upgrade is effectively performed as a copy between a source			The upgrade is effectively performed as a copy between a source
	repository and a temporary destination repository.			repository and a temporary destination repository.

	The source repository is unmodified for as long as possible so the			The source repository is unmodified for as long as possible so the
	upgrade can abort at any time without causing loss of service for			upgrade can abort at any time without causing loss of service for
	readers and without corrupting the source repository.			readers and without corrupting the source repository.
	"""			"""
	assert srcrepo.currentwlock()			assert srcrepo.currentwlock()
	assert dstrepo.currentwlock()			assert dstrepo.currentwlock()

	ui.status(			ui.status(
	_(			_(
	b'(it is safe to interrupt this process any time before '			b'(it is safe to interrupt this process any time before '
	b'data migration completes)\n'			b'data migration completes)\n'
	)			)
	)			)

	if b're-delta-all' in actions:			if b're-delta-all' in upgrade_op.actions:
	deltareuse = revlog.revlog.DELTAREUSENEVER			deltareuse = revlog.revlog.DELTAREUSENEVER
	elif b're-delta-parent' in actions:			elif b're-delta-parent' in upgrade_op.actions:
	deltareuse = revlog.revlog.DELTAREUSESAMEREVS			deltareuse = revlog.revlog.DELTAREUSESAMEREVS
	elif b're-delta-multibase' in actions:			elif b're-delta-multibase' in upgrade_op.actions:
	deltareuse = revlog.revlog.DELTAREUSESAMEREVS			deltareuse = revlog.revlog.DELTAREUSESAMEREVS
	elif b're-delta-fulladd' in actions:			elif b're-delta-fulladd' in upgrade_op.actions:
	deltareuse = revlog.revlog.DELTAREUSEFULLADD			deltareuse = revlog.revlog.DELTAREUSEFULLADD
	else:			else:
	deltareuse = revlog.revlog.DELTAREUSEALWAYS			deltareuse = revlog.revlog.DELTAREUSEALWAYS

	with dstrepo.transaction(b'upgrade') as tr:			with dstrepo.transaction(b'upgrade') as tr:
	_clonerevlogs(			_clonerevlogs(
	ui,			ui,
	srcrepo,			srcrepo,
	dstrepo,			dstrepo,
	tr,			tr,
	deltareuse,			deltareuse,
	b're-delta-multibase' in actions,			b're-delta-multibase' in upgrade_op.actions,
	revlogs=revlogs,			revlogs=upgrade_op.revlogs_to_process,
	)			)

	# Now copy other files in the store directory.			# Now copy other files in the store directory.
	# The sorted() makes execution deterministic.			# The sorted() makes execution deterministic.
	for p, kind, st in sorted(srcrepo.store.vfs.readdir(b'', stat=True)):			for p, kind, st in sorted(srcrepo.store.vfs.readdir(b'', stat=True)):
	if not _filterstorefile(srcrepo, dstrepo, requirements, p, kind, st):			if not _filterstorefile(
				srcrepo, dstrepo, upgrade_op.requirements, p, kind, st
				):
	continue			continue

	srcrepo.ui.status(_(b'copying %s\n') % p)			srcrepo.ui.status(_(b'copying %s\n') % p)
	src = srcrepo.store.rawvfs.join(p)			src = srcrepo.store.rawvfs.join(p)
	dst = dstrepo.store.rawvfs.join(p)			dst = dstrepo.store.rawvfs.join(p)
	util.copyfile(src, dst, copystat=True)			util.copyfile(src, dst, copystat=True)

	_finishdatamigration(ui, srcrepo, dstrepo, requirements)			_finishdatamigration(ui, srcrepo, dstrepo, requirements)
	# We first write the requirements file. Any new requirements will lock			# We first write the requirements file. Any new requirements will lock
	# out legacy clients.			# out legacy clients.
	ui.status(			ui.status(
	_(			_(
	b'finalizing requirements file and making repository readable '			b'finalizing requirements file and making repository readable '
	b'again\n'			b'again\n'
	)			)
	)			)
	scmutil.writereporequirements(srcrepo, requirements)			scmutil.writereporequirements(srcrepo, upgrade_op.requirements)

	# The lock file from the old store won't be removed because nothing has a			# The lock file from the old store won't be removed because nothing has a
	# reference to its new location. So clean it up manually. Alternatively, we			# reference to its new location. So clean it up manually. Alternatively, we
	# could update srcrepo.svfs and other variables to point to the new			# could update srcrepo.svfs and other variables to point to the new
	# location. This is simpler.			# location. This is simpler.
	backupvfs.unlink(b'store/lock')			backupvfs.unlink(b'store/lock')

	return backuppath			return backuppath