This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4717

upgrade: report size of backing files, not internal storage size
ClosedPublic

Authored by indygreg on Sep 24 2018, 1:15 PM.

Download Raw Diff

Details

Reviewers

None

Group Reviewers

hg-reviewers

Commits

rHG8dab7c8a93eb: upgrade: report size of backing files, not internal storage size

Summary

upgrade.py is the only consumer of filelog.index, which I'd like
to eliminate from the file storage interface.

This commit changes the upgrade code to report the storage size
of files by looking at the size of the files backing its storage
instead of looking at the index.

I'm not convinced the approach in this patch will live very long
because it is relying on low-level attributes like "opener" and
"files," which may behave very differently on non-revlog storage.
But the data is only used for reporting purposes and it does get
us one step closer to eliminating "index."

A side-effect of this change is we now report the size of the revlog
index data - not just the revision data. I think this is more
accurate.

Diff Detail

Repository

rHG Mercurial

Lint

Lint Skipped

Unit

Unit Tests Skipped

Event Timeline

indygreg created this revision.Sep 24 2018, 1:15 PM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptSep 24 2018, 1:15 PM

Herald added a subscriber: mercurial-devel. · View Herald Transcript

indygreg added a child revision: D4718: manifest: add rawsize() proxy (API).Sep 24 2018, 1:15 PM

Closed by commit rHG8dab7c8a93eb: upgrade: report size of backing files, not internal storage size (authored by indygreg). · Explain WhySep 26 2018, 1:24 PM

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

			Path	Packages
M			mercurial/upgrade.py (10 lines)
M			tests/test-lfs-serve.t (8 lines)
M			tests/test-upgrade-repo.t (48 lines)

Commit	Parents	Author	Summary	Date
		Gregory Szorc		Sep 24 2018, 12:37 PM

Status	Author	Revision
Closed	indygreg	D4727 filelog: drop _generaldelta attribute (API)
Closed	indygreg	D4726 revlog: drop emitrevisiondeltas() and associated functionality (API)
Closed	indygreg	D4725 changegroup: port to emitrevisions() (issue5976)
Closed	indygreg	D4724 wireprotov2server: port to emitrevisions()
Closed	indygreg	D4723 tests: use more complex file storage test
Closed	indygreg	D4722 revlog: new API to emit revision data
Closed	indygreg	D4721 changegroup: remove reordering control (BC)
Closed	indygreg	D4720 filelog: drop index attribute (API)
Closed	indygreg	D4719 upgrade: use rawsize() instead of revlog index
Closed	indygreg	D4718 manifest: add rawsize() proxy (API)
Closed	indygreg	D4717 upgrade: report size of backing files, not internal storage size
Closed	indygreg	D4715 filelog: store filename directly on revlog instance
Closed	indygreg	D4714 lfs: access revlog directly
Closed	indygreg	D4713 largefiles: automatically load largefiles extension when required (BC)
Closed	indygreg	D4712 lfs: don't add extension to hgrc after clone or share (BC)
Closed	indygreg	D4711 localrepo: automatically load lfs extension when required (BC)
Closed	indygreg	D4710 lfs: add repository feature denoting the use of LFS
Closed	indygreg	D4709 localrepo: define "features" on repository instances (API)
Closed	indygreg	D4708 localrepo: support writing shared file (API)
Closed	indygreg	D4707 localrepo: support shared repo creation
Closed	indygreg	D4706 localrepo: validate directories before creating any
Closed	indygreg	D4705 localrepo: add missing join()
Closed	indygreg	D4704 revlog: use proper version comparison during verify
Closed	indygreg	D4703 filelog: stop proxying checksize() (API)
Closed	indygreg	D4702 filelog: remove version attribute (API)
Closed	indygreg	D4701 verify: start to abstract file verification
Closed	indygreg	D4700 unionrepo: remove _constructmanifest()

Diff 11325

mercurial/upgrade.py

	if unencoded.endswith('.d'):			if unencoded.endswith('.d'):
	continue			continue

	rl = _revlogfrompath(srcrepo, unencoded)			rl = _revlogfrompath(srcrepo, unencoded)
	revcount += len(rl)			revcount += len(rl)

	datasize = 0			datasize = 0
	rawsize = 0			rawsize = 0

				for path in rl.files():
				datasize += rl.opener.stat(path).st_size

	idx = rl.index			idx = rl.index
	for rev in rl:			for rev in rl:
	e = idx[rev]			e = idx[rev]
	datasize += e[1]
	rawsize += e[2]			rawsize += e[2]

	srcsize += datasize			srcsize += datasize
	srcrawsize += rawsize			srcrawsize += rawsize

	# This is for the separate progress bars.			# This is for the separate progress bars.
	if isinstance(rl, changelog.changelog):			if isinstance(rl, changelog.changelog):
	crevcount += len(rl)			crevcount += len(rl)


	ui.note(_('cloning %d revisions from %s\n') % (len(oldrl), unencoded))			ui.note(_('cloning %d revisions from %s\n') % (len(oldrl), unencoded))
	oldrl.clone(tr, newrl, addrevisioncb=oncopiedrevision,			oldrl.clone(tr, newrl, addrevisioncb=oncopiedrevision,
	deltareuse=deltareuse,			deltareuse=deltareuse,
	deltabothparents=deltabothparents)			deltabothparents=deltabothparents)

	datasize = 0			datasize = 0
	idx = newrl.index			for path in newrl.files():
	for rev in newrl:			datasize += newrl.opener.stat(path).st_size
	datasize += idx[rev][1]

	dstsize += datasize			dstsize += datasize

	if isinstance(newrl, changelog.changelog):			if isinstance(newrl, changelog.changelog):
	cdstsize += datasize			cdstsize += datasize
	elif isinstance(newrl, manifest.manifestrevlog):			elif isinstance(newrl, manifest.manifestrevlog):
	mdstsize += datasize			mdstsize += datasize
	else:			else:

tests/test-lfs-serve.t

	requirements			requirements
	preserved: dotencode, fncache, generaldelta, revlogv1, store			preserved: dotencode, fncache, generaldelta, revlogv1, store

	beginning upgrade...			beginning upgrade...
	repository locked and read-only			repository locked and read-only
	creating temporary repository to stage migrated data: * (glob)			creating temporary repository to stage migrated data: * (glob)
	(it is safe to interrupt this process any time before data migration completes)			(it is safe to interrupt this process any time before data migration completes)
	migrating 3 total revisions (1 in filelogs, 1 in manifests, 1 in changelog)			migrating 3 total revisions (1 in filelogs, 1 in manifests, 1 in changelog)
	migrating 132 bytes in store; 129 bytes tracked data			migrating 324 bytes in store; 129 bytes tracked data
	migrating 1 filelogs containing 1 revisions (9 bytes in store; 8 bytes tracked data)			migrating 1 filelogs containing 1 revisions (73 bytes in store; 8 bytes tracked data)
	finished migrating 1 filelog revisions across 1 filelogs; change in size: 0 bytes			finished migrating 1 filelog revisions across 1 filelogs; change in size: 0 bytes
	migrating 1 manifests containing 1 revisions (53 bytes in store; 52 bytes tracked data)			migrating 1 manifests containing 1 revisions (117 bytes in store; 52 bytes tracked data)
	finished migrating 1 manifest revisions across 1 manifests; change in size: 0 bytes			finished migrating 1 manifest revisions across 1 manifests; change in size: 0 bytes
	migrating changelog containing 1 revisions (70 bytes in store; 69 bytes tracked data)			migrating changelog containing 1 revisions (134 bytes in store; 69 bytes tracked data)
	finished migrating 1 changelog revisions; change in size: 0 bytes			finished migrating 1 changelog revisions; change in size: 0 bytes
	finished migrating 3 total revisions; total change in store size: 0 bytes			finished migrating 3 total revisions; total change in store size: 0 bytes
	copying phaseroots			copying phaseroots
	data fully migrated to temporary repository			data fully migrated to temporary repository
	marking source repository as being upgraded; clients will be unable to read from repository			marking source repository as being upgraded; clients will be unable to read from repository
	starting in-place swap of repository data			starting in-place swap of repository data
	replaced files will be backed up at * (glob)			replaced files will be backed up at * (glob)
	replacing store...			replacing store...

tests/test-upgrade-repo.t

	generaldelta			generaldelta
	repository storage will be able to create optimal deltas; new repository data will be smaller and read times should decrease; interacting with other repositories using this storage model should require less network and CPU resources, making "hg push" and "hg pull" faster			repository storage will be able to create optimal deltas; new repository data will be smaller and read times should decrease; interacting with other repositories using this storage model should require less network and CPU resources, making "hg push" and "hg pull" faster

	beginning upgrade...			beginning upgrade...
	repository locked and read-only			repository locked and read-only
	creating temporary repository to stage migrated data: $TESTTMP/upgradegd/.hg/upgrade.* (glob)			creating temporary repository to stage migrated data: $TESTTMP/upgradegd/.hg/upgrade.* (glob)
	(it is safe to interrupt this process any time before data migration completes)			(it is safe to interrupt this process any time before data migration completes)
	migrating 9 total revisions (3 in filelogs, 3 in manifests, 3 in changelog)			migrating 9 total revisions (3 in filelogs, 3 in manifests, 3 in changelog)
	migrating 341 bytes in store; 401 bytes tracked data			migrating 917 bytes in store; 401 bytes tracked data
	migrating 3 filelogs containing 3 revisions (0 bytes in store; 0 bytes tracked data)			migrating 3 filelogs containing 3 revisions (192 bytes in store; 0 bytes tracked data)
	finished migrating 3 filelog revisions across 3 filelogs; change in size: 0 bytes			finished migrating 3 filelog revisions across 3 filelogs; change in size: 0 bytes
	migrating 1 manifests containing 3 revisions (157 bytes in store; 220 bytes tracked data)			migrating 1 manifests containing 3 revisions (349 bytes in store; 220 bytes tracked data)
	finished migrating 3 manifest revisions across 1 manifests; change in size: 0 bytes			finished migrating 3 manifest revisions across 1 manifests; change in size: 0 bytes
	migrating changelog containing 3 revisions (184 bytes in store; 181 bytes tracked data)			migrating changelog containing 3 revisions (376 bytes in store; 181 bytes tracked data)
	finished migrating 3 changelog revisions; change in size: 0 bytes			finished migrating 3 changelog revisions; change in size: 0 bytes
	finished migrating 9 total revisions; total change in store size: 0 bytes			finished migrating 9 total revisions; total change in store size: 0 bytes
	copying phaseroots			copying phaseroots
	data fully migrated to temporary repository			data fully migrated to temporary repository
	marking source repository as being upgraded; clients will be unable to read from repository			marking source repository as being upgraded; clients will be unable to read from repository
	starting in-place swap of repository data			starting in-place swap of repository data
	replaced files will be backed up at $TESTTMP/upgradegd/.hg/upgradebackup.* (glob)			replaced files will be backed up at $TESTTMP/upgradegd/.hg/upgradebackup.* (glob)
	replacing store...			replacing store...
	requirements			requirements
	preserved: dotencode, fncache, generaldelta, revlogv1, store			preserved: dotencode, fncache, generaldelta, revlogv1, store

	beginning upgrade...			beginning upgrade...
	repository locked and read-only			repository locked and read-only
	creating temporary repository to stage migrated data: $TESTTMP/store-filenames/.hg/upgrade.* (glob)			creating temporary repository to stage migrated data: $TESTTMP/store-filenames/.hg/upgrade.* (glob)
	(it is safe to interrupt this process any time before data migration completes)			(it is safe to interrupt this process any time before data migration completes)
	migrating 3 total revisions (1 in filelogs, 1 in manifests, 1 in changelog)			migrating 3 total revisions (1 in filelogs, 1 in manifests, 1 in changelog)
	migrating 109 bytes in store; 107 bytes tracked data			migrating 301 bytes in store; 107 bytes tracked data
	migrating 1 filelogs containing 1 revisions (0 bytes in store; 0 bytes tracked data)			migrating 1 filelogs containing 1 revisions (64 bytes in store; 0 bytes tracked data)
	finished migrating 1 filelog revisions across 1 filelogs; change in size: 0 bytes			finished migrating 1 filelog revisions across 1 filelogs; change in size: 0 bytes
	migrating 1 manifests containing 1 revisions (46 bytes in store; 45 bytes tracked data)			migrating 1 manifests containing 1 revisions (110 bytes in store; 45 bytes tracked data)
	finished migrating 1 manifest revisions across 1 manifests; change in size: 0 bytes			finished migrating 1 manifest revisions across 1 manifests; change in size: 0 bytes
	migrating changelog containing 1 revisions (63 bytes in store; 62 bytes tracked data)			migrating changelog containing 1 revisions (127 bytes in store; 62 bytes tracked data)
	finished migrating 1 changelog revisions; change in size: 0 bytes			finished migrating 1 changelog revisions; change in size: 0 bytes
	finished migrating 3 total revisions; total change in store size: 0 bytes			finished migrating 3 total revisions; total change in store size: 0 bytes
	copying .XX_special_filename			copying .XX_special_filename
	copying phaseroots			copying phaseroots
	data fully migrated to temporary repository			data fully migrated to temporary repository
	marking source repository as being upgraded; clients will be unable to read from repository			marking source repository as being upgraded; clients will be unable to read from repository
	starting in-place swap of repository data			starting in-place swap of repository data
	replaced files will be backed up at $TESTTMP/store-filenames/.hg/upgradebackup.* (glob)			replaced files will be backed up at $TESTTMP/store-filenames/.hg/upgradebackup.* (glob)
	redeltafulladd			redeltafulladd
	each revision will be added as new content to the internal storage; this will likely drastically slow down execution time, but some extensions might need it			each revision will be added as new content to the internal storage; this will likely drastically slow down execution time, but some extensions might need it

	beginning upgrade...			beginning upgrade...
	repository locked and read-only			repository locked and read-only
	creating temporary repository to stage migrated data: $TESTTMP/store-filenames/.hg/upgrade.* (glob)			creating temporary repository to stage migrated data: $TESTTMP/store-filenames/.hg/upgrade.* (glob)
	(it is safe to interrupt this process any time before data migration completes)			(it is safe to interrupt this process any time before data migration completes)
	migrating 3 total revisions (1 in filelogs, 1 in manifests, 1 in changelog)			migrating 3 total revisions (1 in filelogs, 1 in manifests, 1 in changelog)
	migrating 109 bytes in store; 107 bytes tracked data			migrating 301 bytes in store; 107 bytes tracked data
	migrating 1 filelogs containing 1 revisions (0 bytes in store; 0 bytes tracked data)			migrating 1 filelogs containing 1 revisions (64 bytes in store; 0 bytes tracked data)
	finished migrating 1 filelog revisions across 1 filelogs; change in size: 0 bytes			finished migrating 1 filelog revisions across 1 filelogs; change in size: 0 bytes
	migrating 1 manifests containing 1 revisions (46 bytes in store; 45 bytes tracked data)			migrating 1 manifests containing 1 revisions (110 bytes in store; 45 bytes tracked data)
	finished migrating 1 manifest revisions across 1 manifests; change in size: 0 bytes			finished migrating 1 manifest revisions across 1 manifests; change in size: 0 bytes
	migrating changelog containing 1 revisions (63 bytes in store; 62 bytes tracked data)			migrating changelog containing 1 revisions (127 bytes in store; 62 bytes tracked data)
	finished migrating 1 changelog revisions; change in size: 0 bytes			finished migrating 1 changelog revisions; change in size: 0 bytes
	finished migrating 3 total revisions; total change in store size: 0 bytes			finished migrating 3 total revisions; total change in store size: 0 bytes
	copying .XX_special_filename			copying .XX_special_filename
	copying phaseroots			copying phaseroots
	data fully migrated to temporary repository			data fully migrated to temporary repository
	marking source repository as being upgraded; clients will be unable to read from repository			marking source repository as being upgraded; clients will be unable to read from repository
	starting in-place swap of repository data			starting in-place swap of repository data
	replaced files will be backed up at $TESTTMP/store-filenames/.hg/upgradebackup.* (glob)			replaced files will be backed up at $TESTTMP/store-filenames/.hg/upgradebackup.* (glob)
	requirements			requirements
	preserved: dotencode, fncache, generaldelta, largefiles, revlogv1, store			preserved: dotencode, fncache, generaldelta, largefiles, revlogv1, store

	beginning upgrade...			beginning upgrade...
	repository locked and read-only			repository locked and read-only
	creating temporary repository to stage migrated data: $TESTTMP/largefilesrepo/.hg/upgrade.* (glob)			creating temporary repository to stage migrated data: $TESTTMP/largefilesrepo/.hg/upgrade.* (glob)
	(it is safe to interrupt this process any time before data migration completes)			(it is safe to interrupt this process any time before data migration completes)
	migrating 3 total revisions (1 in filelogs, 1 in manifests, 1 in changelog)			migrating 3 total revisions (1 in filelogs, 1 in manifests, 1 in changelog)
	migrating 163 bytes in store; 160 bytes tracked data			migrating 355 bytes in store; 160 bytes tracked data
	migrating 1 filelogs containing 1 revisions (42 bytes in store; 41 bytes tracked data)			migrating 1 filelogs containing 1 revisions (106 bytes in store; 41 bytes tracked data)
	finished migrating 1 filelog revisions across 1 filelogs; change in size: 0 bytes			finished migrating 1 filelog revisions across 1 filelogs; change in size: 0 bytes
	migrating 1 manifests containing 1 revisions (52 bytes in store; 51 bytes tracked data)			migrating 1 manifests containing 1 revisions (116 bytes in store; 51 bytes tracked data)
	finished migrating 1 manifest revisions across 1 manifests; change in size: 0 bytes			finished migrating 1 manifest revisions across 1 manifests; change in size: 0 bytes
	migrating changelog containing 1 revisions (69 bytes in store; 68 bytes tracked data)			migrating changelog containing 1 revisions (133 bytes in store; 68 bytes tracked data)
	finished migrating 1 changelog revisions; change in size: 0 bytes			finished migrating 1 changelog revisions; change in size: 0 bytes
	finished migrating 3 total revisions; total change in store size: 0 bytes			finished migrating 3 total revisions; total change in store size: 0 bytes
	copying phaseroots			copying phaseroots
	data fully migrated to temporary repository			data fully migrated to temporary repository
	marking source repository as being upgraded; clients will be unable to read from repository			marking source repository as being upgraded; clients will be unable to read from repository
	starting in-place swap of repository data			starting in-place swap of repository data
	replaced files will be backed up at $TESTTMP/largefilesrepo/.hg/upgradebackup.* (glob)			replaced files will be backed up at $TESTTMP/largefilesrepo/.hg/upgradebackup.* (glob)
	replacing store...			replacing store...
	requirements			requirements
	preserved: dotencode, fncache, generaldelta, largefiles, lfs, revlogv1, store			preserved: dotencode, fncache, generaldelta, largefiles, lfs, revlogv1, store

	beginning upgrade...			beginning upgrade...
	repository locked and read-only			repository locked and read-only
	creating temporary repository to stage migrated data: $TESTTMP/largefilesrepo/.hg/upgrade.* (glob)			creating temporary repository to stage migrated data: $TESTTMP/largefilesrepo/.hg/upgrade.* (glob)
	(it is safe to interrupt this process any time before data migration completes)			(it is safe to interrupt this process any time before data migration completes)
	migrating 6 total revisions (2 in filelogs, 2 in manifests, 2 in changelog)			migrating 6 total revisions (2 in filelogs, 2 in manifests, 2 in changelog)
	migrating 417 bytes in store; 467 bytes tracked data			migrating 801 bytes in store; 467 bytes tracked data
	migrating 2 filelogs containing 2 revisions (168 bytes in store; 182 bytes tracked data)			migrating 2 filelogs containing 2 revisions (296 bytes in store; 182 bytes tracked data)
	finished migrating 2 filelog revisions across 2 filelogs; change in size: 0 bytes			finished migrating 2 filelog revisions across 2 filelogs; change in size: 0 bytes
	migrating 1 manifests containing 2 revisions (113 bytes in store; 151 bytes tracked data)			migrating 1 manifests containing 2 revisions (241 bytes in store; 151 bytes tracked data)
	finished migrating 2 manifest revisions across 1 manifests; change in size: 0 bytes			finished migrating 2 manifest revisions across 1 manifests; change in size: 0 bytes
	migrating changelog containing 2 revisions (136 bytes in store; 134 bytes tracked data)			migrating changelog containing 2 revisions (264 bytes in store; 134 bytes tracked data)
	finished migrating 2 changelog revisions; change in size: 0 bytes			finished migrating 2 changelog revisions; change in size: 0 bytes
	finished migrating 6 total revisions; total change in store size: 0 bytes			finished migrating 6 total revisions; total change in store size: 0 bytes
	copying phaseroots			copying phaseroots
	copying lfs blob d0beab232adff5ba365880366ad30b1edb85c4c5372442b5d2fe27adc96d653f			copying lfs blob d0beab232adff5ba365880366ad30b1edb85c4c5372442b5d2fe27adc96d653f
	data fully migrated to temporary repository			data fully migrated to temporary repository
	marking source repository as being upgraded; clients will be unable to read from repository			marking source repository as being upgraded; clients will be unable to read from repository
	starting in-place swap of repository data			starting in-place swap of repository data
	replaced files will be backed up at $TESTTMP/largefilesrepo/.hg/upgradebackup.* (glob)			replaced files will be backed up at $TESTTMP/largefilesrepo/.hg/upgradebackup.* (glob)
	redeltaall			redeltaall
	deltas within internal storage will be fully recomputed; this will likely drastically slow down execution time			deltas within internal storage will be fully recomputed; this will likely drastically slow down execution time

	beginning upgrade...			beginning upgrade...
	repository locked and read-only			repository locked and read-only
	creating temporary repository to stage migrated data: $TESTTMP/localconfig/.hg/upgrade.* (glob)			creating temporary repository to stage migrated data: $TESTTMP/localconfig/.hg/upgrade.* (glob)
	(it is safe to interrupt this process any time before data migration completes)			(it is safe to interrupt this process any time before data migration completes)
	migrating 9 total revisions (3 in filelogs, 3 in manifests, 3 in changelog)			migrating 9 total revisions (3 in filelogs, 3 in manifests, 3 in changelog)
	migrating 497 bytes in store; 882 bytes tracked data			migrating 1.05 KB in store; 882 bytes tracked data
	migrating 1 filelogs containing 3 revisions (182 bytes in store; 573 bytes tracked data)			migrating 1 filelogs containing 3 revisions (374 bytes in store; 573 bytes tracked data)
	finished migrating 3 filelog revisions across 1 filelogs; change in size: -63 bytes			finished migrating 3 filelog revisions across 1 filelogs; change in size: -63 bytes
	migrating 1 manifests containing 3 revisions (141 bytes in store; 138 bytes tracked data)			migrating 1 manifests containing 3 revisions (333 bytes in store; 138 bytes tracked data)
	finished migrating 3 manifest revisions across 1 manifests; change in size: 0 bytes			finished migrating 3 manifest revisions across 1 manifests; change in size: 0 bytes
	migrating changelog containing 3 revisions (174 bytes in store; 171 bytes tracked data)			migrating changelog containing 3 revisions (366 bytes in store; 171 bytes tracked data)
	finished migrating 3 changelog revisions; change in size: 0 bytes			finished migrating 3 changelog revisions; change in size: 0 bytes
	finished migrating 9 total revisions; total change in store size: -63 bytes			finished migrating 9 total revisions; total change in store size: -63 bytes
	copying phaseroots			copying phaseroots
	data fully migrated to temporary repository			data fully migrated to temporary repository
	marking source repository as being upgraded; clients will be unable to read from repository			marking source repository as being upgraded; clients will be unable to read from repository
	starting in-place swap of repository data			starting in-place swap of repository data
	replaced files will be backed up at $TESTTMP/localconfig/.hg/upgradebackup.* (glob)			replaced files will be backed up at $TESTTMP/localconfig/.hg/upgradebackup.* (glob)
	replacing store...			replacing store...

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	11325		Sep 24 2018, 1:15 PM	★	★
Diff 2	11388	rHG8dab7c8a93eb92749e927a6cb04a1ffc045a95a2	Sep 24 2018, 12:37 PM	★	★