This is an archive of the discontinued Mercurial Phabricator instance.

Differential D1228

repack: add --packsonly option to repack
ClosedPublic

Authored by phillco on Oct 26 2017, 12:37 AM.

Download Raw Diff

Details

Reviewers

durham

Group Reviewers

Restricted Project

Commits

rFBHGX0c972d6cb3a8: repack: add --packsonly option to repack

Summary

Occasionally, callers to hg repack prefer to skip loose objects and only
repack packfiles. This adds an option to do so.

Diff Detail

Repository

rFBHGX Facebook Mercurial Extensions

Lint

Lint Skipped

Unit

Unit Tests Skipped

Event Timeline

phillco created this revision.Oct 26 2017, 12:37 AM

Herald added a reviewer: Restricted Project. · View Herald TranscriptOct 26 2017, 12:37 AM

phillco added a reviewer: durham.Oct 26 2017, 1:00 AM

Will a future change add this to the incremental repack code?

remotefilelog/repack.py
44	getpack is somewhat of an internal only function, and not really indicative of a pack store. In theory you could have a datapack (not a datapackstore) in the repo.shareddatastore collection which should be repacked as part of --packsonly, but wouldn't trigger this condition. Instead, it might be nice to create a PackOptions struct. Maybe attach it to the ledger, then each store can look at the options and decide what it should do. Stores related to loose objects could exclude them when they see the packonly option set.

This revision now requires changes to proceed.Oct 26 2017, 3:53 PM

phillco updated this revision to Diff 3330.Nov 7 2017, 11:44 PM

phillco added inline comments.Nov 7 2017, 11:46 PM

remotefilelog/basestore.py
59–60	dict is kinda clunky here, maybe we could use `attrs` as Sidd suggested.

Oops, forgot your comment about adding it to incremental repack. I'll do that now.

phillco updated this revision to Diff 3336.Nov 8 2017, 12:02 PM

OK, I added to incremental repacks too.

Requesting changes mainly because I expect that one test to actually fail now, and am confused why it doesn't.

remotefilelog/basestore.py
59	`if options and options.get('packsonly'):`
remotefilelog/contentstore.py
305	We should probably use a constant for the option name, so we don't typo it somewhere.
tests/test-treemanifest-repack.t
189	I think this test is checking that hg repack can repack from revlogs into pack files. So I would actually expect this to fail if you set --packsonly, since in that case the revlogs should not be inspected.

This revision now requires changes to proceed.Nov 9 2017, 11:45 AM

phillco added inline comments.Nov 9 2017, 11:54 AM

tests/test-treemanifest-repack.t
189	Ah, it's because I didn't patch serverepack(). I guess I should.

phillco updated this revision to Diff 3377.Nov 9 2017, 1:16 PM

durham accepted this revision.Nov 9 2017, 1:19 PM

durham added inline comments.

remotefilelog/constants.py
34 ↗	(On Diff #3377)	`markledger`

This revision is now accepted and ready to land.Nov 9 2017, 1:19 PM

landed

Revision Contents
Changeset List

			Path	Packages
M			remotefilelog/__init__.py (7 lines)
M			remotefilelog/basepack.py (4 lines)
M			remotefilelog/basestore.py (4 lines)
M			remotefilelog/contentstore.py (10 lines)
M			remotefilelog/datapack.py (4 lines)
M			remotefilelog/historypack.py (2 lines)
M			remotefilelog/metadatastore.py (6 lines)
M			remotefilelog/repack.py (42 lines)
M			remotefilelog/wirepack.py (2 lines)
M			tests/test-remotefilelog-repack.t (12 lines)
M			tests/test-treemanifest-repack.t (3 lines)
M			treemanifest/__init__.py (2 lines)

Diff 3336

remotefilelog/init.py


	revs = scmutil.revrange(repo, opts.get('rev'))			revs = scmutil.revrange(repo, opts.get('rev'))

	repo.prefetch(revs, repack=opts.get('repack'), pats=pats, opts=opts)			repo.prefetch(revs, repack=opts.get('repack'), pats=pats, opts=opts)

	@command('repack', [			@command('repack', [
	('', 'background', None, _('run in a background process'), None),			('', 'background', None, _('run in a background process'), None),
	('', 'incremental', None, _('do an incremental repack'), None),			('', 'incremental', None, _('do an incremental repack'), None),
				('', 'packsonly', None, _('only repack packs (skip loose objects)'), None),
	], _('hg repack [OPTIONS]'))			], _('hg repack [OPTIONS]'))
	def repack(ui, repo, pats, *opts):			def repack(ui, repo, pats, *opts):
	if opts.get('background'):			if opts.get('background'):
	repackmod.backgroundrepack(repo, incremental=opts.get('incremental'))			repackmod.backgroundrepack(repo, incremental=opts.get('incremental'))
	return			return

				options = {'packsonly': opts.get('packsonly')}

	if opts.get('incremental'):			if opts.get('incremental'):
	repackmod.incrementalrepack(repo)			repackmod.incrementalrepack(repo, options=options)
	else:			else:
	repackmod.fullrepack(repo)			repackmod.fullrepack(repo, options=options)

remotefilelog/basepack.py

	missing = pack.getmissing(missing)			missing = pack.getmissing(missing)

	if missing:			if missing:
	for pack in self.refresh():			for pack in self.refresh():
	missing = pack.getmissing(missing)			missing = pack.getmissing(missing)

	return missing			return missing

	def markledger(self, ledger):			def markledger(self, ledger, options=None):
	for pack in self.packs:			for pack in self.packs:
	pack.markledger(ledger)			pack.markledger(ledger)

	def markforrefresh(self):			def markforrefresh(self):
	"""Tells the store that there may be new pack files, so the next time it			"""Tells the store that there may be new pack files, so the next time it
	has a lookup miss it should check for new files."""			has a lookup miss it should check for new files."""
	self.lastrefresh = 0			self.lastrefresh = 0

	self._data = mmap.mmap(datafp.fileno(), 0, access=mmap.ACCESS_READ)			self._data = mmap.mmap(datafp.fileno(), 0, access=mmap.ACCESS_READ)

	self._pagedin = 0			self._pagedin = 0
	return True			return True

	def getmissing(self, keys):			def getmissing(self, keys):
	raise NotImplemented()			raise NotImplemented()

	def markledger(self, ledger):			def markledger(self, ledger, options=None):
	raise NotImplemented()			raise NotImplemented()

	def cleanup(self, ledger):			def cleanup(self, ledger):
	raise NotImplemented()			raise NotImplemented()

	def __iter__(self):			def __iter__(self):
	raise NotImplemented()			raise NotImplemented()

remotefilelog/basestore.py

	exists = False			exists = False
	if not exists:			if not exists:
	missing.append((name, node))			missing.append((name, node))

	return missing			return missing

	# BELOW THIS ARE IMPLEMENTATIONS OF REPACK SOURCE			# BELOW THIS ARE IMPLEMENTATIONS OF REPACK SOURCE

	def markledger(self, ledger):			def markledger(self, ledger, options=None):
				if options and 'packsonly' in options and options['packsonly']:
				durhamUnsubmitted Not Done `if options and options.get('packsonly'):` durham: `if options and options.get('packsonly'):`
				return
				phillcoAuthorUnsubmitted Not Done dict is kinda clunky here, maybe we could use `attrs` as Sidd suggested. phillco: dict is kinda clunky here, maybe we could use `attrs` as Sidd suggested.
	if self._shared:			if self._shared:
	for filename, nodes in self._getfiles():			for filename, nodes in self._getfiles():
	for node in nodes:			for node in nodes:
	ledger.markdataentry(self, filename, node)			ledger.markdataentry(self, filename, node)
	ledger.markhistoryentry(self, filename, node)			ledger.markhistoryentry(self, filename, node)

	def cleanup(self, ledger):			def cleanup(self, ledger):
	ui = self.ui			ui = self.ui

remotefilelog/contentstore.py

	return missing			return missing

	def addremotefilelognode(self, name, node, data):			def addremotefilelognode(self, name, node, data):
	if self.writestore:			if self.writestore:
	self.writestore.addremotefilelognode(name, node, data)			self.writestore.addremotefilelognode(name, node, data)
	else:			else:
	raise RuntimeError("no writable store configured")			raise RuntimeError("no writable store configured")

	def markledger(self, ledger):			def markledger(self, ledger, options=None):
	for store in self.stores:			for store in self.stores:
	store.markledger(ledger)			store.markledger(ledger, options)

	def markforrefresh(self):			def markforrefresh(self):
	for store in self.stores:			for store in self.stores:
	if util.safehasattr(store, 'markforrefresh'):			if util.safehasattr(store, 'markforrefresh'):
	store.markforrefresh()			store.markforrefresh()

	class remotefilelogcontentstore(basestore.basestore):			class remotefilelogcontentstore(basestore.basestore):
	def __init__(self, args, *kwargs):			def __init__(self, args, *kwargs):
	return self._shared.getmeta(name, node)			return self._shared.getmeta(name, node)

	def add(self, name, node, data):			def add(self, name, node, data):
	raise RuntimeError("cannot add to a remote store")			raise RuntimeError("cannot add to a remote store")

	def getmissing(self, keys):			def getmissing(self, keys):
	return keys			return keys

	def markledger(self, ledger):			def markledger(self, ledger, options=None):
	pass			pass

	class manifestrevlogstore(object):			class manifestrevlogstore(object):
	def __init__(self, repo):			def __init__(self, repo):
	self._store = repo.store			self._store = repo.store
	self._svfs = repo.svfs			self._svfs = repo.svfs
	self._revlogs = dict()			self._revlogs = dict()
	self._cl = revlog.revlog(self._svfs, '00changelog.i')			self._cl = revlog.revlog(self._svfs, '00changelog.i')
	missing.append((name, node))			missing.append((name, node))

	return missing			return missing

	def setrepacklinkrevrange(self, startrev, endrev):			def setrepacklinkrevrange(self, startrev, endrev):
	self._repackstartlinkrev = startrev			self._repackstartlinkrev = startrev
	self._repackendlinkrev = endrev			self._repackendlinkrev = endrev

	def markledger(self, ledger):			def markledger(self, ledger, options=None):
				if options and 'packsonly' in options and options['packsonly']:
				durhamUnsubmitted Not Done We should probably use a constant for the option name, so we don't typo it somewhere. durham: We should probably use a constant for the option name, so we don't typo it somewhere.
				return
	treename = ''			treename = ''
	rl = revlog.revlog(self._svfs, '00manifesttree.i')			rl = revlog.revlog(self._svfs, '00manifesttree.i')
	startlinkrev = self._repackstartlinkrev			startlinkrev = self._repackstartlinkrev
	endlinkrev = self._repackendlinkrev			endlinkrev = self._repackendlinkrev
	for rev in xrange(len(rl) - 1, -1, -1):			for rev in xrange(len(rl) - 1, -1, -1):
	linkrev = rl.linkrev(rev)			linkrev = rl.linkrev(rev)
	if linkrev < startlinkrev:			if linkrev < startlinkrev:
	break			break

remotefilelog/datapack.py

	elif node < midnode:			elif node < midnode:
	end = mid			end = mid
	endnode = midnode			endnode = midnode
	else:			else:
	return None			return None

	return struct.unpack(self.INDEXFORMAT, entry)			return struct.unpack(self.INDEXFORMAT, entry)

	def markledger(self, ledger):			def markledger(self, ledger, options=None):
	for filename, node in self:			for filename, node in self:
	ledger.markdataentry(self, filename, node)			ledger.markdataentry(self, filename, node)

	def cleanup(self, ledger):			def cleanup(self, ledger):
	entries = ledger.sources.get(self, [])			entries = ledger.sources.get(self, [])
	allkeys = set(self)			allkeys = set(self)
	repackedkeys = set((e.filename, e.node) for e in entries if			repackedkeys = set((e.filename, e.node) for e in entries if
	e.datarepacked or e.gced)			e.datarepacked or e.gced)
	if result is None:			if result is None:
	raise KeyError((name, hex(node)))			raise KeyError((name, hex(node)))

	return result			return result

	def add(self, name, node, data):			def add(self, name, node, data):
	raise RuntimeError("cannot add to datapack (%s:%s)" % (name, node))			raise RuntimeError("cannot add to datapack (%s:%s)" % (name, node))

	def markledger(self, ledger):			def markledger(self, ledger, options=None):
	for filename, node in self:			for filename, node in self:
	ledger.markdataentry(self, filename, node)			ledger.markdataentry(self, filename, node)

	def cleanup(self, ledger):			def cleanup(self, ledger):
	entries = ledger.sources.get(self, [])			entries = ledger.sources.get(self, [])
	allkeys = set(self)			allkeys = set(self)
	repackedkeys = set((e.filename, e.node) for e in entries if			repackedkeys = set((e.filename, e.node) for e in entries if
	e.datarepacked or e.gced)			e.datarepacked or e.gced)

remotefilelog/historypack.py

	if node > midnode:			if node > midnode:
	start = mid			start = mid
	startnode = midnode			startnode = midnode
	elif node < midnode:			elif node < midnode:
	end = mid			end = mid
	endnode = midnode			endnode = midnode
	return None			return None

	def markledger(self, ledger):			def markledger(self, ledger, options=None):
	for filename, node in self:			for filename, node in self:
	ledger.markhistoryentry(self, filename, node)			ledger.markhistoryentry(self, filename, node)

	def cleanup(self, ledger):			def cleanup(self, ledger):
	entries = ledger.sources.get(self, [])			entries = ledger.sources.get(self, [])
	allkeys = set(self)			allkeys = set(self)
	repackedkeys = set((e.filename, e.node) for e in entries if			repackedkeys = set((e.filename, e.node) for e in entries if
	e.historyrepacked)			e.historyrepacked)

remotefilelog/metadatastore.py


	def getmissing(self, keys):			def getmissing(self, keys):
	missing = keys			missing = keys
	for store in self.stores:			for store in self.stores:
	if missing:			if missing:
	missing = store.getmissing(missing)			missing = store.getmissing(missing)
	return missing			return missing

	def markledger(self, ledger):			def markledger(self, ledger, options=None):
	for store in self.stores:			for store in self.stores:
	store.markledger(ledger)			store.markledger(ledger, options)

	def getmetrics(self):			def getmetrics(self):
	metrics = [s.getmetrics() for s in self.stores]			metrics = [s.getmetrics() for s in self.stores]
	return shallowutil.sumdicts(*metrics)			return shallowutil.sumdicts(*metrics)

	class remotefilelogmetadatastore(basestore.basestore):			class remotefilelogmetadatastore(basestore.basestore):
	def getancestors(self, name, node, known=None):			def getancestors(self, name, node, known=None):
	"""Returns as many ancestors as we're aware of.			"""Returns as many ancestors as we're aware of.
	return self.getancestors(name, node)[node]			return self.getancestors(name, node)[node]

	def add(self, name, node, data):			def add(self, name, node, data):
	raise RuntimeError("cannot add to a remote store")			raise RuntimeError("cannot add to a remote store")

	def getmissing(self, keys):			def getmissing(self, keys):
	return keys			return keys

	def markledger(self, ledger):			def markledger(self, ledger, options=None):
	pass			pass

remotefilelog/repack.py

	if incremental:			if incremental:
	cmd.append('--incremental')			cmd.append('--incremental')
	msg = _("(running background incremental repack)\n")			msg = _("(running background incremental repack)\n")
	cmd = ' '.join(map(util.shellquote, cmd))			cmd = ' '.join(map(util.shellquote, cmd))

	repo.ui.warn(msg)			repo.ui.warn(msg)
	runshellcommand(cmd, os.environ)			runshellcommand(cmd, os.environ)

	def fullrepack(repo):			def fullrepack(repo, options=None):
				"""If ``packsonly`` is True, stores creating only loose objects are skipped.
				"""
	if util.safehasattr(repo, 'shareddatastores'):			if util.safehasattr(repo, 'shareddatastores'):
	datasource = contentstore.unioncontentstore(*repo.shareddatastores)			datasource = contentstore.unioncontentstore(
				*repo.shareddatastores)
				durhamUnsubmitted Not Done getpack is somewhat of an internal only function, and not really indicative of a pack store. In theory you could have a datapack (not a datapackstore) in the repo.shareddatastore collection which should be repacked as part of --packsonly, but wouldn't trigger this condition. Instead, it might be nice to create a PackOptions struct. Maybe attach it to the ledger, then each store can look at the options and decide what it should do. Stores related to loose objects could exclude them when they see the packonly option set. durham: getpack is somewhat of an internal only function, and not really indicative of a pack store. In…
	historysource = metadatastore.unionmetadatastore(			historysource = metadatastore.unionmetadatastore(
	*repo.sharedhistorystores,			*repo.sharedhistorystores,
	allowincomplete=True)			allowincomplete=True)

	packpath = shallowutil.getcachepackpath(			packpath = shallowutil.getcachepackpath(
	repo,			repo,
	constants.FILEPACK_CATEGORY)			constants.FILEPACK_CATEGORY)
	_runrepack(repo, datasource, historysource, packpath,			_runrepack(repo, datasource, historysource, packpath,
	constants.FILEPACK_CATEGORY)			constants.FILEPACK_CATEGORY, options=options)

	if repo.ui.configbool('treemanifest', 'server'):			if repo.ui.configbool('treemanifest', 'server'):
	treemfmod = extensions.find('treemanifest')			treemfmod = extensions.find('treemanifest')
	treemfmod.serverrepack(repo)			treemfmod.serverrepack(repo)
	elif util.safehasattr(repo.manifestlog, 'datastore'):			elif util.safehasattr(repo.manifestlog, 'datastore'):
	localdata, shareddata = _getmanifeststores(repo)			localdata, shareddata = _getmanifeststores(repo)
	lpackpath, ldstores, lhstores = localdata			lpackpath, ldstores, lhstores = localdata
	spackpath, sdstores, shstores = shareddata			spackpath, sdstores, shstores = shareddata

	# Repack the shared manifest store			# Repack the shared manifest store
	datasource = contentstore.unioncontentstore(*sdstores)			datasource = contentstore.unioncontentstore(*sdstores)
	historysource = metadatastore.unionmetadatastore(			historysource = metadatastore.unionmetadatastore(
	*shstores,			*shstores,
	allowincomplete=True)			allowincomplete=True)
	_runrepack(repo, datasource, historysource, spackpath,			_runrepack(repo, datasource, historysource, spackpath,
	constants.TREEPACK_CATEGORY)			constants.TREEPACK_CATEGORY, options=options)

	# Repack the local manifest store			# Repack the local manifest store
	datasource = contentstore.unioncontentstore(			datasource = contentstore.unioncontentstore(
	*ldstores,			*ldstores,
	allowincomplete=True)			allowincomplete=True)
	historysource = metadatastore.unionmetadatastore(			historysource = metadatastore.unionmetadatastore(
	*lhstores,			*lhstores,
	allowincomplete=True)			allowincomplete=True)
	_runrepack(repo, datasource, historysource, lpackpath,			_runrepack(repo, datasource, historysource, lpackpath,
	constants.TREEPACK_CATEGORY)			constants.TREEPACK_CATEGORY, options=options)


	def incrementalrepack(repo):			def incrementalrepack(repo, options=None):
	"""This repacks the repo by looking at the distribution of pack files in the			"""This repacks the repo by looking at the distribution of pack files in the
	repo and performing the most minimal repack to keep the repo in good shape.			repo and performing the most minimal repack to keep the repo in good shape.
	"""			"""
	if util.safehasattr(repo, 'shareddatastores'):			if util.safehasattr(repo, 'shareddatastores'):
	packpath = shallowutil.getcachepackpath(			packpath = shallowutil.getcachepackpath(
	repo,			repo,
	constants.FILEPACK_CATEGORY)			constants.FILEPACK_CATEGORY)
	_incrementalrepack(repo,			_incrementalrepack(repo,
	repo.shareddatastores,			repo.shareddatastores,
	repo.sharedhistorystores,			repo.sharedhistorystores,
	packpath,			packpath,
	constants.FILEPACK_CATEGORY)			constants.FILEPACK_CATEGORY,
				options=options)

	if repo.ui.configbool('treemanifest', 'server'):			if repo.ui.configbool('treemanifest', 'server'):
	treemfmod = extensions.find('treemanifest')			treemfmod = extensions.find('treemanifest')
	treemfmod.serverrepack(repo, incremental=True)			treemfmod.serverrepack(repo, incremental=True)
	elif util.safehasattr(repo.manifestlog, 'datastore'):			elif util.safehasattr(repo.manifestlog, 'datastore'):
	localdata, shareddata = _getmanifeststores(repo)			localdata, shareddata = _getmanifeststores(repo)
	lpackpath, ldstores, lhstores = localdata			lpackpath, ldstores, lhstores = localdata
	spackpath, sdstores, shstores = shareddata			spackpath, sdstores, shstores = shareddata

	# Repack the shared manifest store			# Repack the shared manifest store
	_incrementalrepack(repo,			_incrementalrepack(repo,
	sdstores,			sdstores,
	shstores,			shstores,
	spackpath,			spackpath,
	constants.TREEPACK_CATEGORY)			constants.TREEPACK_CATEGORY,
				options=options)

	# Repack the local manifest store			# Repack the local manifest store
	_incrementalrepack(repo,			_incrementalrepack(repo,
	ldstores,			ldstores,
	lhstores,			lhstores,
	lpackpath,			lpackpath,
	constants.TREEPACK_CATEGORY,			constants.TREEPACK_CATEGORY,
	allowincompletedata=True)			allowincompletedata=True,
				options=options)

	def _getmanifeststores(repo):			def _getmanifeststores(repo):
	shareddatastores = repo.manifestlog.shareddatastores			shareddatastores = repo.manifestlog.shareddatastores
	localdatastores = repo.manifestlog.localdatastores			localdatastores = repo.manifestlog.localdatastores
	sharedhistorystores = repo.manifestlog.sharedhistorystores			sharedhistorystores = repo.manifestlog.sharedhistorystores
	localhistorystores = repo.manifestlog.localhistorystores			localhistorystores = repo.manifestlog.localhistorystores

	sharedpackpath = shallowutil.getcachepackpath(repo,			sharedpackpath = shallowutil.getcachepackpath(repo,
	(sharedpackpath, shareddatastores, sharedhistorystores))			(sharedpackpath, shareddatastores, sharedhistorystores))

	def _topacks(packpath, files, constructor):			def _topacks(packpath, files, constructor):
	paths = list(os.path.join(packpath, p) for p in files)			paths = list(os.path.join(packpath, p) for p in files)
	packs = list(constructor(p) for p in paths)			packs = list(constructor(p) for p in paths)
	return packs			return packs

	def _incrementalrepack(repo, datastore, historystore, packpath, category,			def _incrementalrepack(repo, datastore, historystore, packpath, category,
	allowincompletedata=False):			allowincompletedata=False, options=None):
	shallowutil.mkstickygroupdir(repo.ui, packpath)			shallowutil.mkstickygroupdir(repo.ui, packpath)

	files = osutil.listdir(packpath, stat=True)			files = osutil.listdir(packpath, stat=True)

	datapacks = _topacks(packpath,			datapacks = _topacks(packpath,
	_computeincrementaldatapack(repo.ui, files),			_computeincrementaldatapack(repo.ui, files),
	datapack.datapack)			datapack.datapack)
	datapacks.extend(s for s in datastore			datapacks.extend(s for s in datastore
	*datapacks,			*datapacks,
	allowincomplete=allowincompletedata),			allowincomplete=allowincompletedata),
	metadatastore.unionmetadatastore(			metadatastore.unionmetadatastore(
	*historypacks,			*historypacks,
	allowincomplete=True),			allowincomplete=True),
	packpath, category,			packpath, category,
	fullhistory=metadatastore.unionmetadatastore(			fullhistory=metadatastore.unionmetadatastore(
	*allhistorypacks,			*allhistorypacks,
	allowincomplete=True))			allowincomplete=True),
				options=options)

	def _computeincrementaldatapack(ui, files):			def _computeincrementaldatapack(ui, files):
	"""Given a set of pack files and a set of generation size limits, this			"""Given a set of pack files and a set of generation size limits, this
	function computes the list of files that should be packed as part of an			function computes the list of files that should be packed as part of an
	incremental repack.			incremental repack.

	It tries to strike a balance between keeping incremental repacks cheap (i.e.			It tries to strike a balance between keeping incremental repacks cheap (i.e.
	packing small things when possible, and rolling the packs up to the big ones			packing small things when possible, and rolling the packs up to the big ones

	# If there aren't any good candidates for a repack,			# If there aren't any good candidates for a repack,
	# repack the two largest ones.			# repack the two largest ones.
	if not chosenpacks and len(generations[0]) > 1:			if not chosenpacks and len(generations[0]) > 1:
	chosenpacks = generations[0]			chosenpacks = generations[0]

	return chosenpacks			return chosenpacks

	def _runrepack(repo, data, history, packpath, category, fullhistory=None):			def _runrepack(repo, data, history, packpath, category, fullhistory=None,
				options=None):
	shallowutil.mkstickygroupdir(repo.ui, packpath)			shallowutil.mkstickygroupdir(repo.ui, packpath)

	def isold(repo, filename, node):			def isold(repo, filename, node):
	"""Check if the file node is older than a limit.			"""Check if the file node is older than a limit.
	Unless a limit is specified in the config the default limit is taken.			Unless a limit is specified in the config the default limit is taken.
	"""			"""
	filectx = repo.filectx(filename, fileid=node)			filectx = repo.filectx(filename, fileid=node)
	filetime = repo[filectx.linkrev()].date()			filetime = repo[filectx.linkrev()].date()

	# Currently default TTL limit is 30 days			# Currently default TTL limit is 30 days
	defaultlimit = 60 * 60 * 24 * 30			defaultlimit = 60 * 60 * 24 * 30
	ttl = repo.ui.configint('remotefilelog', 'nodettl', defaultlimit)			ttl = repo.ui.configint('remotefilelog', 'nodettl', defaultlimit)

	limit = time.time() - ttl			limit = time.time() - ttl
	return filetime[0] < limit			return filetime[0] < limit

	garbagecollect = repo.ui.configbool('remotefilelog', 'gcrepack')			garbagecollect = repo.ui.configbool('remotefilelog', 'gcrepack')
	if not fullhistory:			if not fullhistory:
	fullhistory = history			fullhistory = history
	packer = repacker(repo, data, history, fullhistory, category,			packer = repacker(repo, data, history, fullhistory, category,
	gc=garbagecollect,			gc=garbagecollect, isold=isold, options=options)
	isold=isold)

	# internal config: remotefilelog.datapackversion			# internal config: remotefilelog.datapackversion
	dv = repo.ui.configint('remotefilelog', 'datapackversion', 0)			dv = repo.ui.configint('remotefilelog', 'datapackversion', 0)

	with datapack.mutabledatapack(repo.ui, packpath, version=dv) as dpack:			with datapack.mutabledatapack(repo.ui, packpath, version=dv) as dpack:
	with historypack.mutablehistorypack(repo.ui, packpath) as hpack:			with historypack.mutablehistorypack(repo.ui, packpath) as hpack:
	try:			try:
	packer.run(dpack, hpack)			packer.run(dpack, hpack)

	return keepkeys			return keepkeys

	class repacker(object):			class repacker(object):
	"""Class for orchestrating the repack of data and history information into a			"""Class for orchestrating the repack of data and history information into a
	new format.			new format.
	"""			"""
	def __init__(self, repo, data, history, fullhistory, category, gc=False,			def __init__(self, repo, data, history, fullhistory, category, gc=False,
	isold=None):			isold=None, options=None):
	self.repo = repo			self.repo = repo
	self.data = data			self.data = data
	self.history = history			self.history = history
	self.fullhistory = fullhistory			self.fullhistory = fullhistory
	self.unit = constants.getunits(category)			self.unit = constants.getunits(category)
	self.garbagecollect = gc			self.garbagecollect = gc
				self.options = options
	if self.garbagecollect:			if self.garbagecollect:
	if not isold:			if not isold:
	raise ValueError("Function 'isold' is not properly specified")			raise ValueError("Function 'isold' is not properly specified")
	# use (filename, node) tuple as a keepset key			# use (filename, node) tuple as a keepset key
	self.keepkeys = keepset(repo, lambda f, n : (f, n))			self.keepkeys = keepset(repo, lambda f, n : (f, n))
	self.isold = isold			self.isold = isold

	def run(self, targetdata, targethistory):			def run(self, targetdata, targethistory):
	ledger = repackledger()			ledger = repackledger()

	with self.repo._lock(self.repo.svfs, "repacklock", False, None,			with self.repo._lock(self.repo.svfs, "repacklock", False, None,
	None, _('repacking %s') % self.repo.origroot):			None, _('repacking %s') % self.repo.origroot):
	self.repo.hook('prerepack')			self.repo.hook('prerepack')

	# Populate ledger from source			# Populate ledger from source
	self.data.markledger(ledger)			self.data.markledger(ledger, options=self.options)
	self.history.markledger(ledger)			self.history.markledger(ledger, options=self.options)

	# Run repack			# Run repack
	self.repackdata(ledger, targetdata)			self.repackdata(ledger, targetdata)
	self.repackhistory(ledger, targethistory)			self.repackhistory(ledger, targethistory)

	# Call cleanup on each source			# Call cleanup on each source
	for source in ledger.sources:			for source in ledger.sources:
	source.cleanup(ledger)			source.cleanup(ledger)

remotefilelog/wirepack.py


	for node, deltabase, delta in readdeltas(fh):			for node, deltabase, delta in readdeltas(fh):
	data[(filename, node)] = (delta, deltabase)			data[(filename, node)] = (delta, deltabase)
	count += 1			count += 1

	if count == 0 and filename == '':			if count == 0 and filename == '':
	break			break

	def markledger(self, ledger):			def markledger(self, ledger, options=None):
	pass			pass

	def cleanup(self, ledger):			def cleanup(self, ledger):
	pass			pass

tests/test-remotefilelog-repack.t

	$ find $CACHEDIR -type f \| sort			$ find $CACHEDIR -type f \| sort
	$TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/d4a3ed9310e5bd9887e3bf779da5077efab28216			$TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/d4a3ed9310e5bd9887e3bf779da5077efab28216
	$TESTTMP/hgcache/master/packs/276d308429d0303762befa376788300f0310f90e.histidx			$TESTTMP/hgcache/master/packs/276d308429d0303762befa376788300f0310f90e.histidx
	$TESTTMP/hgcache/master/packs/276d308429d0303762befa376788300f0310f90e.histpack			$TESTTMP/hgcache/master/packs/276d308429d0303762befa376788300f0310f90e.histpack
	$TESTTMP/hgcache/master/packs/8e25dec685d5e0bb1f1b39df3acebda0e0d75c6e.dataidx			$TESTTMP/hgcache/master/packs/8e25dec685d5e0bb1f1b39df3acebda0e0d75c6e.dataidx
	$TESTTMP/hgcache/master/packs/8e25dec685d5e0bb1f1b39df3acebda0e0d75c6e.datapack			$TESTTMP/hgcache/master/packs/8e25dec685d5e0bb1f1b39df3acebda0e0d75c6e.datapack
	$TESTTMP/hgcache/repos			$TESTTMP/hgcache/repos

				# First assert that with --packsonly, the loose object will be ignored:

				$ hg repack --packsonly

				$ find $CACHEDIR -type f \| sort
				$TESTTMP/hgcache/master/11/f6ad8ec52a2984abaafd7c3b516503785c2072/d4a3ed9310e5bd9887e3bf779da5077efab28216
				$TESTTMP/hgcache/master/packs/276d308429d0303762befa376788300f0310f90e.histidx
				$TESTTMP/hgcache/master/packs/276d308429d0303762befa376788300f0310f90e.histpack
				$TESTTMP/hgcache/master/packs/8e25dec685d5e0bb1f1b39df3acebda0e0d75c6e.dataidx
				$TESTTMP/hgcache/master/packs/8e25dec685d5e0bb1f1b39df3acebda0e0d75c6e.datapack
				$TESTTMP/hgcache/repos

	$ hg repack --traceback			$ hg repack --traceback

	$ find $CACHEDIR -type f \| sort			$ find $CACHEDIR -type f \| sort
	$TESTTMP/hgcache/master/packs/077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histidx			$TESTTMP/hgcache/master/packs/077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histidx
	$TESTTMP/hgcache/master/packs/077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histpack			$TESTTMP/hgcache/master/packs/077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histpack
	$TESTTMP/hgcache/master/packs/935861cae0be6ce41a0d47a529e4d097e9e68a69.dataidx			$TESTTMP/hgcache/master/packs/935861cae0be6ce41a0d47a529e4d097e9e68a69.dataidx
	$TESTTMP/hgcache/master/packs/935861cae0be6ce41a0d47a529e4d097e9e68a69.datapack			$TESTTMP/hgcache/master/packs/935861cae0be6ce41a0d47a529e4d097e9e68a69.datapack
	$TESTTMP/hgcache/repos			$TESTTMP/hgcache/repos

tests/test-treemanifest-repack.t

	> server=True			> server=True
	> EOF			> EOF
	$ hg backfilltree			$ hg backfilltree
	$ cat .hg/store/fncache \| sort			$ cat .hg/store/fncache \| sort
	data/a.i			data/a.i
	data/dir/b.i			data/dir/b.i
	meta/dir/00manifest.i			meta/dir/00manifest.i

	$ hg repack			Also verify that --packsonly doesn't stop packs from being repacked:
				$ hg repack --packsonly
				durhamUnsubmitted Not Done I think this test is checking that hg repack can repack from revlogs into pack files. So I would actually expect this to fail if you set --packsonly, since in that case the revlogs should not be inspected. durham: I think this test is checking that hg repack can repack from revlogs into pack files. So I…
				phillcoAuthorUnsubmitted Not Done Ah, it's because I didn't patch serverepack(). I guess I should. phillco: Ah, it's because I didn't patch serverepack(). I guess I should.
	$ ls .hg/cache/packs/manifests			$ ls .hg/cache/packs/manifests
	56e8c6f0ca2a324b8b5ca1a2730323a1b4d0793a.dataidx			56e8c6f0ca2a324b8b5ca1a2730323a1b4d0793a.dataidx
	56e8c6f0ca2a324b8b5ca1a2730323a1b4d0793a.datapack			56e8c6f0ca2a324b8b5ca1a2730323a1b4d0793a.datapack
	7535b6084226436bbdff33043969e7fa963e8428.histidx			7535b6084226436bbdff33043969e7fa963e8428.histidx
	7535b6084226436bbdff33043969e7fa963e8428.histpack			7535b6084226436bbdff33043969e7fa963e8428.histpack
	$ hg debugdatapack .hg/cache/packs/manifests/*.datapack			$ hg debugdatapack .hg/cache/packs/manifests/*.datapack
	.hg/cache/packs/manifests/56e8c6f0ca2a324b8b5ca1a2730323a1b4d0793a:			.hg/cache/packs/manifests/56e8c6f0ca2a324b8b5ca1a2730323a1b4d0793a:
	(empty name):			(empty name):

treemanifest/init.py

	return [(name, node, None, nullid, revision)]			return [(name, node, None, nullid, revision)]

	def add(self, name, node, data):			def add(self, name, node, data):
	raise RuntimeError("cannot add to a remote store")			raise RuntimeError("cannot add to a remote store")

	def getmissing(self, keys):			def getmissing(self, keys):
	return keys			return keys

	def markledger(self, ledger):			def markledger(self, ledger, options=None):
	pass			pass

	def getmetrics(self):			def getmetrics(self):
	return {}			return {}

	def serverrepack(repo, incremental=False):			def serverrepack(repo, incremental=False):
	packpath = repo.vfs.join('cache/packs/%s' % PACK_CATEGORY)			packpath = repo.vfs.join('cache/packs/%s' % PACK_CATEGORY)

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	3101		Oct 26 2017, 12:37 AM	★	★
Diff 2	3330		Nov 7 2017, 11:44 PM	★	★
Diff 3	3336		Nov 8 2017, 12:02 PM	★	★
Diff 4	3377		Nov 9 2017, 1:16 PM	★	★