This is an archive of the discontinued Mercurial Phabricator instance.

Differential D500

repack: fix incremental repack for large quantities of packs
ClosedPublic

Authored by durham on Aug 23 2017, 6:19 PM.
Tags
None
Subscribers
quark

Details

Reviewers
quark
Group Reviewers
Restricted Project
Commits
rFBHGX4e56faf2431d: repack: fix incremental repack for large quantities of packs
Summary

Previously incremental repack would only take the top 3 packs in a generation of
packs. For instance, if there were 100 packs between 1KB and 1MB, it would only
repack the first 3, instead of repacking a whole bunch because it's cheap. This
patch fixes it to grab as many packs in a generation as it can fit under the
limit (default of 100MB of pack data).

This will fix an issue where we're seeing users with 100's of tiny pack files
that aren't getting repacked fast enough.

Test Plan

Added a test

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

durham created this revision.Aug 23 2017, 6:19 PM
quark accepted this revision.Aug 24 2017, 12:27 AM
quark added a subscriber: quark.

Great feature!

remotefilelog/repack.py
183

Maybe we can codemod all config to use the new configitem framework someday.

This revision is now accepted and ready to land.Aug 24 2017, 12:27 AM
Closed by commit rFBHGX4e56faf2431d: repack: fix incremental repack for large quantities of packs (authored by durham). · Explain WhyAug 24 2017, 12:45 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 1248

remotefilelog/__init__.py

``remotefilelog.bgprefetchrevs`` specifies revisions to fetch on commit and ``remotefilelog.bgprefetchrevs`` specifies revisions to fetch on commit and
update, and on other commands that use them. Different from pullprefetch. update, and on other commands that use them. Different from pullprefetch.
``remotefilelog.gcrepack`` does garbage collection during repack when True ``remotefilelog.gcrepack`` does garbage collection during repack when True
``remotefilelog.nodettl`` specifies maximum TTL of a node in seconds before ``remotefilelog.nodettl`` specifies maximum TTL of a node in seconds before
it is garbage collected it is garbage collected
``remotefilelog.repackonhggc`` runs repack on hg gc when True ``remotefilelog.repackonhggc`` runs repack on hg gc when True
``remotefilelog.prefetchdays`` specifies the maximum age of a commit in ``remotefilelog.prefetchdays`` specifies the maximum age of a commit in
days after which it is no longer prefetched. days after which it is no longer prefetched.
``remotefilelog.data.maxrepackpacks`` the maximum number of pack files to
include in an incremental data repack.
``remotefilelog.history.maxrepackpacks`` the maximum number of pack files to
include in an incremental history repack.
""" """
from . import fileserverclient, remotefilelog, remotefilectx, shallowstore from . import fileserverclient, remotefilelog, remotefilectx, shallowstore
import shallowbundle, debugcommands, remotefilelogserver, shallowverifier import shallowbundle, debugcommands, remotefilelogserver, shallowverifier
import shallowutil, shallowrepo import shallowutil, shallowrepo
import repack as repackmod import repack as repackmod
from mercurial.node import hex from mercurial.node import hex
from mercurial.i18n import _ from mercurial.i18n import _

remotefilelog/repack.py

['1GB', '100MB', '1MB']) ['1GB', '100MB', '1MB'])
generations = list(sorted((util.sizetoint(s) for s in generations), generations = list(sorted((util.sizetoint(s) for s in generations),
reverse=True)) reverse=True))
generations.append(0) generations.append(0)
gencountlimit = ui.configint('remotefilelog', 'data.gencountlimit', 2) gencountlimit = ui.configint('remotefilelog', 'data.gencountlimit', 2)
repacksizelimit = ui.configbytes('remotefilelog', 'data.repacksizelimit', repacksizelimit = ui.configbytes('remotefilelog', 'data.repacksizelimit',
'100MB') '100MB')
maxrepackpacks = ui.configint('remotefilelog', 'data.maxrepackpacks', 50)
quarkUnsubmitted
Not Done

Maybe we can codemod all config to use the new configitem framework someday.

quark: Maybe we can codemod all `config` to use the new `configitem` framework someday.
return _computeincrementalpack(ui, files, generations, datapack.PACKSUFFIX, return _computeincrementalpack(ui, files, generations, datapack.PACKSUFFIX,
datapack.INDEXSUFFIX, gencountlimit, repacksizelimit) datapack.INDEXSUFFIX, gencountlimit,
repacksizelimit, maxrepackpacks)
def _computeincrementalhistorypack(ui, files): def _computeincrementalhistorypack(ui, files):
generations = ui.configlist("remotefilelog", "history.generations", generations = ui.configlist("remotefilelog", "history.generations",
['100MB']) ['100MB'])
generations = list(sorted((util.sizetoint(s) for s in generations), generations = list(sorted((util.sizetoint(s) for s in generations),
reverse=True)) reverse=True))
generations.append(0) generations.append(0)
gencountlimit = ui.configint('remotefilelog', 'history.gencountlimit', 2) gencountlimit = ui.configint('remotefilelog', 'history.gencountlimit', 2)
repacksizelimit = ui.configbytes('remotefilelog', 'history.repacksizelimit', repacksizelimit = ui.configbytes('remotefilelog', 'history.repacksizelimit',
'100MB') '100MB')
maxrepackpacks = ui.configint('remotefilelog', 'history.maxrepackpacks', 50)
return _computeincrementalpack(ui, files, generations, return _computeincrementalpack(ui, files, generations,
historypack.PACKSUFFIX, historypack.INDEXSUFFIX, gencountlimit, historypack.PACKSUFFIX, historypack.INDEXSUFFIX, gencountlimit,
repacksizelimit) repacksizelimit, maxrepackpacks)
def _computeincrementalpack(ui, files, limits, packsuffix, indexsuffix, def _computeincrementalpack(ui, files, limits, packsuffix, indexsuffix,
gencountlimit, repacksizelimit): gencountlimit, repacksizelimit, maxrepackpacks):
# Group the packs by generation (i.e. by size) # Group the packs by generation (i.e. by size)
generations = [] generations = []
for i in xrange(len(limits)): for i in xrange(len(limits)):
generations.append([]) generations.append([])
sizes = {} sizes = {}
fileset = set(fn for fn, mode, stat in files) fileset = set(fn for fn, mode, stat in files)
for filename, mode, stat in files: for filename, mode, stat in files:
if not filename.endswith(packsuffix): if not filename.endswith(packsuffix):
continue continue
prefix = filename[:-len(packsuffix)] prefix = filename[:-len(packsuffix)]
# Don't process a pack if it doesn't have an index. # Don't process a pack if it doesn't have an index.
if (prefix + indexsuffix) not in fileset: if (prefix + indexsuffix) not in fileset:
continue continue
size = stat.st_size size = stat.st_size
sizes[prefix] = size sizes[prefix] = size
for i, limit in enumerate(limits): for i, limit in enumerate(limits):
if size > limit: if size > limit:
generations[i].append(prefix) generations[i].append(prefix)
break break
# Find the largest generation with more than 2 packs and repack it. # Steps for picking what packs to repack:
# 1. Pick the largest generation with >2 pack files.
# 2. Take the smallest three packs.
# 3. While total-size-of-packs < repacksizelimit: add another pack
# Find the largest generation with more than gencountlimit packs
genpacks = []
for i, limit in enumerate(limits): for i, limit in enumerate(limits):
if len(generations[i]) > gencountlimit: if len(generations[i]) > gencountlimit:
# Try to repack 3 things at once. This means if we run an # Sort to be smallest last, for easy popping later
# incremental repack right after we add a new pack file, we'll still genpacks.extend(sorted(generations[i], reverse=True,
# decrease the total number of pack files. key=lambda x: sizes[x]))
count = 3
if sum(sizes[n] for n in generations[i]) < repacksizelimit:
count = len(generations[i])
return sorted(generations[i], key=lambda x: sizes[x])[:count]
# If no generation has more than 2 packs, repack as many as fit into the
# limit
small = set().union(*generations[1:])
if len(small) > 1:
total = 0
packs = []
for pack in sorted(small, key=lambda x: sizes[x]):
size = sizes[pack]
if total + size < repacksizelimit:
packs.append(pack)
total += size
else:
break break
if len(packs) > 1: # Take as many packs from the generation as we can
return packs chosenpacks = genpacks[-3:]
genpacks = genpacks[:-3]
# If there aren't small ones to repack, repack the two largest ones. repacksize = sum(sizes[n] for n in chosenpacks)
if len(generations[0]) > 1: while (repacksize < repacksizelimit and genpacks and
return generations[0] len(chosenpacks) < maxrepackpacks):
chosenpacks.append(genpacks.pop())
repacksize += sizes[chosenpacks[-1]]
# If there aren't any good candidates for a repack,
# repack the two largest ones.
if not chosenpacks and len(generations[0]) > 1:
chosenpacks = generations[0]
return [] return chosenpacks
def _runrepack(repo, data, history, packpath, category): def _runrepack(repo, data, history, packpath, category):
shallowutil.mkstickygroupdir(repo.ui, packpath) shallowutil.mkstickygroupdir(repo.ui, packpath)
def isold(repo, filename, node): def isold(repo, filename, node):
"""Check if the file node is older than a limit. """Check if the file node is older than a limit.
Unless a limit is specified in the config the default limit is taken. Unless a limit is specified in the config the default limit is taken.
""" """

tests/test-treemanifest-repack.t

$ echo b >> dir/b && hg commit -Aqm 'modify dir/b' $ echo b >> dir/b && hg commit -Aqm 'modify dir/b'
$ ls_l .hg/store/packs/manifests | grep datapack $ ls_l .hg/store/packs/manifests | grep datapack
-r--r--r-- 374 201094db51b761cd78352c055b3135178aadfec5.datapack -r--r--r-- 374 201094db51b761cd78352c055b3135178aadfec5.datapack
-r--r--r-- 248 21501384df03b8489b366c5218be639fa08830e4.datapack -r--r--r-- 248 21501384df03b8489b366c5218be639fa08830e4.datapack
-r--r--r-- 248 347263bf1efbdb5bf7e1d1565b6b504073fb9093.datapack -r--r--r-- 248 347263bf1efbdb5bf7e1d1565b6b504073fb9093.datapack
-r--r--r-- 248 544a3b46a61732209116ae50847ec333b75e3765.datapack -r--r--r-- 248 544a3b46a61732209116ae50847ec333b75e3765.datapack
-r--r--r-- 248 863908ef8149261ab0d891c2344d8e8766c39441.datapack -r--r--r-- 248 863908ef8149261ab0d891c2344d8e8766c39441.datapack
-r--r--r-- 248 d7e689a91ac63385be120a118af9ce8663748f28.datapack -r--r--r-- 248 d7e689a91ac63385be120a118af9ce8663748f28.datapack
$ cd .hg/store/packs/manifests
$ cp d7e689a91ac63385be120a118af9ce8663748f28.datapack x7e689a91ac63385be120a118af9ce8663748f28.datapack
$ cp d7e689a91ac63385be120a118af9ce8663748f28.dataidx x7e689a91ac63385be120a118af9ce8663748f28.dataidx
$ cp 21501384df03b8489b366c5218be639fa08830e4.datapack x1501384df03b8489b366c5218be639fa08830e4.datapack
$ cp 21501384df03b8489b366c5218be639fa08830e4.dataidx x1501384df03b8489b366c5218be639fa08830e4.dataidx
$ cp 347263bf1efbdb5bf7e1d1565b6b504073fb9093.datapack x47263bf1efbdb5bf7e1d1565b6b504073fb9093.datapack
$ cp 347263bf1efbdb5bf7e1d1565b6b504073fb9093.dataidx x47263bf1efbdb5bf7e1d1565b6b504073fb9093.dataidx
$ cd ../../../../
- repack incremental kicks in once there are a number of packs - repack incremental kicks in once there are a number of packs
$ hg repack --incremental --config remotefilelog.data.generations=300,200 - (set the repacksizelimit so that we test that we only repack up to 1500 bytes,
- and it leaves one x** datapack behind)
$ hg repack --incremental --config remotefilelog.data.generations=300,200 --config remotefilelog.data.repacksizelimit=1500B
$ ls_l .hg/store/packs/manifests | grep datapack $ ls_l .hg/store/packs/manifests | grep datapack
-r--r--r-- 374 201094db51b761cd78352c055b3135178aadfec5.datapack -r--r--r-- 374 201094db51b761cd78352c055b3135178aadfec5.datapack
-r--r--r-- 1188 785af77b59c45dd43e3e0e63929d77665c505387.datapack -r--r--r-- 1188 785af77b59c45dd43e3e0e63929d77665c505387.datapack
-r--r--r-- 248 x47263bf1efbdb5bf7e1d1565b6b504073fb9093.datapack
$ rm -f .hg/store/packs/manifests/x*
Test repacking from revlogs to pack files on the server Test repacking from revlogs to pack files on the server
$ cd ../master $ cd ../master
$ cat >> .hg/hgrc <<EOF $ cat >> .hg/hgrc <<EOF
> [extensions] > [extensions]
> treemanifest=$TESTDIR/../treemanifest > treemanifest=$TESTDIR/../treemanifest
> remotefilelog=$TESTDIR/../remotefilelog > remotefilelog=$TESTDIR/../remotefilelog