This is an archive of the discontinued Mercurial Phabricator instance.

Differential D1588

incremental-repack: do not repack files over a configuration based size
ClosedPublic

Authored by singhsrb on Dec 4 2017, 8:11 PM.
Tags
None
Subscribers
durham

Details

Reviewers
durham
Group Reviewers
Restricted Project
Commits
rFBHGX69ccc796d4ba: incremental-repack: do not repack files over a configuration based size
Summary

Incrementally repacking too big files results in more problems than
solving them like the repack taking a long time to finish and eventually
culminating in a one huge pack file. Therefore, it is preferable to ignore
files over a certain size during the incremental repack.

Test Plan

Added a test and ran all the tests.

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

singhsrb created this revision.Dec 4 2017, 8:11 PM
Herald added a reviewer: Restricted Project. · View Herald TranscriptDec 4 2017, 8:11 PM
durham accepted this revision.Dec 4 2017, 11:16 PM
durham added a subscriber: durham.
durham added inline comments.
remotefilelog/repack.py
250

Why 400MB? Seems low. What kind of numbers are we seeing for history pack size maximums?

This revision is now accepted and ready to land.Dec 4 2017, 11:16 PM
singhsrb added inline comments.Dec 5 2017, 2:46 PM
remotefilelog/repack.py
250

I don't have a distribution to support the claim but usually the history packs are not a problem. They are usually lesser in number (1 or 2) even when there are plenty of datapacks or huge datapacks and max out at 1GB from all the instances I have seen. I am fairly confident that the number of > 400 MB history packs after this change would be quite small (~10-20).

Closed by commit rFBHGX69ccc796d4ba: incremental-repack: do not repack files over a configuration based size (authored by singhsrb). · Explain WhyDec 5 2017, 4:05 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 4132

remotefilelog/__init__.py

pack files required to be considered part of a generation. In particular, pack files required to be considered part of a generation. In particular,
minimum number of packs files > gencountlimit. minimum number of packs files > gencountlimit.
``remotefilelog.data.generations`` list for specifying the lower bound of ``remotefilelog.data.generations`` list for specifying the lower bound of
each generation of the data pack files. For example, list ['100MB','1MB'] each generation of the data pack files. For example, list ['100MB','1MB']
or ['1MB', '100MB'] will lead to three generations: [0, 1MB), [ or ['1MB', '100MB'] will lead to three generations: [0, 1MB), [
1MB, 100MB) and [100MB, infinity). 1MB, 100MB) and [100MB, infinity).
``remotefilelog.data.maxrepackpacks`` the maximum number of pack files to ``remotefilelog.data.maxrepackpacks`` the maximum number of pack files to
include in an incremental data repack. include in an incremental data repack.
``remotefilelog.data.repackmaxpacksize`` the maximum size of a pack file for
it to be considered for an incremental data repack.
``remotefilelog.data.repacksizelimit`` the maximum total size of pack files ``remotefilelog.data.repacksizelimit`` the maximum total size of pack files
to include in an incremental data repack. to include in an incremental data repack.
``remotefilelog.history.gencountlimit`` constraints the minimum number of ``remotefilelog.history.gencountlimit`` constraints the minimum number of
history pack files required to be considered part of a generation. In history pack files required to be considered part of a generation. In
particular, minimum number of packs files > gencountlimit. particular, minimum number of packs files > gencountlimit.
``remotefilelog.history.generations`` list for specifying the lower bound of ``remotefilelog.history.generations`` list for specifying the lower bound of
each generation of the history pack files. For example, list [ each generation of the history pack files. For example, list [
'100MB', '1MB'] or ['1MB', '100MB'] will lead to three generations: [ '100MB', '1MB'] or ['1MB', '100MB'] will lead to three generations: [
0, 1MB), [1MB, 100MB) and [100MB, infinity). 0, 1MB), [1MB, 100MB) and [100MB, infinity).
``remotefilelog.history.maxrepackpacks`` the maximum number of pack files to ``remotefilelog.history.maxrepackpacks`` the maximum number of pack files to
include in an incremental history repack. include in an incremental history repack.
``remotefilelog.history.repackmaxpacksize`` the maximum size of a pack file
for it to be considered for an incremental history repack.
``remotefilelog.history.repacksizelimit`` the maximum total size of pack ``remotefilelog.history.repacksizelimit`` the maximum total size of pack
files to include in an incremental history repack. files to include in an incremental history repack.
""" """
from . import fileserverclient, remotefilelog, remotefilectx, shallowstore from . import fileserverclient, remotefilelog, remotefilectx, shallowstore
import shallowbundle, debugcommands, remotefilelogserver, shallowverifier import shallowbundle, debugcommands, remotefilelogserver, shallowverifier
import shallowutil, shallowrepo import shallowutil, shallowrepo
import repack as repackmod import repack as repackmod

remotefilelog/repack.py

def _computeincrementaldatapack(ui, files): def _computeincrementaldatapack(ui, files):
opts = { opts = {
'gencountlimit' : ui.configint( 'gencountlimit' : ui.configint(
'remotefilelog', 'data.gencountlimit', 2), 'remotefilelog', 'data.gencountlimit', 2),
'generations' : ui.configlist( 'generations' : ui.configlist(
'remotefilelog', 'data.generations', ['1GB', '100MB', '1MB']), 'remotefilelog', 'data.generations', ['1GB', '100MB', '1MB']),
'maxrepackpacks' : ui.configint( 'maxrepackpacks' : ui.configint(
'remotefilelog', 'data.maxrepackpacks', 50), 'remotefilelog', 'data.maxrepackpacks', 50),
'repackmaxpacksize' : ui.configbytes(
'remotefilelog', 'data.repackmaxpacksize', '4GB'),
'repacksizelimit' : ui.configbytes( 'repacksizelimit' : ui.configbytes(
'remotefilelog', 'data.repacksizelimit', '100MB'), 'remotefilelog', 'data.repacksizelimit', '100MB'),
} }
packfiles = _allpackfileswithsuffix( packfiles = _allpackfileswithsuffix(
files, datapack.PACKSUFFIX, datapack.INDEXSUFFIX) files, datapack.PACKSUFFIX, datapack.INDEXSUFFIX)
return _computeincrementalpack(packfiles, opts) return _computeincrementalpack(packfiles, opts)
def _computeincrementalhistorypack(ui, files): def _computeincrementalhistorypack(ui, files):
opts = { opts = {
'gencountlimit' : ui.configint( 'gencountlimit' : ui.configint(
'remotefilelog', 'history.gencountlimit', 2), 'remotefilelog', 'history.gencountlimit', 2),
'generations' : ui.configlist( 'generations' : ui.configlist(
'remotefilelog', 'history.generations', ['100MB']), 'remotefilelog', 'history.generations', ['100MB']),
'maxrepackpacks' : ui.configint( 'maxrepackpacks' : ui.configint(
'remotefilelog', 'history.maxrepackpacks', 50), 'remotefilelog', 'history.maxrepackpacks', 50),
'repackmaxpacksize' : ui.configbytes(
'remotefilelog', 'history.repackmaxpacksize', '400MB'),
durhamUnsubmitted
Not Done

Why 400MB? Seems low. What kind of numbers are we seeing for history pack size maximums?

durham: Why 400MB? Seems low. What kind of numbers are we seeing for history pack size maximums?
singhsrbAuthorUnsubmitted
Not Done

I don't have a distribution to support the claim but usually the history packs are not a problem. They are usually lesser in number (1 or 2) even when there are plenty of datapacks or huge datapacks and max out at 1GB from all the instances I have seen. I am fairly confident that the number of > 400 MB history packs after this change would be quite small (~10-20).

singhsrb: I don't have a distribution to support the claim but usually the history packs are not a…
'repacksizelimit' : ui.configbytes( 'repacksizelimit' : ui.configbytes(
'remotefilelog', 'history.repacksizelimit', '100MB'), 'remotefilelog', 'history.repacksizelimit', '100MB'),
} }
packfiles = _allpackfileswithsuffix( packfiles = _allpackfileswithsuffix(
files, historypack.PACKSUFFIX, historypack.INDEXSUFFIX) files, historypack.PACKSUFFIX, historypack.INDEXSUFFIX)
return _computeincrementalpack(packfiles, opts) return _computeincrementalpack(packfiles, opts)
# Group the packs by generation (i.e. by size) # Group the packs by generation (i.e. by size)
generations = [] generations = []
for i in xrange(len(limits)): for i in xrange(len(limits)):
generations.append([]) generations.append([])
sizes = {} sizes = {}
for prefix, mode, stat in files: for prefix, mode, stat in files:
size = stat.st_size size = stat.st_size
if size > opts['repackmaxpacksize']:
continue
sizes[prefix] = size sizes[prefix] = size
for i, limit in enumerate(limits): for i, limit in enumerate(limits):
if size > limit: if size > limit:
generations[i].append(prefix) generations[i].append(prefix)
break break
# Steps for picking what packs to repack: # Steps for picking what packs to repack:
# 1. Pick the largest generation with > gencountlimit pack files. # 1. Pick the largest generation with > gencountlimit pack files.

tests/test-remotefilelog-repack.t

-r--r--r-- 65 6c499d21350d79f92fd556b4b7a902569d88e3c9.datapack -r--r--r-- 65 6c499d21350d79f92fd556b4b7a902569d88e3c9.datapack
-r--r--r-- 61 817d294043bd21a3de01f807721971abe45219ce.datapack -r--r--r-- 61 817d294043bd21a3de01f807721971abe45219ce.datapack
-r--r--r-- 63 ff45add45ab3f59c4f75efc6a087d86c821219d6.datapack -r--r--r-- 63 ff45add45ab3f59c4f75efc6a087d86c821219d6.datapack
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 254 077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histpack -r--r--r-- 254 077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histpack
-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack -r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack
-r--r--r-- 172 276d308429d0303762befa376788300f0310f90e.histpack -r--r--r-- 172 276d308429d0303762befa376788300f0310f90e.histpack
-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack -r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack
$ hg repack --incremental
For the data packs, setting the limit for the repackmaxpacksize to be 64 such
that data pack with size 65 is more than the limit. This effectively ensures
that no generation has 3 packs and therefore, no packs are chosen for the
incremental repacking. As for the history packs, setting repackmaxpacksize to be
0 which should always result in no repacking.
$ hg repack --incremental --config remotefilelog.data.repackmaxpacksize=64 \
> --config remotefilelog.history.repackmaxpacksize=0
$ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack
-r--r--r-- 65 6c499d21350d79f92fd556b4b7a902569d88e3c9.datapack
-r--r--r-- 61 817d294043bd21a3de01f807721971abe45219ce.datapack
-r--r--r-- 63 ff45add45ab3f59c4f75efc6a087d86c821219d6.datapack
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 254 077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histpack
-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack
-r--r--r-- 172 276d308429d0303762befa376788300f0310f90e.histpack
-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack
Setting limit for the repackmaxpacksize to be the size of the biggest pack file
which ensures that it is effectively ignored in the incremental repacking.
$ hg repack --incremental --config remotefilelog.data.repackmaxpacksize=65 \
> --config remotefilelog.history.repackmaxpacksize=336
$ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack -r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack
-r--r--r-- 225 8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack -r--r--r-- 225 8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack -r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack
1 gen3 pack, 1 gen0 pack - does nothing 1 gen3 pack, 1 gen0 pack - does nothing
$ hg repack --incremental $ hg repack --incremental