This is an archive of the discontinued Mercurial Phabricator instance.

Differential D1254

debugdatapack: add blob size
ClosedPublic

Authored by phillco on Oct 26 2017, 7:57 PM.
Tags
None
Subscribers
None

Details

Reviewers
durham
Group Reviewers
Restricted Project
Commits
rFBHGXff699b16c0bf: debugdatapack: add blob size
Summary

It's useful to know the size of the resultant object, not just the delta entry.

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

phillco created this revision.Oct 26 2017, 7:57 PM
Herald added a reviewer: Restricted Project. · View Herald TranscriptOct 26 2017, 7:57 PM
phillco updated this revision to Diff 3132.Oct 26 2017, 8:03 PM
phillco updated this revision to Diff 3134.
phillco retitled this revision from repack: add blob size to hg debugdatapack to debugdatapack: add blob sizes.Oct 30 2017, 11:48 PM
phillco added a reviewer: durham.
phillco retitled this revision from debugdatapack: add blob sizes to debugdatapack: add blob size.Oct 30 2017, 11:53 PM
phillco updated this revision to Diff 3166.
durham accepted this revision.Oct 31 2017, 3:55 PM
This revision is now accepted and ready to land.Oct 31 2017, 3:55 PM

Revision Contents
Changeset List

Diff 3166

remotefilelog/debugcommands.py

# debugcommands.py - debug logic for remotefilelog # debugcommands.py - debug logic for remotefilelog
# #
# Copyright 2013 Facebook, Inc. # Copyright 2013 Facebook, Inc.
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
from __future__ import absolute_import from __future__ import absolute_import
from mercurial import error, filelog, revlog from mercurial import error, filelog, revlog
from mercurial.node import bin, hex, nullid, short from mercurial.node import bin, hex, nullid, short
from mercurial.i18n import _ from mercurial.i18n import _
from . import ( from . import (
constants,
datapack, datapack,
fileserverclient, fileserverclient,
historypack, historypack,
shallowrepo, shallowrepo,
shallowutil, shallowutil,
) )
from .lz4wrapper import lz4decompress from .lz4wrapper import lz4decompress
import hashlib, os import hashlib, os
else: else:
hashformatter = short hashformatter = short
hashlen = 14 hashlen = 14
lastfilename = None lastfilename = None
for filename, node, deltabase, deltalen in dpack.iterentries(): for filename, node, deltabase, deltalen in dpack.iterentries():
if filename != lastfilename: if filename != lastfilename:
ui.write("\n%s\n" % filename) ui.write("\n%s\n" % filename)
ui.write("%s%s%s\n" % ( ui.write("%s%s%s%s\n" % (
"Node".ljust(hashlen), "Node".ljust(hashlen),
"Delta Base".ljust(hashlen), "Delta Base".ljust(hashlen),
"Delta Length".ljust(6))) "Delta Length".ljust(14),
"Blob Size".ljust(9)))
lastfilename = filename lastfilename = filename
ui.write("%s %s %s\n" % ( # Metadata could be missing, in which case it will be an empty dict.
hashformatter(node), hashformatter(deltabase), deltalen)) meta = dpack.getmeta(filename, node)
if constants.METAKEYSIZE in meta:
blobsize = meta[constants.METAKEYSIZE]
else:
blobsize = "(missing)"
ui.write("%s %s %s%s\n" % (
hashformatter(node),
hashformatter(deltabase),
str(deltalen).ljust(14),
blobsize))
def dumpdeltachain(ui, deltachain, **opts): def dumpdeltachain(ui, deltachain, **opts):
hashformatter = hex hashformatter = hex
hashlen = 40 hashlen = 40
lastfilename = None lastfilename = None
for filename, node, filename, deltabasenode, delta in deltachain: for filename, node, filename, deltabasenode, delta in deltachain:
if filename != lastfilename: if filename != lastfilename:

tests/test-remotefilelog-bgprefetch.t

$TESTTMP/hgcache/repos $TESTTMP/hgcache/repos
# Ensure that file 'w' was prefetched - it was not part of the update operation and therefore # Ensure that file 'w' was prefetched - it was not part of the update operation and therefore
# could only be downloaded by the background prefetch # could only be downloaded by the background prefetch
$ hg debugdatapack $TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.datapack $ hg debugdatapack $TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.datapack
w w
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
bb6ccd5dceaa 000000000000 2 bb6ccd5dceaa 000000000000 2 2
x x
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
ef95c5376f34 000000000000 3 ef95c5376f34 000000000000 3 3
1406e7411862 ef95c5376f34 14 1406e7411862 ef95c5376f34 14 2
y y
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
076f5e2225b3 000000000000 2 076f5e2225b3 000000000000 2 2
z z
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
69a1b6752270 000000000000 2 69a1b6752270 000000000000 2 2
# background prefetch with repack on commit when wcprevset configured # background prefetch with repack on commit when wcprevset configured
$ cat >> .hg/hgrc <<EOF $ cat >> .hg/hgrc <<EOF
> [remotefilelog] > [remotefilelog]
> bgprefetchrevs=0:: > bgprefetchrevs=0::
> EOF > EOF
$TESTTMP/hgcache/repos $TESTTMP/hgcache/repos
# Ensure that file 'w' was prefetched - it was not part of the commit operation and therefore # Ensure that file 'w' was prefetched - it was not part of the commit operation and therefore
# could only be downloaded by the background prefetch # could only be downloaded by the background prefetch
$ hg debugdatapack $TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.datapack $ hg debugdatapack $TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.datapack
w w
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
bb6ccd5dceaa 000000000000 2 bb6ccd5dceaa 000000000000 2 2
x x
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
ef95c5376f34 000000000000 3 ef95c5376f34 000000000000 3 3
1406e7411862 ef95c5376f34 14 1406e7411862 ef95c5376f34 14 2
y y
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
076f5e2225b3 000000000000 2 076f5e2225b3 000000000000 2 2
z z
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
69a1b6752270 000000000000 2 69a1b6752270 000000000000 2 2
# background prefetch with repack on rebase when wcprevset configured # background prefetch with repack on rebase when wcprevset configured
$ hg up -r 2 $ hg up -r 2
3 files updated, 0 files merged, 3 files removed, 0 files unresolved 3 files updated, 0 files merged, 3 files removed, 0 files unresolved
(leaving bookmark temporary) (leaving bookmark temporary)
$ clearcache $ clearcache
$ find $CACHEDIR -type f | sort $ find $CACHEDIR -type f | sort
$ hg rebase -s temporary -d foo $ hg rebase -s temporary -d foo
rebasing 3:58147a5b5242 "b" (temporary tip) rebasing 3:58147a5b5242 "b" (temporary tip)
saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/58147a5b5242-c3678817-rebase.hg (glob) saved backup bundle to $TESTTMP/shallow/.hg/strip-backup/58147a5b5242-c3678817-rebase.hg (glob)
3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over *s (glob) 3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over *s (glob)
$ sleep 1 $ sleep 1
$ hg debugwaitonprefetch >/dev/null 2>%1 $ hg debugwaitonprefetch >/dev/null 2>%1
$ sleep 1 $ sleep 1
$ hg debugwaitonrepack >/dev/null 2>%1 $ hg debugwaitonrepack >/dev/null 2>%1
# Ensure that file 'y' was prefetched - it was not part of the rebase operation and therefore # Ensure that file 'y' was prefetched - it was not part of the rebase operation and therefore
# could only be downloaded by the background prefetch # could only be downloaded by the background prefetch
$ hg debugdatapack $TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.datapack $ hg debugdatapack $TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.datapack
w w
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
bb6ccd5dceaa 000000000000 2 bb6ccd5dceaa 000000000000 2 2
x x
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
ef95c5376f34 000000000000 3 ef95c5376f34 000000000000 3 3
1406e7411862 ef95c5376f34 14 1406e7411862 ef95c5376f34 14 2
y y
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
076f5e2225b3 000000000000 2 076f5e2225b3 000000000000 2 2
z z
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
69a1b6752270 000000000000 2 69a1b6752270 000000000000 2 2
# Check that foregound prefetch with no arguments blocks until background prefetches finish # Check that foregound prefetch with no arguments blocks until background prefetches finish
$ hg up -r 3 $ hg up -r 3
2 files updated, 0 files merged, 0 files removed, 0 files unresolved 2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ clearcache $ clearcache
$ hg prefetch --repack $ hg prefetch --repack
waiting for lock on prefetching in $TESTTMP/shallow held by process * on host * (glob) (?) waiting for lock on prefetching in $TESTTMP/shallow held by process * on host * (glob) (?)
$TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.dataidx $TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.dataidx
$TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.datapack $TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.datapack
$TESTTMP/hgcache/repos $TESTTMP/hgcache/repos
# Ensure that files were prefetched # Ensure that files were prefetched
$ hg debugdatapack $TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.datapack $ hg debugdatapack $TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.datapack
w w
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
bb6ccd5dceaa 000000000000 2 bb6ccd5dceaa 000000000000 2 2
x x
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
ef95c5376f34 000000000000 3 ef95c5376f34 000000000000 3 3
1406e7411862 ef95c5376f34 14 1406e7411862 ef95c5376f34 14 2
y y
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
076f5e2225b3 000000000000 2 076f5e2225b3 000000000000 2 2
z z
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
69a1b6752270 000000000000 2 69a1b6752270 000000000000 2 2
# Check that foreground prefetch fetches revs specified by '. + draft() + bgprefetchrevs + pullprefetch' # Check that foreground prefetch fetches revs specified by '. + draft() + bgprefetchrevs + pullprefetch'
$ clearcache $ clearcache
$ hg prefetch --repack $ hg prefetch --repack
waiting for lock on prefetching in $TESTTMP/shallow held by process * on host * (glob) (?) waiting for lock on prefetching in $TESTTMP/shallow held by process * on host * (glob) (?)
got lock after * seconds (glob) (?) got lock after * seconds (glob) (?)
(running background incremental repack) (running background incremental repack)
* files fetched over 1 fetches - (* misses, 0.00% hit ratio) over *s (glob) (?) * files fetched over 1 fetches - (* misses, 0.00% hit ratio) over *s (glob) (?)
$ sleep 0.5 $ sleep 0.5
$ hg debugwaitonrepack >/dev/null 2>%1 $ hg debugwaitonrepack >/dev/null 2>%1
$ find $CACHEDIR -type f | sort $ find $CACHEDIR -type f | sort
$TESTTMP/hgcache/master/packs/27c52c105a1ddf8c75143a6b279b04c24b1f4bee.histidx $TESTTMP/hgcache/master/packs/27c52c105a1ddf8c75143a6b279b04c24b1f4bee.histidx
$TESTTMP/hgcache/master/packs/27c52c105a1ddf8c75143a6b279b04c24b1f4bee.histpack $TESTTMP/hgcache/master/packs/27c52c105a1ddf8c75143a6b279b04c24b1f4bee.histpack
$TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.dataidx $TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.dataidx
$TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.datapack $TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.datapack
$TESTTMP/hgcache/repos $TESTTMP/hgcache/repos
# Ensure that files were prefetched # Ensure that files were prefetched
$ hg debugdatapack $TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.datapack $ hg debugdatapack $TESTTMP/hgcache/master/packs/8299d5a1030f073f4adbb3b6bd2ad3bdcc276df0.datapack
w w
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
bb6ccd5dceaa 000000000000 2 bb6ccd5dceaa 000000000000 2 2
x x
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
ef95c5376f34 000000000000 3 ef95c5376f34 000000000000 3 3
1406e7411862 ef95c5376f34 14 1406e7411862 ef95c5376f34 14 2
y y
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
076f5e2225b3 000000000000 2 076f5e2225b3 000000000000 2 2
z z
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
69a1b6752270 000000000000 2 69a1b6752270 000000000000 2 2
# Test that if data was prefetched and repacked we dont need to prefetch it again # Test that if data was prefetched and repacked we dont need to prefetch it again
# It ensures that Mercurial looks not only in loose files but in packs as well # It ensures that Mercurial looks not only in loose files but in packs as well
$ hg prefetch --repack $ hg prefetch --repack
(running background incremental repack) (running background incremental repack)

tests/test-remotefilelog-gcrepack.t

$TESTTMP/hgcache/master/packs/9a2ea858fe2967db9b6ea4c0ca238881cae9d6eb.histpack $TESTTMP/hgcache/master/packs/9a2ea858fe2967db9b6ea4c0ca238881cae9d6eb.histpack
$TESTTMP/hgcache/master/packs/f7a942a6e4673d2c7b697fdd926ca2d153831ca4.datapack $TESTTMP/hgcache/master/packs/f7a942a6e4673d2c7b697fdd926ca2d153831ca4.datapack
# Ensure that all file versions were prefetched # Ensure that all file versions were prefetched
$ hg debugdatapack $TESTTMP/hgcache/master/packs/f7a942a6e4673d2c7b697fdd926ca2d153831ca4.datapack $ hg debugdatapack $TESTTMP/hgcache/master/packs/f7a942a6e4673d2c7b697fdd926ca2d153831ca4.datapack
x x
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
1406e7411862 000000000000 2 1406e7411862 000000000000 2 2
y y
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
50dbc4572b8e 000000000000 3 50dbc4572b8e 000000000000 3 3
076f5e2225b3 50dbc4572b8e 14 076f5e2225b3 50dbc4572b8e 14 2
# Test garbage collection during repack # Test garbage collection during repack
$ cat >> .hg/hgrc <<EOF $ cat >> .hg/hgrc <<EOF
> [remotefilelog] > [remotefilelog]
> bgprefetchrevs=tip > bgprefetchrevs=tip
> gcrepack=True > gcrepack=True
> nodettl=86400 > nodettl=86400
> EOF > EOF
$ hg repack $ hg repack
$ sleep 0.5 $ sleep 0.5
$ hg debugwaitonrepack >/dev/null 2>%1 $ hg debugwaitonrepack >/dev/null 2>%1
$ find $CACHEDIR | sort | grep ".datapack\|.histpack" $ find $CACHEDIR | sort | grep ".datapack\|.histpack"
$TESTTMP/hgcache/master/packs/05baa499c6b07f2bf0ea3d2c8151da1cb86f5e33.datapack $TESTTMP/hgcache/master/packs/05baa499c6b07f2bf0ea3d2c8151da1cb86f5e33.datapack
$TESTTMP/hgcache/master/packs/9a2ea858fe2967db9b6ea4c0ca238881cae9d6eb.histpack $TESTTMP/hgcache/master/packs/9a2ea858fe2967db9b6ea4c0ca238881cae9d6eb.histpack
# Ensure that file 'x' was garbage collected. It should be GCed because it is not in the keepset # Ensure that file 'x' was garbage collected. It should be GCed because it is not in the keepset
# and is old (commit date is 0.0 in tests). Ensure that file 'y' is present as it is in the keepset. # and is old (commit date is 0.0 in tests). Ensure that file 'y' is present as it is in the keepset.
$ hg debugdatapack $TESTTMP/hgcache/master/packs/05baa499c6b07f2bf0ea3d2c8151da1cb86f5e33.datapack $ hg debugdatapack $TESTTMP/hgcache/master/packs/05baa499c6b07f2bf0ea3d2c8151da1cb86f5e33.datapack
y y
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
50dbc4572b8e 000000000000 3 50dbc4572b8e 000000000000 3 3
# Prefetch all data again and repack for later garbage collection # Prefetch all data again and repack for later garbage collection
$ cat >> .hg/hgrc <<EOF $ cat >> .hg/hgrc <<EOF
> [remotefilelog] > [remotefilelog]
> bgprefetchrevs=all() > bgprefetchrevs=all()
> EOF > EOF
$ hg prefetch $ hg prefetch
2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob) 2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over *s (glob)
$ hg repack $ hg repack
$ sleep 0.5 $ sleep 0.5
$ hg debugwaitonrepack >/dev/null 2>%1 $ hg debugwaitonrepack >/dev/null 2>%1
$ find $CACHEDIR | sort | grep ".datapack\|.histpack" $ find $CACHEDIR | sort | grep ".datapack\|.histpack"
$TESTTMP/hgcache/master/packs/9a2ea858fe2967db9b6ea4c0ca238881cae9d6eb.histpack $TESTTMP/hgcache/master/packs/9a2ea858fe2967db9b6ea4c0ca238881cae9d6eb.histpack
$TESTTMP/hgcache/master/packs/f7a942a6e4673d2c7b697fdd926ca2d153831ca4.datapack $TESTTMP/hgcache/master/packs/f7a942a6e4673d2c7b697fdd926ca2d153831ca4.datapack
# Ensure that all file versions were prefetched # Ensure that all file versions were prefetched
$ hg debugdatapack $TESTTMP/hgcache/master/packs/f7a942a6e4673d2c7b697fdd926ca2d153831ca4.datapack $ hg debugdatapack $TESTTMP/hgcache/master/packs/f7a942a6e4673d2c7b697fdd926ca2d153831ca4.datapack
x x
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
1406e7411862 000000000000 2 1406e7411862 000000000000 2 2
y y
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
50dbc4572b8e 000000000000 3 50dbc4572b8e 000000000000 3 3
076f5e2225b3 50dbc4572b8e 14 076f5e2225b3 50dbc4572b8e 14 2
# Test garbage collection during repack. Ensure that new files are not removed even though they are not in the keepset # Test garbage collection during repack. Ensure that new files are not removed even though they are not in the keepset
# For the purposes of the test the TTL of a file is set to current time + 100 seconds. i.e. all commits in tests have # For the purposes of the test the TTL of a file is set to current time + 100 seconds. i.e. all commits in tests have
# a date of 1970 and therefore to prevent garbage collection we have to set nodettl to be farther from 1970 than we are now. # a date of 1970 and therefore to prevent garbage collection we have to set nodettl to be farther from 1970 than we are now.
$ cat >> .hg/hgrc <<EOF $ cat >> .hg/hgrc <<EOF
> [remotefilelog] > [remotefilelog]
> bgprefetchrevs= > bgprefetchrevs=
> nodettl=$(($(date +%s) + 100)) > nodettl=$(($(date +%s) + 100))
> EOF > EOF
$ hg repack $ hg repack
$ sleep 0.5 $ sleep 0.5
$ hg debugwaitonrepack >/dev/null 2>%1 $ hg debugwaitonrepack >/dev/null 2>%1
$ find $CACHEDIR | sort | grep ".datapack\|.histpack" $ find $CACHEDIR | sort | grep ".datapack\|.histpack"
$TESTTMP/hgcache/master/packs/9a2ea858fe2967db9b6ea4c0ca238881cae9d6eb.histpack $TESTTMP/hgcache/master/packs/9a2ea858fe2967db9b6ea4c0ca238881cae9d6eb.histpack
$TESTTMP/hgcache/master/packs/f7a942a6e4673d2c7b697fdd926ca2d153831ca4.datapack $TESTTMP/hgcache/master/packs/f7a942a6e4673d2c7b697fdd926ca2d153831ca4.datapack
# Ensure that all file versions were prefetched # Ensure that all file versions were prefetched
$ hg debugdatapack $TESTTMP/hgcache/master/packs/f7a942a6e4673d2c7b697fdd926ca2d153831ca4.datapack $ hg debugdatapack $TESTTMP/hgcache/master/packs/f7a942a6e4673d2c7b697fdd926ca2d153831ca4.datapack
x x
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
1406e7411862 000000000000 2 1406e7411862 000000000000 2 2
y y
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
50dbc4572b8e 000000000000 3 50dbc4572b8e 000000000000 3 3
076f5e2225b3 50dbc4572b8e 14 076f5e2225b3 50dbc4572b8e 14 2

tests/test-remotefilelog-repack-fast.t

$TESTTMP/hgcache/master/packs/8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack $TESTTMP/hgcache/master/packs/8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack
$TESTTMP/hgcache/repos $TESTTMP/hgcache/repos
# Test debug commands # Test debug commands
$ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack $ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack
x x
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
1bb2e6237e03 000000000000 8 1bb2e6237e03 000000000000 8 8
d4a3ed9310e5 1bb2e6237e03 12 d4a3ed9310e5 1bb2e6237e03 12 6
aee31534993a d4a3ed9310e5 12 aee31534993a d4a3ed9310e5 12 4
$ hg debugdatapack --long $TESTTMP/hgcache/master/packs/*.datapack $ hg debugdatapack --long $TESTTMP/hgcache/master/packs/*.datapack
x x
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
1bb2e6237e035c8f8ef508e281f1ce075bc6db72 0000000000000000000000000000000000000000 8 1bb2e6237e035c8f8ef508e281f1ce075bc6db72 0000000000000000000000000000000000000000 8 8
d4a3ed9310e5bd9887e3bf779da5077efab28216 1bb2e6237e035c8f8ef508e281f1ce075bc6db72 12 d4a3ed9310e5bd9887e3bf779da5077efab28216 1bb2e6237e035c8f8ef508e281f1ce075bc6db72 12 6
aee31534993a501858fb6dd96a065671922e7d51 d4a3ed9310e5bd9887e3bf779da5077efab28216 12 aee31534993a501858fb6dd96a065671922e7d51 d4a3ed9310e5bd9887e3bf779da5077efab28216 12 4
$ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack --node d4a3ed9310e5bd9887e3bf779da5077efab28216 $ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack --node d4a3ed9310e5bd9887e3bf779da5077efab28216
x x
Node Delta Base Delta SHA1 Delta Length Node Delta Base Delta SHA1 Delta Length
d4a3ed9310e5bd9887e3bf779da5077efab28216 1bb2e6237e035c8f8ef508e281f1ce075bc6db72 77029ab56e83ea2115dd53ff87483682abe5d7ca 12 d4a3ed9310e5bd9887e3bf779da5077efab28216 1bb2e6237e035c8f8ef508e281f1ce075bc6db72 77029ab56e83ea2115dd53ff87483682abe5d7ca 12
Node Delta Base Delta SHA1 Delta Length Node Delta Base Delta SHA1 Delta Length
1bb2e6237e035c8f8ef508e281f1ce075bc6db72 0000000000000000000000000000000000000000 7ca8c71a64f7b56380e77573da2f7a5fdd2ecdb5 8 1bb2e6237e035c8f8ef508e281f1ce075bc6db72 0000000000000000000000000000000000000000 7ca8c71a64f7b56380e77573da2f7a5fdd2ecdb5 8
$ hg debughistorypack $TESTTMP/hgcache/master/packs/*.histidx $ hg debughistorypack $TESTTMP/hgcache/master/packs/*.histidx
$ cp -R $TESTTMP/backuppacks $TESTTMP/hgcache/master/packs $ cp -R $TESTTMP/backuppacks $TESTTMP/hgcache/master/packs
# Test repacking datapack without history # Test repacking datapack without history
$ rm -rf $CACHEDIR/master/packs/*hist* $ rm -rf $CACHEDIR/master/packs/*hist*
$ hg repack $ hg repack
$ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack $ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack
x x
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
1bb2e6237e03 000000000000 8 1bb2e6237e03 000000000000 8 8
aee31534993a 000000000000 4 aee31534993a 000000000000 4 4
d4a3ed9310e5 000000000000 6 d4a3ed9310e5 000000000000 6 6
y y
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
577959738234 000000000000 70 577959738234 000000000000 70 8
$ hg cat -r ".^" x $ hg cat -r ".^" x
x x
x x
x x
x x
Incremental repack Incremental repack

tests/test-remotefilelog-repack.t

$TESTTMP/hgcache/master/packs/8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack $TESTTMP/hgcache/master/packs/8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack
$TESTTMP/hgcache/repos $TESTTMP/hgcache/repos
# Test debug commands # Test debug commands
$ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack $ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack
x x
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
1bb2e6237e03 000000000000 8 1bb2e6237e03 000000000000 8 8
d4a3ed9310e5 1bb2e6237e03 12 d4a3ed9310e5 1bb2e6237e03 12 6
aee31534993a d4a3ed9310e5 12 aee31534993a d4a3ed9310e5 12 4
$ hg debugdatapack --long $TESTTMP/hgcache/master/packs/*.datapack $ hg debugdatapack --long $TESTTMP/hgcache/master/packs/*.datapack
x x
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
1bb2e6237e035c8f8ef508e281f1ce075bc6db72 0000000000000000000000000000000000000000 8 1bb2e6237e035c8f8ef508e281f1ce075bc6db72 0000000000000000000000000000000000000000 8 8
d4a3ed9310e5bd9887e3bf779da5077efab28216 1bb2e6237e035c8f8ef508e281f1ce075bc6db72 12 d4a3ed9310e5bd9887e3bf779da5077efab28216 1bb2e6237e035c8f8ef508e281f1ce075bc6db72 12 6
aee31534993a501858fb6dd96a065671922e7d51 d4a3ed9310e5bd9887e3bf779da5077efab28216 12 aee31534993a501858fb6dd96a065671922e7d51 d4a3ed9310e5bd9887e3bf779da5077efab28216 12 4
$ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack --node d4a3ed9310e5bd9887e3bf779da5077efab28216 $ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack --node d4a3ed9310e5bd9887e3bf779da5077efab28216
x x
Node Delta Base Delta SHA1 Delta Length Node Delta Base Delta SHA1 Delta Length
d4a3ed9310e5bd9887e3bf779da5077efab28216 1bb2e6237e035c8f8ef508e281f1ce075bc6db72 77029ab56e83ea2115dd53ff87483682abe5d7ca 12 d4a3ed9310e5bd9887e3bf779da5077efab28216 1bb2e6237e035c8f8ef508e281f1ce075bc6db72 77029ab56e83ea2115dd53ff87483682abe5d7ca 12
Node Delta Base Delta SHA1 Delta Length Node Delta Base Delta SHA1 Delta Length
1bb2e6237e035c8f8ef508e281f1ce075bc6db72 0000000000000000000000000000000000000000 7ca8c71a64f7b56380e77573da2f7a5fdd2ecdb5 8 1bb2e6237e035c8f8ef508e281f1ce075bc6db72 0000000000000000000000000000000000000000 7ca8c71a64f7b56380e77573da2f7a5fdd2ecdb5 8
$ hg debughistorypack $TESTTMP/hgcache/master/packs/*.histidx $ hg debughistorypack $TESTTMP/hgcache/master/packs/*.histidx
$ cp -R $TESTTMP/backuppacks $TESTTMP/hgcache/master/packs $ cp -R $TESTTMP/backuppacks $TESTTMP/hgcache/master/packs
# Test repacking datapack without history # Test repacking datapack without history
$ rm -rf $CACHEDIR/master/packs/*hist* $ rm -rf $CACHEDIR/master/packs/*hist*
$ hg repack $ hg repack
$ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack $ hg debugdatapack $TESTTMP/hgcache/master/packs/*.datapack
x x
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
1bb2e6237e03 000000000000 8 1bb2e6237e03 000000000000 8 8
aee31534993a 000000000000 4 aee31534993a 000000000000 4 4
d4a3ed9310e5 000000000000 6 d4a3ed9310e5 000000000000 6 6
y y
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
577959738234 000000000000 70 577959738234 000000000000 70 8
$ hg cat -r ".^" x $ hg cat -r ".^" x
x x
x x
x x
x x
Incremental repack Incremental repack
aee31534993a => 1406e7411862 000000000000 a89d614e2364 aee31534993a => 1406e7411862 000000000000 a89d614e2364
1406e7411862 => 000000000000 000000000000 b292c1e3311f 1406e7411862 => 000000000000 000000000000 b292c1e3311f
Test limiting the max delta chain length Test limiting the max delta chain length
$ hg repack --config packs.maxchainlen=1 $ hg repack --config packs.maxchainlen=1
$ hg debugdatapack $TESTTMP/hgcache/master/packs/*.dataidx $ hg debugdatapack $TESTTMP/hgcache/master/packs/*.dataidx
x x
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
1bb2e6237e03 000000000000 8 1bb2e6237e03 000000000000 8 (missing)
d4a3ed9310e5 1bb2e6237e03 12 d4a3ed9310e5 1bb2e6237e03 12 (missing)
aee31534993a 000000000000 4 aee31534993a 000000000000 4 (missing)
1406e7411862 aee31534993a 12 1406e7411862 aee31534993a 12 (missing)
y y
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
577959738234 000000000000 8 577959738234 000000000000 8 (missing)

tests/test-treemanifest-repack.t

- Verify datapack contents - Verify datapack contents
$ for i in $CACHEDIR/master/packs/manifests/*.datapack; do $ for i in $CACHEDIR/master/packs/manifests/*.datapack; do
> echo $i > echo $i
> hg debugdatapack "$i" > hg debugdatapack "$i"
> done > done
$TESTTMP/hgcache/master/packs/manifests/65df85879cdd898607ee3f323a0b61edc7de25b8.datapack $TESTTMP/hgcache/master/packs/manifests/65df85879cdd898607ee3f323a0b61edc7de25b8.datapack
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
a0c8bcbbb45c 000000000000 43 a0c8bcbbb45c 000000000000 43 (missing)
$TESTTMP/hgcache/master/packs/manifests/bb55d9105672c45d4f82df15bd091a555ef02c79.datapack $TESTTMP/hgcache/master/packs/manifests/bb55d9105672c45d4f82df15bd091a555ef02c79.datapack
dir dir
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
23226e7a252c 000000000000 43 23226e7a252c 000000000000 43 (missing)
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
1832e0765de9 000000000000 89 1832e0765de9 000000000000 89 (missing)
- Verify histpack contents - Verify histpack contents
$ for i in $CACHEDIR/master/packs/manifests/*.histpack; do $ for i in $CACHEDIR/master/packs/manifests/*.histpack; do
> echo $i > echo $i
> hg debughistorypack "$i" > hg debughistorypack "$i"
> done > done
$TESTTMP/hgcache/master/packs/manifests/4301ce26f4c07686220c7f57d80b466cfba9899e.histpack $TESTTMP/hgcache/master/packs/manifests/4301ce26f4c07686220c7f57d80b466cfba9899e.histpack
$ ls_l $CACHEDIR/master/packs/manifests | grep pack $ ls_l $CACHEDIR/master/packs/manifests | grep pack
-r--r--r-- 262 7535b6084226436bbdff33043969e7fa963e8428.histpack -r--r--r-- 262 7535b6084226436bbdff33043969e7fa963e8428.histpack
-r--r--r-- 327 be1a1ae9828d9bd09ab8ba765db358f99fd09ea7.datapack -r--r--r-- 327 be1a1ae9828d9bd09ab8ba765db358f99fd09ea7.datapack
$ hg debugdatapack $CACHEDIR/master/packs/manifests/*.datapack $ hg debugdatapack $CACHEDIR/master/packs/manifests/*.datapack
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
1832e0765de9 000000000000 89 1832e0765de9 000000000000 89 (missing)
a0c8bcbbb45c 1832e0765de9 12 a0c8bcbbb45c 1832e0765de9 12 (missing)
dir dir
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
23226e7a252c 000000000000 43 23226e7a252c 000000000000 43 (missing)
$ hg debughistorypack $CACHEDIR/master/packs/manifests/*.histpack $ hg debughistorypack $CACHEDIR/master/packs/manifests/*.histpack
Node P1 Node P2 Node Link Node Copy From Node P1 Node P2 Node Link Node Copy From
1832e0765de9 a0c8bcbbb45c 000000000000 8e83608cbe60 1832e0765de9 a0c8bcbbb45c 000000000000 8e83608cbe60
a0c8bcbbb45c 000000000000 000000000000 1f0dee641bb7 a0c8bcbbb45c 000000000000 000000000000 1f0dee641bb7
$ ls .hg/cache/packs/manifests $ ls .hg/cache/packs/manifests
56e8c6f0ca2a324b8b5ca1a2730323a1b4d0793a.dataidx 56e8c6f0ca2a324b8b5ca1a2730323a1b4d0793a.dataidx
56e8c6f0ca2a324b8b5ca1a2730323a1b4d0793a.datapack 56e8c6f0ca2a324b8b5ca1a2730323a1b4d0793a.datapack
7535b6084226436bbdff33043969e7fa963e8428.histidx 7535b6084226436bbdff33043969e7fa963e8428.histidx
7535b6084226436bbdff33043969e7fa963e8428.histpack 7535b6084226436bbdff33043969e7fa963e8428.histpack
$ hg debugdatapack .hg/cache/packs/manifests/*.datapack $ hg debugdatapack .hg/cache/packs/manifests/*.datapack
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
1832e0765de9 000000000000 89 1832e0765de9 000000000000 89 89
a0c8bcbbb45c 1832e0765de9 12 a0c8bcbbb45c 1832e0765de9 12 43
dir dir
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
23226e7a252c 000000000000 43 23226e7a252c 000000000000 43 43
Test incremental revlog repacking Test incremental revlog repacking
# 1. Make commit that we'll need to repack # 1. Make commit that we'll need to repack
$ echo >> a $ echo >> a
$ hg commit -Aqm 'modify a' $ hg commit -Aqm 'modify a'
$ hg debugindex .hg/store/00manifesttree.i $ hg debugindex .hg/store/00manifesttree.i
rev offset length delta linkrev nodeid p1 p2 rev offset length delta linkrev nodeid p1 p2
0 0 44 -1 0 a0c8bcbbb45c 000000000000 000000000000 0 0 44 -1 0 a0c8bcbbb45c 000000000000 000000000000
$ mv .hg/store/00manifesttree.i.bak .hg/store/00manifesttree.i $ mv .hg/store/00manifesttree.i.bak .hg/store/00manifesttree.i
Test incremental repack with limited revs only repacks those revs Test incremental repack with limited revs only repacks those revs
$ rm -rf .hg/cache/packs/manifests $ rm -rf .hg/cache/packs/manifests
$ hg repack --incremental --config treemanifest.repackstartrev=1 --config treemanifest.repackendrev=1 $ hg repack --incremental --config treemanifest.repackstartrev=1 --config treemanifest.repackendrev=1
$ hg debugdatapack .hg/cache/packs/manifests/*.datapack $ hg debugdatapack .hg/cache/packs/manifests/*.datapack
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
1832e0765de9 000000000000 89 1832e0765de9 000000000000 89 89
dir dir
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
23226e7a252c 000000000000 43 23226e7a252c 000000000000 43 43

tests/test-treemanifest-server.t

$ mkdir subdir2 $ mkdir subdir2
$ echo z >> subdir2/z $ echo z >> subdir2/z
$ hg commit -qAm "add subdir2/z" $ hg commit -qAm "add subdir2/z"
2 trees fetched over * (glob) 2 trees fetched over * (glob)
$ hg debugdatapack $CACHEDIR/master/packs/manifests/*.dataidx $ hg debugdatapack $CACHEDIR/master/packs/manifests/*.dataidx
subdir subdir
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
bc0c2c938b92 000000000000 43 bc0c2c938b92 000000000000 43 (missing)
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
85b359fdb09e 000000000000 49 85b359fdb09e 000000000000 49 (missing)
$ hg debugdatapack .hg/store/packs/manifests/*.dataidx $ hg debugdatapack .hg/store/packs/manifests/*.dataidx
subdir2 subdir2
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
ddb35f099a64 000000000000 43 ddb35f099a64 000000000000 43 (missing)
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
54cbf534b62b 000000000000 99 54cbf534b62b 000000000000 99 (missing)
Test pushing without pushrebase fails Test pushing without pushrebase fails
$ hg push $ hg push
pushing to ssh://user@dummy/master pushing to ssh://user@dummy/master
searching for changes searching for changes
remote: adding changesets remote: adding changesets
remote: adding manifests remote: adding manifests
remote: transaction abort! remote: transaction abort!

tests/test-treemanifest.t

-r--r--r-- 1146 678f597a73b2b96f2e120c84ef8a84069a250266.dataidx -r--r--r-- 1146 678f597a73b2b96f2e120c84ef8a84069a250266.dataidx
-r--r--r-- 315 678f597a73b2b96f2e120c84ef8a84069a250266.datapack -r--r--r-- 315 678f597a73b2b96f2e120c84ef8a84069a250266.datapack
-r--r--r-- 1224 ed1a27864c5d25f144a51961ad6e79088f2a7571.histidx -r--r--r-- 1224 ed1a27864c5d25f144a51961ad6e79088f2a7571.histidx
-r--r--r-- 265 ed1a27864c5d25f144a51961ad6e79088f2a7571.histpack -r--r--r-- 265 ed1a27864c5d25f144a51961ad6e79088f2a7571.histpack
$ hg debugdatapack $CACHEDIR/master/packs/manifests/678f597a73b2b96f2e120c84ef8a84069a250266 $ hg debugdatapack $CACHEDIR/master/packs/manifests/678f597a73b2b96f2e120c84ef8a84069a250266
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
bc0c2c938b92 000000000000 43 bc0c2c938b92 000000000000 43 (missing)
subdir subdir
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
ddb35f099a64 000000000000 43 ddb35f099a64 000000000000 43 (missing)
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
70f2c6726cec bc0c2c938b92 61 70f2c6726cec bc0c2c938b92 61 (missing)
Test that commit creates local trees Test that commit creates local trees
$ hg up -q tip $ hg up -q tip
1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ echo z >> subdir/z $ echo z >> subdir/z
$ hg commit -qAm 'modify subdir/z' $ hg commit -qAm 'modify subdir/z'
$ ls_l .hg/store/packs/manifests $ ls_l .hg/store/packs/manifests
-r--r--r-- 1196 57710544ca24ac4f36682ec279959879c92a3275.histidx -r--r--r-- 1196 57710544ca24ac4f36682ec279959879c92a3275.histidx
-r--r--r-- 183 57710544ca24ac4f36682ec279959879c92a3275.histpack -r--r--r-- 183 57710544ca24ac4f36682ec279959879c92a3275.histpack
-r--r--r-- 1106 a7f7e084adff88a01cf76909345be1e56ee704a9.dataidx -r--r--r-- 1106 a7f7e084adff88a01cf76909345be1e56ee704a9.dataidx
-r--r--r-- 254 a7f7e084adff88a01cf76909345be1e56ee704a9.datapack -r--r--r-- 254 a7f7e084adff88a01cf76909345be1e56ee704a9.datapack
$ hg debugdatapack .hg/store/packs/manifests/a7f7e084adff88a01cf76909345be1e56ee704a9 $ hg debugdatapack .hg/store/packs/manifests/a7f7e084adff88a01cf76909345be1e56ee704a9
subdir subdir
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
ac728a786423 000000000000 43 ac728a786423 000000000000 43 (missing)
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
7a911436836f 000000000000 92 7a911436836f 000000000000 92 (missing)
Test that manifest matchers work Test that manifest matchers work
$ hg status --rev 1 --rev 2 -I subdir/a $ hg status --rev 1 --rev 2 -I subdir/a
$ hg status --rev 1 --rev 2 -I subdir/z $ hg status --rev 1 --rev 2 -I subdir/z
M subdir/z M subdir/z
Test config validation Test config validation
$ hg log -r . --config extensions.fastmanifest=! $ hg log -r . --config extensions.fastmanifest=!
rebasing 4:f096b21e165f "modify subdir/y again" (tip) rebasing 4:f096b21e165f "modify subdir/y again" (tip)
saved backup bundle to $TESTTMP/client/.hg/strip-backup/6a2476258ba5-a90056a1-rebase.hg (glob) saved backup bundle to $TESTTMP/client/.hg/strip-backup/6a2476258ba5-a90056a1-rebase.hg (glob)
$ hg log -r '.^::.' -T '{manifest}\n' $ hg log -r '.^::.' -T '{manifest}\n'
3:0d05c20bb7eb 3:0d05c20bb7eb
4:8289b85c6a30 4:8289b85c6a30
$ hg debugdatapack .hg/store/packs/manifests/5a5fb42e99986c90ac86b57d184561c44238b7b7.datapack $ hg debugdatapack .hg/store/packs/manifests/5a5fb42e99986c90ac86b57d184561c44238b7b7.datapack
subdir subdir
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
a4e2f032ee0f 000000000000 43 a4e2f032ee0f 000000000000 43 (missing)
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
0d05c20bb7eb 000000000000 92 0d05c20bb7eb 000000000000 92 (missing)
subdir subdir
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
ad0a48a2ec1e 000000000000 43 ad0a48a2ec1e 000000000000 43 (missing)
Node Delta Base Delta Length Node Delta Base Delta Length Blob Size
8289b85c6a30 000000000000 92 8289b85c6a30 000000000000 92 (missing)
Test treemanifest with sparse enabled Test treemanifest with sparse enabled
$ cat >> .hg/hgrc <<EOF $ cat >> .hg/hgrc <<EOF
> [extensions] > [extensions]
> sparse = $TESTDIR/../hgext3rd/fbsparse.py > sparse = $TESTDIR/../hgext3rd/fbsparse.py
> reset = $TESTDIR/../hgext3rd/reset.py > reset = $TESTDIR/../hgext3rd/reset.py
> EOF > EOF
$ hg sparse -I subdir $ hg sparse -I subdir