This is an archive of the discontinued Mercurial Phabricator instance.

Differential D54

datapack: allow 'long' for metadata types
ClosedPublic

Authored by durham on Jul 11 2017, 7:20 PM.
Tags
None
Subscribers
None

Details

Reviewers
quark
phillco
Group Reviewers
Restricted Project
Commits
rFBHGXb176b79b1c1d: datapack: allow 'long' for metadata types
Summary

Previously the code required that sizes be of type int. Since python plays loose
with integer types, we also need to support long.

Test Plan

The existing test-remotefilelog-repack-fast.t test was completely
broken. It only enabled fast datapacks for the server repo, not the clients.
Enabling it for the clients as well catches this issue and verifies the fix.

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

durham created this revision.Jul 11 2017, 7:20 PM
quark accepted this revision.Jul 11 2017, 7:28 PM

Thanks for fixing this!

remotefilelog/shallowutil.py
159

Ah, TIL isinstance accepts a tuple.

tests/test-remotefilelog-repack-fast.t
6–10

This could be probably unified with the non-fast version by using #testcases

This revision is now accepted and ready to land.Jul 11 2017, 7:28 PM
phillco accepted this revision.Jul 11 2017, 7:30 PM
Closed by commit rFBHGXb176b79b1c1d: datapack: allow 'long' for metadata types (authored by durham). · Explain WhyJul 11 2017, 11:29 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 82

remotefilelog/shallowutil.py

metabuf += k metabuf += k
metabuf += struct.pack('!H', len(v)) metabuf += struct.pack('!H', len(v))
metabuf += v metabuf += v
# len(metabuf) is guaranteed representable in 4 bytes, because there are # len(metabuf) is guaranteed representable in 4 bytes, because there are
# only 256 keys, and for each value, len(value) <= 0xfffe. # only 256 keys, and for each value, len(value) <= 0xfffe.
return metabuf return metabuf
_metaitemtypes = { _metaitemtypes = {
constants.METAKEYFLAG: int, constants.METAKEYFLAG: (int, long),
constants.METAKEYSIZE: int, constants.METAKEYSIZE: (int, long),
} }
def buildpackmeta(metadict): def buildpackmeta(metadict):
"""like _buildpackmeta, but typechecks metadict and normalize it. """like _buildpackmeta, but typechecks metadict and normalize it.
This means, METAKEYSIZE and METAKEYSIZE should have integers as values, This means, METAKEYSIZE and METAKEYSIZE should have integers as values,
and METAKEYFLAG will be dropped if its value is 0. and METAKEYFLAG will be dropped if its value is 0.
""" """
newmeta = {} newmeta = {}
for k, v in (metadict or {}).iteritems(): for k, v in (metadict or {}).iteritems():
expectedtype = _metaitemtypes.get(k, bytes) expectedtype = _metaitemtypes.get(k, (bytes,))
quarkUnsubmitted
Not Done

Ah, TIL isinstance accepts a tuple.

quark: Ah, TIL `isinstance` accepts a tuple.
if not isinstance(v, expectedtype): if not isinstance(v, expectedtype):
raise error.ProgrammingError('packmeta: wrong type of key %s' % k) raise error.ProgrammingError('packmeta: wrong type of key %s' % k)
# normalize int to binary buffer # normalize int to binary buffer
if expectedtype is int: if int in expectedtype:
# optimization: remove flag if it's 0 to save space # optimization: remove flag if it's 0 to save space
if k == constants.METAKEYFLAG and v == 0: if k == constants.METAKEYFLAG and v == 0:
continue continue
v = int2bin(v) v = int2bin(v)
newmeta[k] = v newmeta[k] = v
return _buildpackmeta(newmeta) return _buildpackmeta(newmeta)
def parsepackmeta(metabuf): def parsepackmeta(metabuf):
"""like _parsepackmeta, but convert fields to desired types automatically. """like _parsepackmeta, but convert fields to desired types automatically.
This means, METAKEYFLAG and METAKEYSIZE fields will be converted to This means, METAKEYFLAG and METAKEYSIZE fields will be converted to
integers. integers.
""" """
metadict = _parsepackmeta(metabuf) metadict = _parsepackmeta(metabuf)
for k, v in metadict.iteritems(): for k, v in metadict.iteritems():
if k in _metaitemtypes and _metaitemtypes[k] is int: if k in _metaitemtypes and int in _metaitemtypes[k]:
metadict[k] = bin2int(v) metadict[k] = bin2int(v)
return metadict return metadict
def int2bin(n): def int2bin(n):
"""convert a non-negative integer to raw binary buffer""" """convert a non-negative integer to raw binary buffer"""
buf = bytearray() buf = bytearray()
while n > 0: while n > 0:
buf.insert(0, n & 0xff) buf.insert(0, n & 0xff)

tests/library.sh

${PYTHON:-python} -c 'import lz4' || exit 80 ${PYTHON:-python} -c 'import lz4' || exit 80
CACHEDIR=$PWD/hgcache CACHEDIR=$PWD/hgcache
cat >> $HGRCPATH <<EOF cat >> $HGRCPATH <<EOF
[remotefilelog] [remotefilelog]
cachepath=$CACHEDIR cachepath=$CACHEDIR
debug=True debug=True
historypackv1=True historypackv1=True
datapackversion=1
[extensions] [extensions]
remotefilelog= remotefilelog=
rebase= rebase=
mq= mq=
[ui] [ui]
ssh=python "$TESTDIR/dummyssh" ssh=python "$TESTDIR/dummyssh"
[server] [server]
preferuncompressed=True preferuncompressed=True

tests/test-remotefilelog-repack-fast.t

$ PYTHONPATH=$TESTDIR/..:$PYTHONPATH $ PYTHONPATH=$TESTDIR/..:$PYTHONPATH
$ export PYTHONPATH $ export PYTHONPATH
$ . "$TESTDIR/library.sh" $ . "$TESTDIR/library.sh"
$ cat >> $HGRCPATH <<EOF
> [remotefilelog]
> fastdatapack=True
> EOF
quarkUnsubmitted
Not Done

This could be probably unified with the non-fast version by using #testcases

quark: This could be probably unified with the non-fast version by using `#testcases`
$ hginit master $ hginit master
$ cd master $ cd master
$ cat >> .hg/hgrc <<EOF $ cat >> .hg/hgrc <<EOF
> [remotefilelog] > [remotefilelog]
> server=True > server=True
> serverexpiration=-1 > serverexpiration=-1
> fastdatapack=True
> EOF > EOF
$ echo x > x $ echo x > x
$ hg commit -qAm x $ hg commit -qAm x
$ echo x >> x $ echo x >> x
$ hg commit -qAm x2 $ hg commit -qAm x2
$ cd .. $ cd ..
$ hgcloneshallow ssh://user@dummy/master shallow -q $ hgcloneshallow ssh://user@dummy/master shallow -q

tests/test-treemanifest-repack.t

Node P1 Node P2 Node Link Node Copy From Node P1 Node P2 Node Link Node Copy From
23226e7a252c 000000000000 000000000000 8e83608cbe60 23226e7a252c 000000000000 000000000000 8e83608cbe60
- Repack and reverify - Repack and reverify
$ hg repack $ hg repack
$ ls_l $CACHEDIR/master/packs/manifests | grep pack $ ls_l $CACHEDIR/master/packs/manifests | grep pack
-r--r--r-- 262 7535b6084226436bbdff33043969e7fa963e8428.histpack -r--r--r-- 262 7535b6084226436bbdff33043969e7fa963e8428.histpack
-r--r--r-- 315 d24c358c968883e3b6c4bd6a85845dfb28fd8de6.datapack -r--r--r-- 327 be1a1ae9828d9bd09ab8ba765db358f99fd09ea7.datapack
$ hg debugdatapack $CACHEDIR/master/packs/manifests/*.datapack $ hg debugdatapack $CACHEDIR/master/packs/manifests/*.datapack
Node Delta Base Delta Length Node Delta Base Delta Length
1832e0765de9 000000000000 89 1832e0765de9 000000000000 89
a0c8bcbbb45c 1832e0765de9 12 a0c8bcbbb45c 1832e0765de9 12
# Test repacking local manifest packs # Test repacking local manifest packs
$ hg up -q 1 $ hg up -q 1
$ echo a >> a && hg commit -Aqm 'modify a' $ echo a >> a && hg commit -Aqm 'modify a'
$ echo b >> dir/b && hg commit -Aqm 'modify dir/b' $ echo b >> dir/b && hg commit -Aqm 'modify dir/b'
$ ls_l .hg/store/packs/manifests | grep datapack $ ls_l .hg/store/packs/manifests | grep datapack
-r--r--r-- 248 5d1716bbef6e7200192de6509055d1ee31a4172c.datapack -r--r--r-- 248 5d1716bbef6e7200192de6509055d1ee31a4172c.datapack
-r--r--r-- 146 cffef142da32f3e52c1779490e5d0ddac5f9b82b.datapack -r--r--r-- 146 cffef142da32f3e52c1779490e5d0ddac5f9b82b.datapack
$ hg repack $ hg repack
$ ls -l .hg/store/packs/manifests | grep datapack $ ls_l .hg/store/packs/manifests | grep datapack
* 362 * 9e5a04938d53f5b418e4f1b6a413d250f231f60a.datapack (glob) -r--r--r-- 374 201094db51b761cd78352c055b3135178aadfec5.datapack
# Test incremental repacking of trees # Test incremental repacking of trees
$ echo b >> dir/b && hg commit -Aqm 'modify dir/b' $ echo b >> dir/b && hg commit -Aqm 'modify dir/b'
$ echo b >> dir/b && hg commit -Aqm 'modify dir/b' $ echo b >> dir/b && hg commit -Aqm 'modify dir/b'
$ ls -l .hg/store/packs/manifests | grep datapack $ ls_l .hg/store/packs/manifests | grep datapack
* 248 * 21501384df03b8489b366c5218be639fa08830e4.datapack (glob) -r--r--r-- 374 201094db51b761cd78352c055b3135178aadfec5.datapack
* 362 * 9e5a04938d53f5b418e4f1b6a413d250f231f60a.datapack (glob) -r--r--r-- 248 21501384df03b8489b366c5218be639fa08830e4.datapack
* 248 * d7e689a91ac63385be120a118af9ce8663748f28.datapack (glob) -r--r--r-- 248 d7e689a91ac63385be120a118af9ce8663748f28.datapack
- repack incremental does nothing here because there are so few packs - repack incremental does nothing here because there are so few packs
$ hg repack --incremental --config remotefilelog.data.generations=300,200 --config remotefilelog.data.repacksizelimit=300 $ hg repack --incremental --config remotefilelog.data.generations=300,200 --config remotefilelog.data.repacksizelimit=300
$ ls -l .hg/store/packs/manifests | grep datapack $ ls_l .hg/store/packs/manifests | grep datapack
* 248 * 21501384df03b8489b366c5218be639fa08830e4.datapack (glob) -r--r--r-- 374 201094db51b761cd78352c055b3135178aadfec5.datapack
* 362 * 9e5a04938d53f5b418e4f1b6a413d250f231f60a.datapack (glob) -r--r--r-- 248 21501384df03b8489b366c5218be639fa08830e4.datapack
* 248 * d7e689a91ac63385be120a118af9ce8663748f28.datapack (glob) -r--r--r-- 248 d7e689a91ac63385be120a118af9ce8663748f28.datapack
$ echo b >> dir/b && hg commit -Aqm 'modify dir/b' $ echo b >> dir/b && hg commit -Aqm 'modify dir/b'
$ echo b >> dir/b && hg commit -Aqm 'modify dir/b' $ echo b >> dir/b && hg commit -Aqm 'modify dir/b'
$ echo b >> dir/b && hg commit -Aqm 'modify dir/b' $ echo b >> dir/b && hg commit -Aqm 'modify dir/b'
$ ls -l .hg/store/packs/manifests | grep datapack $ ls_l .hg/store/packs/manifests | grep datapack
* 248 * 21501384df03b8489b366c5218be639fa08830e4.datapack (glob) -r--r--r-- 374 201094db51b761cd78352c055b3135178aadfec5.datapack
* 248 * 347263bf1efbdb5bf7e1d1565b6b504073fb9093.datapack (glob) -r--r--r-- 248 21501384df03b8489b366c5218be639fa08830e4.datapack
* 248 * 544a3b46a61732209116ae50847ec333b75e3765.datapack (glob) -r--r--r-- 248 347263bf1efbdb5bf7e1d1565b6b504073fb9093.datapack
* 248 * 863908ef8149261ab0d891c2344d8e8766c39441.datapack (glob) -r--r--r-- 248 544a3b46a61732209116ae50847ec333b75e3765.datapack
* 362 * 9e5a04938d53f5b418e4f1b6a413d250f231f60a.datapack (glob) -r--r--r-- 248 863908ef8149261ab0d891c2344d8e8766c39441.datapack
* 248 * d7e689a91ac63385be120a118af9ce8663748f28.datapack (glob) -r--r--r-- 248 d7e689a91ac63385be120a118af9ce8663748f28.datapack
- repack incremental kicks in once there are a number of packs - repack incremental kicks in once there are a number of packs
$ hg repack --incremental --config remotefilelog.data.generations=300,200 $ hg repack --incremental --config remotefilelog.data.generations=300,200
$ ls -l .hg/store/packs/manifests | grep datapack $ ls_l .hg/store/packs/manifests | grep datapack
* 1148 * 57360ff79b595e6474abacd912600d61b5e5c840.datapack (glob) -r--r--r-- 374 201094db51b761cd78352c055b3135178aadfec5.datapack
* 362 * 9e5a04938d53f5b418e4f1b6a413d250f231f60a.datapack (glob) -r--r--r-- 1188 785af77b59c45dd43e3e0e63929d77665c505387.datapack
Test repacking from revlogs to pack files on the server Test repacking from revlogs to pack files on the server
$ cd ../master $ cd ../master
$ cat >> .hg/hgrc <<EOF $ cat >> .hg/hgrc <<EOF
> [extensions] > [extensions]
> treemanifest=$TESTDIR/../treemanifest > treemanifest=$TESTDIR/../treemanifest
> remotefilelog=$TESTDIR/../remotefilelog > remotefilelog=$TESTDIR/../remotefilelog
> [remotefilelog] > [remotefilelog]
> server=True > server=True
> [treemanifest] > [treemanifest]
> server=True > server=True
> EOF > EOF
$ hg backfilltree $ hg backfilltree
$ cat .hg/store/fncache | sort $ cat .hg/store/fncache | sort
data/a.i data/a.i
data/dir/b.i data/dir/b.i
meta/dir/00manifest.i meta/dir/00manifest.i
$ hg repack $ hg repack
$ ls .hg/cache/packs/manifests $ ls .hg/cache/packs/manifests
56e8c6f0ca2a324b8b5ca1a2730323a1b4d0793a.dataidx
56e8c6f0ca2a324b8b5ca1a2730323a1b4d0793a.datapack
7535b6084226436bbdff33043969e7fa963e8428.histidx 7535b6084226436bbdff33043969e7fa963e8428.histidx
7535b6084226436bbdff33043969e7fa963e8428.histpack 7535b6084226436bbdff33043969e7fa963e8428.histpack
d24c358c968883e3b6c4bd6a85845dfb28fd8de6.dataidx
d24c358c968883e3b6c4bd6a85845dfb28fd8de6.datapack
$ hg debugdatapack .hg/cache/packs/manifests/*.datapack $ hg debugdatapack .hg/cache/packs/manifests/*.datapack
Node Delta Base Delta Length Node Delta Base Delta Length
1832e0765de9 000000000000 89 1832e0765de9 000000000000 89
a0c8bcbbb45c 1832e0765de9 12 a0c8bcbbb45c 1832e0765de9 12
dir dir