( )⚙ D7068 sidedatacopies: only store an entry if it has values

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D7068

sidedatacopies: only store an entry if it has values
ClosedPublic

Authored by marmoute on Oct 12 2019, 12:45 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG30570a056fa8: sidedatacopies: only store an entry if it has values
Summary

This will make for a shorter storage and help use to write faster code in simple
case. This change already provided a speed boost for copy tracing.

For example here is combined time of running copies tracing on a 6989 pairs of
revision on the pypy repos:

before: 771s
after: 631s - 18%

This also has a very positive impact on changelog size. For example here are the
number for the 00changelog.d file of pypy.

before: 30449712 Bytes
after: 24973718 Bytes - 18%

This give an overall quite acceptable overhead for storing copies into the
changelog:

filelog-only: 23370586
sidedata: 24973718 Bytes + 7%

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

marmoute created this revision.Oct 12 2019, 12:45 PM
marmoute added a commit: rHG30570a056fa8: sidedatacopies: only store an entry if it has values.Oct 15 2019, 6:11 PM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG30570a056fa8: sidedatacopies: only store an entry if it has values (authored by marmoute). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 17187

mercurial/changelog.py

if p2copies is not None: if p2copies is not None:
extra[b'p2copies'] = p2copies extra[b'p2copies'] = p2copies
if filesadded is not None: if filesadded is not None:
extra[b'filesadded'] = filesadded extra[b'filesadded'] = filesadded
if filesremoved is not None: if filesremoved is not None:
extra[b'filesremoved'] = filesremoved extra[b'filesremoved'] = filesremoved
elif self._copiesstorage == b'changeset-sidedata': elif self._copiesstorage == b'changeset-sidedata':
sidedata = {} sidedata = {}
if p1copies is not None: if p1copies:
sidedata[sidedatamod.SD_P1COPIES] = p1copies sidedata[sidedatamod.SD_P1COPIES] = p1copies
if p2copies is not None: if p2copies:
sidedata[sidedatamod.SD_P2COPIES] = p2copies sidedata[sidedatamod.SD_P2COPIES] = p2copies
if filesadded is not None: if filesadded:
sidedata[sidedatamod.SD_FILESADDED] = filesadded sidedata[sidedatamod.SD_FILESADDED] = filesadded
if filesremoved is not None: if filesremoved:
sidedata[sidedatamod.SD_FILESREMOVED] = filesremoved sidedata[sidedatamod.SD_FILESREMOVED] = filesremoved
if not sidedata:
sidedata = None
if extra: if extra:
extra = encodeextra(extra) extra = encodeextra(extra)
parseddate = b"%s %s" % (parseddate, extra) parseddate = b"%s %s" % (parseddate, extra)
l = [hex(manifest), user, parseddate] + sortedfiles + [b"", desc] l = [hex(manifest), user, parseddate] + sortedfiles + [b"", desc]
text = b"\n".join(l) text = b"\n".join(l)
return self.addrevision( return self.addrevision(
text, transaction, len(self), p1, p2, sidedata=sidedata text, transaction, len(self), p1, p2, sidedata=sidedata

mercurial/copies.py

sidedata = {} sidedata = {}
if any([filescopies, filesadded, filesremoved]): if any([filescopies, filesadded, filesremoved]):
sortedfiles = sorted(ctx.files()) sortedfiles = sorted(ctx.files())
p1copies, p2copies = filescopies p1copies, p2copies = filescopies
p1copies = encodecopies(sortedfiles, p1copies) p1copies = encodecopies(sortedfiles, p1copies)
p2copies = encodecopies(sortedfiles, p2copies) p2copies = encodecopies(sortedfiles, p2copies)
filesadded = encodefileindices(sortedfiles, filesadded) filesadded = encodefileindices(sortedfiles, filesadded)
filesremoved = encodefileindices(sortedfiles, filesremoved) filesremoved = encodefileindices(sortedfiles, filesremoved)
if p1copies:
sidedata[sidedatamod.SD_P1COPIES] = p1copies sidedata[sidedatamod.SD_P1COPIES] = p1copies
if p2copies:
sidedata[sidedatamod.SD_P2COPIES] = p2copies sidedata[sidedatamod.SD_P2COPIES] = p2copies
if filesadded:
sidedata[sidedatamod.SD_FILESADDED] = filesadded sidedata[sidedatamod.SD_FILESADDED] = filesadded
if filesremoved:
sidedata[sidedatamod.SD_FILESREMOVED] = filesremoved sidedata[sidedatamod.SD_FILESREMOVED] = filesremoved
return sidedata return sidedata
def getsidedataadder(srcrepo, destrepo): def getsidedataadder(srcrepo, destrepo):
def sidedatacompanion(revlog, rev): def sidedatacompanion(revlog, rev):
sidedata = {} sidedata = {}
if util.safehasattr(revlog, 'filteredrevs'): # this is a changelog if util.safehasattr(revlog, 'filteredrevs'): # this is a changelog
sidedata = _getsidedata(srcrepo, rev) sidedata = _getsidedata(srcrepo, rev)

tests/test-copies-in-changeset.t

1 1
2 2
p1copies: 0\x00a (esc) p1copies: 0\x00a (esc)
1\x00a (esc) 1\x00a (esc)
2\x00a (esc) 2\x00a (esc)
#else #else
$ hg debugsidedata -c -v -- -1 $ hg debugsidedata -c -v -- -1
4 sidedata entries 2 sidedata entries
entry-0010 size 11 entry-0010 size 11
'0\x00a\n1\x00a\n2\x00a' '0\x00a\n1\x00a\n2\x00a'
entry-0011 size 0
''
entry-0012 size 5 entry-0012 size 5
'0\n1\n2' '0\n1\n2'
entry-0013 size 0
''
#endif #endif
$ hg showcopies $ hg showcopies
a -> b a -> b
a -> c a -> c
a -> d a -> d
#if extra #if extra
files: b b2 files: b b2
filesadded: 1 filesadded: 1
filesremoved: 0 filesremoved: 0
p1copies: 1\x00b (esc) p1copies: 1\x00b (esc)
#else #else
$ hg debugsidedata -c -v -- -1 $ hg debugsidedata -c -v -- -1
4 sidedata entries 3 sidedata entries
entry-0010 size 3 entry-0010 size 3
'1\x00b' '1\x00b'
entry-0011 size 0
''
entry-0012 size 1 entry-0012 size 1
'1' '1'
entry-0013 size 1 entry-0013 size 1
'0' '0'
#endif #endif
$ hg showcopies $ hg showcopies
b -> b2 b -> b2
$ hg changesetcopies $ hg changesetcopies
files: c files: c
p1copies: 0\x00b2 (esc) p1copies: 0\x00b2 (esc)
#else #else
$ hg debugsidedata -c -v -- -1 $ hg debugsidedata -c -v -- -1
4 sidedata entries 1 sidedata entries
entry-0010 size 4 entry-0010 size 4
'0\x00b2' '0\x00b2'
entry-0011 size 0
''
entry-0012 size 0
''
entry-0013 size 0
''
#endif #endif
$ hg showcopies $ hg showcopies
b2 -> c b2 -> c
#if extra #if extra
$ hg debugindex c $ hg debugindex c
2 2
p1copies: 0\x00a (esc) p1copies: 0\x00a (esc)
2\x00f (esc) 2\x00f (esc)
p2copies: 1\x00d (esc) p2copies: 1\x00d (esc)
#else #else
$ hg debugsidedata -c -v -- -1 $ hg debugsidedata -c -v -- -1
4 sidedata entries 3 sidedata entries
entry-0010 size 7 entry-0010 size 7
'0\x00a\n2\x00f' '0\x00a\n2\x00f'
entry-0011 size 3 entry-0011 size 3
'1\x00d' '1\x00d'
entry-0012 size 5 entry-0012 size 5
'0\n1\n2' '0\n1\n2'
entry-0013 size 0
''
#endif #endif
$ hg showcopies $ hg showcopies
a -> g a -> g
d -> h d -> h
f -> i f -> i
Test writing to both changeset and filelog Test writing to both changeset and filelog
$ hg cp a j $ hg cp a j
#if extra #if extra
$ hg ci -m 'copy a to j' --config experimental.copies.write-to=compatibility $ hg ci -m 'copy a to j' --config experimental.copies.write-to=compatibility
$ hg changesetcopies $ hg changesetcopies
files: j files: j
filesadded: 0 filesadded: 0
filesremoved: filesremoved:
p1copies: 0\x00a (esc) p1copies: 0\x00a (esc)
p2copies: p2copies:
#else #else
$ hg ci -m 'copy a to j' $ hg ci -m 'copy a to j'
$ hg debugsidedata -c -v -- -1 $ hg debugsidedata -c -v -- -1
4 sidedata entries 2 sidedata entries
entry-0010 size 3 entry-0010 size 3
'0\x00a' '0\x00a'
entry-0011 size 0
''
entry-0012 size 1 entry-0012 size 1
'0' '0'
entry-0013 size 0
''
#endif #endif
$ hg debugdata j 0 $ hg debugdata j 0
\x01 (esc) \x01 (esc)
copy: a copy: a
copyrev: b789fdd96dc2f3bd229c1dd8eedf0fc60e2b68e3 copyrev: b789fdd96dc2f3bd229c1dd8eedf0fc60e2b68e3
\x01 (esc) \x01 (esc)
a a
$ hg showcopies $ hg showcopies
saved backup bundle to $TESTTMP/repo/.hg/strip-backup/*-*-amend.hg (glob) saved backup bundle to $TESTTMP/repo/.hg/strip-backup/*-*-amend.hg (glob)
$ hg changesetcopies $ hg changesetcopies
files: j files: j
#else #else
$ hg ci --amend -m 'copy a to j, v2' $ hg ci --amend -m 'copy a to j, v2'
saved backup bundle to $TESTTMP/repo/.hg/strip-backup/*-*-amend.hg (glob) saved backup bundle to $TESTTMP/repo/.hg/strip-backup/*-*-amend.hg (glob)
$ hg debugsidedata -c -v -- -1 $ hg debugsidedata -c -v -- -1
4 sidedata entries 2 sidedata entries
entry-0010 size 3 entry-0010 size 3
'0\x00a' '0\x00a'
entry-0011 size 0
''
entry-0012 size 1 entry-0012 size 1
'0' '0'
entry-0013 size 0
''
#endif #endif
$ hg showcopies --config experimental.copies.read-from=filelog-only $ hg showcopies --config experimental.copies.read-from=filelog-only
a -> j a -> j
The entries should be written to extras even if they're empty (so the client The entries should be written to extras even if they're empty (so the client
won't have to fall back to reading from filelogs) won't have to fall back to reading from filelogs)
$ echo x >> j $ echo x >> j
#if extra #if extra
$ hg ci -m 'modify j' --config experimental.copies.write-to=compatibility $ hg ci -m 'modify j' --config experimental.copies.write-to=compatibility
$ hg changesetcopies $ hg changesetcopies
files: j files: j
filesadded: filesadded:
filesremoved: filesremoved:
p1copies: p1copies:
p2copies: p2copies:
#else #else
$ hg ci -m 'modify j' $ hg ci -m 'modify j'
$ hg debugsidedata -c -v -- -1 $ hg debugsidedata -c -v -- -1
4 sidedata entries
entry-0010 size 0
''
entry-0011 size 0
''
entry-0012 size 0
''
entry-0013 size 0
''
#endif #endif
Test writing only to filelog Test writing only to filelog
$ hg cp a k $ hg cp a k
#if extra #if extra
$ hg ci -m 'copy a to k' --config experimental.copies.write-to=filelog-only $ hg ci -m 'copy a to k' --config experimental.copies.write-to=filelog-only
$ hg changesetcopies $ hg changesetcopies
files: k files: k
#else #else
$ hg ci -m 'copy a to k' $ hg ci -m 'copy a to k'
$ hg debugsidedata -c -v -- -1 $ hg debugsidedata -c -v -- -1
4 sidedata entries 2 sidedata entries
entry-0010 size 3 entry-0010 size 3
'0\x00a' '0\x00a'
entry-0011 size 0
''
entry-0012 size 1 entry-0012 size 1
'0' '0'
entry-0013 size 0
''
#endif #endif
$ hg debugdata k 0 $ hg debugdata k 0
\x01 (esc) \x01 (esc)
copy: a copy: a
copyrev: b789fdd96dc2f3bd229c1dd8eedf0fc60e2b68e3 copyrev: b789fdd96dc2f3bd229c1dd8eedf0fc60e2b68e3
\x01 (esc) \x01 (esc)
a a
generaldelta: yes yes yes generaldelta: yes yes yes
sparserevlog: yes yes yes sparserevlog: yes yes yes
sidedata: yes yes no sidedata: yes yes no
copies-sdc: yes yes no copies-sdc: yes yes no
plain-cl-delta: yes yes yes plain-cl-delta: yes yes yes
compression: zlib zlib zlib compression: zlib zlib zlib
compression-level: default default default compression-level: default default default
$ hg debugsidedata -c -- 0 $ hg debugsidedata -c -- 0
4 sidedata entries 1 sidedata entries
entry-0010 size 0
entry-0011 size 0
entry-0012 size 1 entry-0012 size 1
entry-0013 size 0
$ hg debugsidedata -c -- 1 $ hg debugsidedata -c -- 1
4 sidedata entries 1 sidedata entries
entry-0010 size 0
entry-0011 size 0
entry-0012 size 0
entry-0013 size 1 entry-0013 size 1
$ hg debugsidedata -m -- 0 $ hg debugsidedata -m -- 0
$ cat << EOF > .hg/hgrc $ cat << EOF > .hg/hgrc
> [format] > [format]
> exp-use-side-data = yes > exp-use-side-data = yes
> exp-use-copies-side-data-changeset = no > exp-use-copies-side-data-changeset = no
> EOF > EOF
$ hg debugupgraderepo --run --quiet --no-backup > /dev/null $ hg debugupgraderepo --run --quiet --no-backup > /dev/null
generaldelta: yes yes yes generaldelta: yes yes yes
sparserevlog: yes yes yes sparserevlog: yes yes yes
sidedata: yes yes no sidedata: yes yes no
copies-sdc: yes yes no copies-sdc: yes yes no
plain-cl-delta: yes yes yes plain-cl-delta: yes yes yes
compression: zlib zlib zlib compression: zlib zlib zlib
compression-level: default default default compression-level: default default default
$ hg debugsidedata -c -- 0 $ hg debugsidedata -c -- 0
4 sidedata entries 1 sidedata entries
entry-0010 size 0
entry-0011 size 0
entry-0012 size 1 entry-0012 size 1
entry-0013 size 0
$ hg debugsidedata -c -- 1 $ hg debugsidedata -c -- 1
4 sidedata entries 1 sidedata entries
entry-0010 size 0
entry-0011 size 0
entry-0012 size 0
entry-0013 size 1 entry-0013 size 1
$ hg debugsidedata -m -- 0 $ hg debugsidedata -m -- 0
#endif #endif
$ cd .. $ cd ..