( )⚙ D10032 sidedata-exchange: rewrite sidedata on-the-fly whenever possible

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10032

sidedata-exchange: rewrite sidedata on-the-fly whenever possible
ClosedPublic

Authored by Alphare on Feb 19 2021, 6:16 AM.

Details

Reviewers
indygreg
marmoute
Group Reviewers
hg-reviewers
Commits
rHGba8e508a8e69: sidedata-exchange: rewrite sidedata on-the-fly whenever possible
Summary

When a A exchanges with B, the difference of their supported sidedata categories
is made, and the responsibility is always with the client to generated it:

  • If A pushes to B and B requires category foo that A does not have, A will need to generate it when sending it to B.
  • If A pulls from B and A needs category foo, it will generate foo before the end of the transaction.
  • Any category that is not required is removed. If peers are not compatible, abort.

It is forbidden to rewrite sidedata for a rev that already has sidedata, since
that would introduce unreachable (garbage) data in the data file, something
we're not prepared for yet.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

Alphare created this revision.Feb 19 2021, 6:16 AM
baymax updated this revision to Diff 25999.Mar 1 2021, 11:52 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

Alphare updated this revision to Diff 26095.Mar 4 2021, 10:18 AM
marmoute added a subscriber: marmoute.Mar 5 2021, 7:09 AM

Looks okay.

mercurial/changegroup.py
265–271

small nit: use an intermediate variable to avoid the spread on 5 lines.

376–381

The TODO comment should be duplicated near the declaration of the experimental config since we need this fixed before getting it out of experimental.

Alphare updated this revision to Diff 26226.Mar 10 2021, 2:12 PM
marmoute requested changes to this revision.Mar 11 2021, 1:21 PM
This revision now requires changes to proceed.Mar 11 2021, 1:21 PM
Alphare updated this revision to Diff 26259.Mar 12 2021, 6:34 AM
marmoute requested changes to this revision.Mar 12 2021, 1:11 PM
This revision now requires changes to proceed.Mar 12 2021, 1:11 PM
marmoute accepted this revision.Mar 15 2021, 7:01 AM
Alphare added a commit: rHGba8e508a8e69: sidedata-exchange: rewrite sidedata on-the-fly whenever possible.Mar 15 2021, 2:35 PM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHGba8e508a8e69: sidedata-exchange: rewrite sidedata on-the-fly whenever possible (authored by Alphare). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mmercurial/changegroup.py (97 lines)
Mmercurial/revlog.py (55 lines)
Mtests/test-copies-in-changeset.t (12 lines)
AMtests/test-sidedata-exchange.t (473 lines)
Ptests/testlib/ext-sidedata-2.py (106 lines)
Copied from tests/testlib/ext-sidedata.py
Ptests/testlib/ext-sidedata-3.py (118 lines)
Copied from tests/testlib/ext-sidedata.py
Ptests/testlib/ext-sidedata-4.py (87 lines)
Copied from tests/testlib/ext-sidedata.py
AMtests/testlib/ext-sidedata-5.py (81 lines)
Mtests/testlib/ext-sidedata.py (16 lines)

Diff 25677

mercurial/changegroup.py

# changegroup.py - Mercurial changegroup manipulation functions # changegroup.py - Mercurial changegroup manipulation functions
# #
# Copyright 2006 Matt Mackall <mpm@selenic.com> # Copyright 2006 Matt Mackall <mpm@selenic.com>
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
from __future__ import absolute_import from __future__ import absolute_import
import collections
import os import os
import struct import struct
import weakref import weakref
from .i18n import _ from .i18n import _
from .node import ( from .node import (
hex, hex,
nullid, nullid,
yield chunkheader(len(chunk)) yield chunkheader(len(chunk))
pos = 0 pos = 0
while pos < len(chunk): while pos < len(chunk):
next = pos + 2 ** 20 next = pos + 2 ** 20
yield chunk[pos:next] yield chunk[pos:next]
pos = next pos = next
yield closechunk() yield closechunk()
def _unpackmanifests(self, repo, revmap, trp, prog): def _unpackmanifests(self, repo, revmap, trp, prog, addrevisioncb=None):
self.callback = prog.increment self.callback = prog.increment
# no need to check for empty manifest group here: # no need to check for empty manifest group here:
# if the result of the merge of 1 and 2 is the same in 3 and 4, # if the result of the merge of 1 and 2 is the same in 3 and 4,
# no new manifest will be created and the manifest group will # no new manifest will be created and the manifest group will
# be empty during the pull # be empty during the pull
self.manifestheader() self.manifestheader()
deltas = self.deltaiter() deltas = self.deltaiter()
repo.manifestlog.getstorage(b'').addgroup(deltas, revmap, trp) repo.manifestlog.getstorage(b'').addgroup(
deltas,
revmap,
trp,
addrevisioncb=addrevisioncb,
)
prog.complete() prog.complete()
marmouteUnsubmitted
Done

small nit: use an intermediate variable to avoid the spread on 5 lines.

marmoute: small nit: use an intermediate variable to avoid the spread on 5 lines.
self.callback = None self.callback = None
def apply( def apply(
self, self,
repo, repo,
tr, tr,
srctype, srctype,
url, url,
changesets = clend - clstart changesets = clend - clstart
progress.complete() progress.complete()
del deltas del deltas
# TODO Python 2.7 removal # TODO Python 2.7 removal
# del efilesset # del efilesset
efilesset = None efilesset = None
self.callback = None self.callback = None
# Keep track of the (non-changelog) revlogs we've updated and their
# range of new revisions for sidedata rewrite.
# TODO do something more efficient than keeping the reference to
# the revlogs, especially memory-wise.
touched_manifests = {}
touched_filelogs = {}
marmouteUnsubmitted
Done

The TODO comment should be duplicated near the declaration of the experimental config since we need this fixed before getting it out of experimental.

marmoute: The TODO comment should be duplicated near the declaration of the experimental config since we…
# pull off the manifest group # pull off the manifest group
repo.ui.status(_(b"adding manifests\n")) repo.ui.status(_(b"adding manifests\n"))
# We know that we'll never have more manifests than we had # We know that we'll never have more manifests than we had
# changesets. # changesets.
progress = repo.ui.makeprogress( progress = repo.ui.makeprogress(
_(b'manifests'), unit=_(b'chunks'), total=changesets _(b'manifests'), unit=_(b'chunks'), total=changesets
) )
self._unpackmanifests(repo, revmap, trp, progress) on_manifest_rev = None
if sidedata_helpers and b'manifest' in sidedata_helpers[1]:
def on_manifest_rev(manifest, rev):
range = touched_manifests.get(manifest)
if not range:
touched_manifests[manifest] = (rev, rev)
else:
assert rev == range[1] + 1
touched_manifests[manifest] = (range[0], rev)
self._unpackmanifests(
repo,
revmap,
trp,
progress,
addrevisioncb=on_manifest_rev,
)
needfiles = {} needfiles = {}
if repo.ui.configbool(b'server', b'validate'): if repo.ui.configbool(b'server', b'validate'):
cl = repo.changelog cl = repo.changelog
ml = repo.manifestlog ml = repo.manifestlog
# validate incoming csets have their manifests # validate incoming csets have their manifests
for cset in pycompat.xrange(clstart, clend): for cset in pycompat.xrange(clstart, clend):
mfnode = cl.changelogrevision(cset).manifest mfnode = cl.changelogrevision(cset).manifest
mfest = ml[mfnode].readdelta() mfest = ml[mfnode].readdelta()
# store file nodes we must see # store file nodes we must see
for f, n in pycompat.iteritems(mfest): for f, n in pycompat.iteritems(mfest):
needfiles.setdefault(f, set()).add(n) needfiles.setdefault(f, set()).add(n)
on_filelog_rev = None
if sidedata_helpers and b'filelog' in sidedata_helpers[1]:
def on_filelog_rev(filelog, rev):
range = touched_filelogs.get(filelog)
if not range:
touched_filelogs[filelog] = (rev, rev)
else:
assert rev == range[1] + 1
touched_filelogs[filelog] = (range[0], rev)
# process the files # process the files
repo.ui.status(_(b"adding file changes\n")) repo.ui.status(_(b"adding file changes\n"))
newrevs, newfiles = _addchangegroupfiles( newrevs, newfiles = _addchangegroupfiles(
repo, self, revmap, trp, efiles, needfiles repo,
self,
revmap,
trp,
efiles,
needfiles,
addrevisioncb=on_filelog_rev,
)
if sidedata_helpers:
if b'changelog' in sidedata_helpers[1]:
cl.rewrite_sidedata(
sidedata_helpers, clstart, clend - 1, b"changelog"
)
for mf, (startrev, endrev) in touched_manifests.items():
mf.rewrite_sidedata(
sidedata_helpers, startrev, endrev, b"manifest"
)
for fl, (startrev, endrev) in touched_filelogs.items():
fl.rewrite_sidedata(
sidedata_helpers, startrev, endrev, b"filelog"
) )
# making sure the value exists # making sure the value exists
tr.changes.setdefault(b'changegroup-count-changesets', 0) tr.changes.setdefault(b'changegroup-count-changesets', 0)
tr.changes.setdefault(b'changegroup-count-revisions', 0) tr.changes.setdefault(b'changegroup-count-revisions', 0)
tr.changes.setdefault(b'changegroup-count-files', 0) tr.changes.setdefault(b'changegroup-count-files', 0)
tr.changes.setdefault(b'changegroup-count-heads', 0) tr.changes.setdefault(b'changegroup-count-heads', 0)
# some code use bundle operation for internal purpose. They usually # some code use bundle operation for internal purpose. They usually
deltaheadersize = deltaheader.size deltaheadersize = deltaheader.size
version = b'03' version = b'03'
_grouplistcount = 2 # One list of manifests and one list of files _grouplistcount = 2 # One list of manifests and one list of files
def _deltaheader(self, headertuple, prevnode): def _deltaheader(self, headertuple, prevnode):
node, p1, p2, deltabase, cs, flags = headertuple node, p1, p2, deltabase, cs, flags = headertuple
return node, p1, p2, deltabase, cs, flags return node, p1, p2, deltabase, cs, flags
def _unpackmanifests(self, repo, revmap, trp, prog): def _unpackmanifests(self, repo, revmap, trp, prog, addrevisioncb=None):
super(cg3unpacker, self)._unpackmanifests(repo, revmap, trp, prog) super(cg3unpacker, self)._unpackmanifests(
repo, revmap, trp, prog, addrevisioncb=addrevisioncb
)
for chunkdata in iter(self.filelogheader, {}): for chunkdata in iter(self.filelogheader, {}):
# If we get here, there are directory manifests in the changegroup # If we get here, there are directory manifests in the changegroup
d = chunkdata[b"filename"] d = chunkdata[b"filename"]
repo.ui.debug(b"adding %s revisions\n" % d) repo.ui.debug(b"adding %s revisions\n" % d)
deltas = self.deltaiter() deltas = self.deltaiter()
if not repo.manifestlog.getstorage(d).addgroup(deltas, revmap, trp): if not repo.manifestlog.getstorage(d).addgroup(
deltas, revmap, trp, addrevisioncb=addrevisioncb
):
raise error.Abort(_(b"received dir revlog group is empty")) raise error.Abort(_(b"received dir revlog group is empty"))
class cg4unpacker(cg3unpacker): class cg4unpacker(cg3unpacker):
"""Unpacker for cg4 streams. """Unpacker for cg4 streams.
cg4 streams add support for exchanging sidedata. cg4 streams add support for exchanging sidedata.
""" """
repo.filtername is None and heads == sorted(repo.heads()) repo.filtername is None and heads == sorted(repo.heads())
) )
repo.hook(b'preoutgoing', throw=True, source=source) repo.hook(b'preoutgoing', throw=True, source=source)
_changegroupinfo(repo, csets, source) _changegroupinfo(repo, csets, source)
return bundler.generate(commonrevs, csets, fastpathlinkrev, source) return bundler.generate(commonrevs, csets, fastpathlinkrev, source)
def _addchangegroupfiles(repo, source, revmap, trp, expectedfiles, needfiles): def _addchangegroupfiles(
repo,
source,
revmap,
trp,
expectedfiles,
needfiles,
addrevisioncb=None,
):
revisions = 0 revisions = 0
files = 0 files = 0
progress = repo.ui.makeprogress( progress = repo.ui.makeprogress(
_(b'files'), unit=_(b'files'), total=expectedfiles _(b'files'), unit=_(b'files'), total=expectedfiles
) )
for chunkdata in iter(source.filelogheader, {}): for chunkdata in iter(source.filelogheader, {}):
files += 1 files += 1
f = chunkdata[b"filename"] f = chunkdata[b"filename"]
repo.ui.debug(b"adding %s revisions\n" % f) repo.ui.debug(b"adding %s revisions\n" % f)
progress.increment() progress.increment()
fl = repo.file(f) fl = repo.file(f)
o = len(fl) o = len(fl)
try: try:
deltas = source.deltaiter() deltas = source.deltaiter()
if not fl.addgroup(deltas, revmap, trp): added = fl.addgroup(
deltas,
revmap,
trp,
addrevisioncb=addrevisioncb,
)
if not added:
raise error.Abort(_(b"received file revlog group is empty")) raise error.Abort(_(b"received file revlog group is empty"))
except error.CensoredBaseError as e: except error.CensoredBaseError as e:
raise error.Abort(_(b"received delta base is censored: %s") % e) raise error.Abort(_(b"received delta base is censored: %s") % e)
revisions += len(fl) - o revisions += len(fl) - o
if f in needfiles: if f in needfiles:
needs = needfiles[f] needs = needfiles[f]
for new in pycompat.xrange(o, len(fl)): for new in pycompat.xrange(o, len(fl)):
n = fl.node(new) n = fl.node(new)

mercurial/revlog.py

d[b'trackedsize'] = sum(map(self.rawsize, iter(self))) d[b'trackedsize'] = sum(map(self.rawsize, iter(self)))
if storedsize: if storedsize:
d[b'storedsize'] = sum( d[b'storedsize'] = sum(
self.opener.stat(path).st_size for path in self.files() self.opener.stat(path).st_size for path in self.files()
) )
return d return d
def rewrite_sidedata(self, helpers, startrev, endrev, kind):
if self.version & 0xFFFF != REVLOGV2:
return
# inline are not yet supported because they suffer from an issue when
# rewriting them (since it's not an append-only operation).
# See issue6485.
assert not self._inline
if not helpers[1] and not helpers[2]:
# Nothing to generate or remove
return
if kind not in (b"changelog", b"manifest", b"filelog"):
raise error.ProgrammingError(b"Unexpected revlog kind '%s'", kind)
new_entries = []
# append the new sidedata
with self._datafp(b'a+') as fp:
# Maybe this bug still exists, see revlog._writeentry
fp.seek(0, os.SEEK_END)
current_offset = fp.tell()
for rev in range(startrev, endrev + 1):
entry = self.index[rev]
new_sidedata = storageutil.run_sidedata_helpers(
store=self,
sidedata_helpers=helpers,
sidedata={},
rev=rev,
kind=kind,
)
serialized_sidedata = sidedatautil.serialize_sidedata(
new_sidedata
)
if entry[10] != 0 or entry[11] != 0:
# rewriting entries that already have sidedata is not
# supported yet, because it introduces garbage data in the
# revlog.
msg = "Rewriting existing sidedata is not supported yet"
raise error.Abort(msg)
entry = entry[:10]
entry += (current_offset, len(serialized_sidedata))
fp.write(serialized_sidedata)
new_entries.append(entry)
current_offset += len(serialized_sidedata)
# rewrite the new index entries
with self._indexfp(b'w+') as fp:
fp.seek(startrev * self._io.size)
for i, entry in enumerate(new_entries):
rev = startrev + i
self.index.replace(rev, entry)
packed = self._io.packentry(entry, self.node, self.version, rev)
fp.write(packed)

tests/test-copies-in-changeset.t

saved backup bundle to $TESTTMP/repo/.hg/strip-backup/*-*-amend.hg (glob) saved backup bundle to $TESTTMP/repo/.hg/strip-backup/*-*-amend.hg (glob)
$ hg changesetcopies $ hg changesetcopies
files: j files: j
#else #else
$ hg ci --amend -m 'copy a to j, v2' $ hg ci --amend -m 'copy a to j, v2'
saved backup bundle to $TESTTMP/repo/.hg/strip-backup/*-*-amend.hg (glob) saved backup bundle to $TESTTMP/repo/.hg/strip-backup/*-*-amend.hg (glob)
$ hg debugsidedata -c -v -- -1 $ hg debugsidedata -c -v -- -1
1 sidedata entries (missing-correct-output !) 1 sidedata entries
entry-0014 size 24 (missing-correct-output !) entry-0014 size 24
'\x00\x00\x00\x02\x00\x00\x00\x00\x01\x00\x00\x00\x00\x06\x00\x00\x00\x02\x00\x00\x00\x00aj' (missing-correct-output !) '\x00\x00\x00\x02\x00\x00\x00\x00\x01\x00\x00\x00\x00\x06\x00\x00\x00\x02\x00\x00\x00\x00aj'
#endif #endif
$ hg showcopies --config experimental.copies.read-from=filelog-only $ hg showcopies --config experimental.copies.read-from=filelog-only
a -> j (sidedata missing-correct-output !) a -> j
a -> j (no-sidedata !)
The entries should be written to extras even if they're empty (so the client The entries should be written to extras even if they're empty (so the client
won't have to fall back to reading from filelogs) won't have to fall back to reading from filelogs)
$ echo x >> j $ echo x >> j
#if extra #if extra
$ hg ci -m 'modify j' --config experimental.copies.write-to=compatibility $ hg ci -m 'modify j' --config experimental.copies.write-to=compatibility
$ hg changesetcopies $ hg changesetcopies
files: j files: j
filesadded: filesadded:
on-disk merge (no conflicts), so we force it to be in-memory on-disk merge (no conflicts), so we force it to be in-memory
with no fallback. with no fallback.
$ hg rebase -d 1 --config rebase.experimental.inmemory=yes --config devel.rebase.force-in-memory-merge=yes $ hg rebase -d 1 --config rebase.experimental.inmemory=yes --config devel.rebase.force-in-memory-merge=yes
rebasing 2:* tip "rename a to b" (glob) rebasing 2:* tip "rename a to b" (glob)
merging a and b to b merging a and b to b
saved backup bundle to $TESTTMP/rebase-rename/.hg/strip-backup/*-*-rebase.hg (glob) saved backup bundle to $TESTTMP/rebase-rename/.hg/strip-backup/*-*-rebase.hg (glob)
$ hg st --change . --copies $ hg st --change . --copies
A b A b
a (sidedata missing-correct-output !) a
a (no-sidedata !)
R a R a
$ cd .. $ cd ..
Test splitting a commit Test splitting a commit
$ hg init split $ hg init split
$ cd split $ cd split
$ echo a > a $ echo a > a

tests/test-sidedata-exchange.t

  • This file was added.
===========================
Tests for sidedata exchange
===========================
Check simple exchange behavior
==============================
Pusher and pushed have sidedata enabled
---------------------------------------
$ hg init sidedata-source --config format.exp-use-side-data=yes
$ cat << EOF >> sidedata-source/.hg/hgrc
> [extensions]
> testsidedata=$TESTDIR/testlib/ext-sidedata-5.py
> EOF
$ hg init sidedata-target --config format.exp-use-side-data=yes
$ cat << EOF >> sidedata-target/.hg/hgrc
> [extensions]
> testsidedata=$TESTDIR/testlib/ext-sidedata-5.py
> EOF
$ cd sidedata-source
$ echo a > a
$ echo b > b
$ echo c > c
$ hg commit -Am "initial"
adding a
adding b
adding c
$ echo aa > a
$ hg commit -m "other"
$ hg push -r . ../sidedata-target
pushing to ../sidedata-target
searching for changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 4 changes to 3 files
$ hg -R ../sidedata-target debugsidedata -c 0
2 sidedata entries
entry-0001 size 4
entry-0002 size 32
$ hg -R ../sidedata-target debugsidedata -c 1 -v
2 sidedata entries
entry-0001 size 4
'\x00\x00\x00:'
entry-0002 size 32
'\xa3\xee4v\x99\x85$\x9f\x1f\x8dKe\x0f\xc3\x9d-\xc9\xb5%[\x15=h\xe9\xf2O\xb5\xd9\x1f*\xff\xe5'
$ hg -R ../sidedata-target debugsidedata -m 0
2 sidedata entries
entry-0001 size 4
entry-0002 size 32
$ hg -R ../sidedata-target debugsidedata -m 1 -v
2 sidedata entries
entry-0001 size 4
'\x00\x00\x00\x81'
entry-0002 size 32
'-bL\xc5\xa4uu"#\xac\x1b`,\xc0\xbc\x9d\xf5\xac\xf0\x1d\x89)2\xf8N\xb1\x14m\xce\xd7\xbc\xae'
$ hg -R ../sidedata-target debugsidedata a 0
2 sidedata entries
entry-0001 size 4
entry-0002 size 32
$ hg -R ../sidedata-target debugsidedata a 1 -v
2 sidedata entries
entry-0001 size 4
'\x00\x00\x00\x03'
entry-0002 size 32
'\xd9\xcd\x81UvL5C\xf1\x0f\xad\x8aH\rt17Fo\x8dU!<\x8e\xae\xfc\xd1/\x06\xd4:\x80'
$ cd ..
Puller and pulled have sidedata enabled
---------------------------------------
$ rm -rf sidedata-source sidedata-target
$ hg init sidedata-source --config format.exp-use-side-data=yes
$ cat << EOF >> sidedata-source/.hg/hgrc
> [extensions]
> testsidedata=$TESTDIR/testlib/ext-sidedata-5.py
> EOF
$ hg init sidedata-target --config format.exp-use-side-data=yes
$ cat << EOF >> sidedata-target/.hg/hgrc
> [extensions]
> testsidedata=$TESTDIR/testlib/ext-sidedata-5.py
> EOF
$ cd sidedata-source
$ echo a > a
$ echo b > b
$ echo c > c
$ hg commit -Am "initial"
adding a
adding b
adding c
$ echo aa > a
$ hg commit -m "other"
$ hg pull -R ../sidedata-target ../sidedata-source
pulling from ../sidedata-source
requesting all changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 4 changes to 3 files
new changesets 05da661850d7:7ec8b4049447
(run 'hg update' to get a working copy)
$ hg -R ../sidedata-target debugsidedata -c 0
2 sidedata entries
entry-0001 size 4
entry-0002 size 32
$ hg -R ../sidedata-target debugsidedata -c 1 -v
2 sidedata entries
entry-0001 size 4
'\x00\x00\x00:'
entry-0002 size 32
'\xa3\xee4v\x99\x85$\x9f\x1f\x8dKe\x0f\xc3\x9d-\xc9\xb5%[\x15=h\xe9\xf2O\xb5\xd9\x1f*\xff\xe5'
$ hg -R ../sidedata-target debugsidedata -m 0
2 sidedata entries
entry-0001 size 4
entry-0002 size 32
$ hg -R ../sidedata-target debugsidedata -m 1 -v
2 sidedata entries
entry-0001 size 4
'\x00\x00\x00\x81'
entry-0002 size 32
'-bL\xc5\xa4uu"#\xac\x1b`,\xc0\xbc\x9d\xf5\xac\xf0\x1d\x89)2\xf8N\xb1\x14m\xce\xd7\xbc\xae'
$ hg -R ../sidedata-target debugsidedata a 0
2 sidedata entries
entry-0001 size 4
entry-0002 size 32
$ hg -R ../sidedata-target debugsidedata a 1 -v
2 sidedata entries
entry-0001 size 4
'\x00\x00\x00\x03'
entry-0002 size 32
'\xd9\xcd\x81UvL5C\xf1\x0f\xad\x8aH\rt17Fo\x8dU!<\x8e\xae\xfc\xd1/\x06\xd4:\x80'
$ cd ..
Now on to asymmetric configs.
Pusher has sidedata enabled, pushed does not
--------------------------------------------
$ rm -rf sidedata-source sidedata-target
$ hg init sidedata-source --config format.exp-use-side-data=yes
$ cat << EOF >> sidedata-source/.hg/hgrc
> [extensions]
> testsidedata=$TESTDIR/testlib/ext-sidedata-5.py
> EOF
$ hg init sidedata-target --config format.exp-use-side-data=no
$ cd sidedata-source
$ echo a > a
$ echo b > b
$ echo c > c
$ hg commit -Am "initial"
adding a
adding b
adding c
$ echo aa > a
$ hg commit -m "other"
$ hg push -r . ../sidedata-target --traceback
pushing to ../sidedata-target
searching for changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 4 changes to 3 files
$ hg -R ../sidedata-target log -G
o changeset: 1:7ec8b4049447
| tag: tip
| user: test
| date: Thu Jan 01 00:00:00 1970 +0000
| summary: other
|
o changeset: 0:05da661850d7
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: initial
$ hg -R ../sidedata-target debugsidedata -c 0
$ hg -R ../sidedata-target debugsidedata -c 1 -v
$ hg -R ../sidedata-target debugsidedata -m 0
$ hg -R ../sidedata-target debugsidedata -m 1 -v
$ hg -R ../sidedata-target debugsidedata a 0
$ hg -R ../sidedata-target debugsidedata a 1 -v
$ cd ..
Pulled has sidedata enabled, puller does not
--------------------------------------------
$ rm -rf sidedata-source sidedata-target
$ hg init sidedata-source --config format.exp-use-side-data=yes
$ cat << EOF >> sidedata-source/.hg/hgrc
> [extensions]
> testsidedata=$TESTDIR/testlib/ext-sidedata-5.py
> EOF
$ hg init sidedata-target --config format.exp-use-side-data=no
$ cd sidedata-source
$ echo a > a
$ echo b > b
$ echo c > c
$ hg commit -Am "initial"
adding a
adding b
adding c
$ echo aa > a
$ hg commit -m "other"
$ hg pull -R ../sidedata-target ../sidedata-source
pulling from ../sidedata-source
requesting all changes
adding changesets
adding manifests
adding file changes
added 2 changesets with 4 changes to 3 files
new changesets 05da661850d7:7ec8b4049447
(run 'hg update' to get a working copy)
$ hg -R ../sidedata-target log -G
o changeset: 1:7ec8b4049447
| tag: tip
| user: test
| date: Thu Jan 01 00:00:00 1970 +0000
| summary: other
|
o changeset: 0:05da661850d7
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: initial
$ hg -R ../sidedata-target debugsidedata -c 0
$ hg -R ../sidedata-target debugsidedata -c 1 -v
$ hg -R ../sidedata-target debugsidedata -m 0
$ hg -R ../sidedata-target debugsidedata -m 1 -v
$ hg -R ../sidedata-target debugsidedata a 0
$ hg -R ../sidedata-target debugsidedata a 1 -v
$ cd ..
Check sidedata exchange with on-the-fly generation and removal
==============================================================
(Push) Target has strict superset of the source
-----------------------------------------------
$ hg init source-repo --config format.exp-use-side-data=yes
$ hg init target-repo --config format.exp-use-side-data=yes
$ cat << EOF >> target-repo/.hg/hgrc
> [extensions]
> testsidedata=$TESTDIR/testlib/ext-sidedata.py
> EOF
$ cd source-repo
$ echo aaa > a
$ hg add a
$ hg commit -m a
$ echo aaa > b
$ hg add b
$ hg commit -m b
$ echo xxx >> a
$ hg commit -m aa
No sidedata is generated in the source
$ hg debugsidedata -c 0
Check that sidedata capabilities are advertised
$ hg debugcapabilities ../target-repo | grep sidedata
exp-wanted-sidedata=1,2
We expect the client to abort the push since it's not capable of generating
what the server is asking
$ hg push -r . ../target-repo
pushing to ../target-repo
abort: cannot push: required sidedata category not supported by this client: '1'
[255]
Add the required capabilities
$ cat << EOF >> .hg/hgrc
> [extensions]
> testsidedata2=$TESTDIR/testlib/ext-sidedata-2.py
> EOF
We expect the target to have sidedata that was generated by the source on push
$ hg push -r . ../target-repo
pushing to ../target-repo
searching for changes
adding changesets
adding manifests
adding file changes
added 3 changesets with 3 changes to 2 files
$ cd ../target-repo
$ hg debugsidedata -c 0
2 sidedata entries
entry-0001 size 4
entry-0002 size 32
$ hg debugsidedata -c 1 -v
2 sidedata entries
entry-0001 size 4
'\x00\x00\x006'
entry-0002 size 32
'\x98\t\xf9\xc4v\xf0\xc5P\x90\xf7wRf\xe8\xe27e\xfc\xc1\x93\xa4\x96\xd0\x1d\x97\xaaG\x1d\xd7t\xfa\xde'
$ hg debugsidedata -m 2
2 sidedata entries
entry-0001 size 4
entry-0002 size 32
$ hg debugsidedata a 1
2 sidedata entries
entry-0001 size 4
entry-0002 size 32
$ cd ..
(Push) Difference is not subset/superset
----------------------------------------
Source has one in common, one missing and one more sidedata category with the
target.
$ rm -rf source-repo target-repo
$ hg init source-repo --config format.exp-use-side-data=yes
$ cat << EOF >> source-repo/.hg/hgrc
> [extensions]
> testsidedata3=$TESTDIR/testlib/ext-sidedata-3.py
> EOF
$ hg init target-repo --config format.exp-use-side-data=yes
$ cat << EOF >> target-repo/.hg/hgrc
> [extensions]
> testsidedata4=$TESTDIR/testlib/ext-sidedata-4.py
> EOF
$ cd source-repo
$ echo aaa > a
$ hg add a
$ hg commit -m a
$ echo aaa > b
$ hg add b
$ hg commit -m b
$ echo xxx >> a
$ hg commit -m aa
Check that sidedata capabilities are advertised
$ hg debugcapabilities . | grep sidedata
exp-wanted-sidedata=1,2
$ hg debugcapabilities ../target-repo | grep sidedata
exp-wanted-sidedata=2,3
Sidedata is generated in the source, but only the right categories (entry-0001 and entry-0002)
$ hg debugsidedata -c 0
2 sidedata entries
entry-0001 size 4
entry-0002 size 32
$ hg debugsidedata -c 1 -v
2 sidedata entries
entry-0001 size 4
'\x00\x00\x006'
entry-0002 size 32
'\x98\t\xf9\xc4v\xf0\xc5P\x90\xf7wRf\xe8\xe27e\xfc\xc1\x93\xa4\x96\xd0\x1d\x97\xaaG\x1d\xd7t\xfa\xde'
$ hg debugsidedata -m 2
2 sidedata entries
entry-0001 size 4
entry-0002 size 32
$ hg debugsidedata a 1
2 sidedata entries
entry-0001 size 4
entry-0002 size 32
We expect the target to have sidedata that was generated by the source on push,
and also removed the sidedata categories that are not supported by the target.
Namely, we expect entry-0002 (only exchanged) and entry-0003 (generated),
but not entry-0001.
$ hg push -r . ../target-repo --traceback
pushing to ../target-repo
searching for changes
adding changesets
adding manifests
adding file changes
added 3 changesets with 3 changes to 2 files
$ cd ../target-repo
$ hg log -G
o changeset: 2:40f977031323
| tag: tip
| user: test
| date: Thu Jan 01 00:00:00 1970 +0000
| summary: aa
|
o changeset: 1:2707720c6597
| user: test
| date: Thu Jan 01 00:00:00 1970 +0000
| summary: b
|
o changeset: 0:7049e48789d7
user: test
date: Thu Jan 01 00:00:00 1970 +0000
summary: a
$ hg debugsidedata -c 0
2 sidedata entries
entry-0002 size 32
entry-0003 size 48
$ hg debugsidedata -c 1 -v
2 sidedata entries
entry-0002 size 32
'\x98\t\xf9\xc4v\xf0\xc5P\x90\xf7wRf\xe8\xe27e\xfc\xc1\x93\xa4\x96\xd0\x1d\x97\xaaG\x1d\xd7t\xfa\xde'
entry-0003 size 48
'\x87\xcf\xdfI/\xb5\xed\xeaC\xc1\xf0S\xf3X\x1c\xcc\x00m\xee\xe6#\xc1\xe3\xcaB8Fk\x82e\xfc\xc01\xf6\xb7\xb9\xb3([\xf6D\xa6\xcf\x9b\xea\x11{\x08'
$ hg debugsidedata -m 2
2 sidedata entries
entry-0002 size 32
entry-0003 size 48
$ hg debugsidedata a 1
2 sidedata entries
entry-0002 size 32
entry-0003 size 48
$ cd ..
(Pull) Target has strict superset of the source
-----------------------------------------------
$ rm -rf source-repo target-repo
$ hg init source-repo --config format.exp-use-side-data=yes
$ hg init target-repo --config format.exp-use-side-data=yes
$ cat << EOF >> target-repo/.hg/hgrc
> [extensions]
> testsidedata=$TESTDIR/testlib/ext-sidedata.py
> EOF
$ cd source-repo
$ echo aaa > a
$ hg add a
$ hg commit -m a
$ echo aaa > b
$ hg add b
$ hg commit -m b
$ echo xxx >> a
$ hg commit -m aa
No sidedata is generated in the source
$ hg debugsidedata -c 0
Check that sidedata capabilities are advertised
$ hg debugcapabilities ../target-repo | grep sidedata
exp-wanted-sidedata=1,2
$ cd ../target-repo
Add the required capabilities
$ cat << EOF >> .hg/hgrc
> [extensions]
> testsidedata2=$TESTDIR/testlib/ext-sidedata-2.py
> EOF
We expect the target to have sidedata that it generated on-the-fly during pull
$ hg pull -r . ../source-repo --traceback
pulling from ../source-repo
adding changesets
adding manifests
adding file changes
added 3 changesets with 3 changes to 2 files
new changesets 7049e48789d7:40f977031323
(run 'hg update' to get a working copy)
$ hg debugsidedata -c 0 --traceback
2 sidedata entries
entry-0001 size 4
entry-0002 size 32
$ hg debugsidedata -c 1 -v --traceback
2 sidedata entries
entry-0001 size 4
'\x00\x00\x006'
entry-0002 size 32
'\x98\t\xf9\xc4v\xf0\xc5P\x90\xf7wRf\xe8\xe27e\xfc\xc1\x93\xa4\x96\xd0\x1d\x97\xaaG\x1d\xd7t\xfa\xde'
$ hg debugsidedata -m 2
2 sidedata entries
entry-0001 size 4
entry-0002 size 32
$ hg debugsidedata a 1
2 sidedata entries
entry-0001 size 4
entry-0002 size 32
$ cd ..

tests/testlib/ext-sidedata-2.py

  • This file was copied from tests/testlib/ext-sidedata.py.
# ext-sidedata.py - small extension to test the sidedata logic # coding: utf8
# ext-sidedata-2.py - small extension to test (differently) the sidedata logic
# #
# Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net) # Simulates a client for a complex sidedata exchange.
#
# Copyright 2021 Raphaël Gomès <rgomes@octobus.net>
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
from __future__ import absolute_import from __future__ import absolute_import
import hashlib import hashlib
import struct import struct
from mercurial.node import ( from mercurial.revlogutils import sidedata as sidedatamod
nullid,
nullrev,
)
from mercurial import (
extensions,
requirements,
revlog,
)
from mercurial.upgrade_utils import engine as upgrade_engine
from mercurial.revlogutils import sidedata
def wrapaddrevision( def compute_sidedata_1(repo, revlog, rev, sidedata, text=None):
orig, self, text, transaction, link, p1, p2, *args, **kwargs sidedata = sidedata.copy()
): if text is None:
if kwargs.get('sidedata') is None: text = revlog.revision(rev)
kwargs['sidedata'] = {} sidedata[sidedatamod.SD_TEST1] = struct.pack('>I', len(text))
sd = kwargs['sidedata'] return sidedata
## let's store some arbitrary data just for testing
# text length
sd[sidedata.SD_TEST1] = struct.pack('>I', len(text))
# and sha2 hashes
sha256 = hashlib.sha256(text).digest()
sd[sidedata.SD_TEST2] = struct.pack('>32s', sha256)
return orig(self, text, transaction, link, p1, p2, *args, **kwargs)
def wrap_revisiondata(orig, self, nodeorrev, *args, **kwargs): def compute_sidedata_2(repo, revlog, rev, sidedata, text=None):
text, sd = orig(self, nodeorrev, *args, **kwargs) sidedata = sidedata.copy()
if getattr(self, 'sidedatanocheck', False): if text is None:
return text, sd
if self.version & 0xFFFF != 2:
return text, sd
if nodeorrev != nullrev and nodeorrev != nullid:
if len(text) != struct.unpack('>I', sd[sidedata.SD_TEST1])[0]:
raise RuntimeError('text size mismatch')
expected = sd[sidedata.SD_TEST2]
got = hashlib.sha256(text).digest()
if got != expected:
raise RuntimeError('sha256 mismatch')
return text, sd
def wrapgetsidedatacompanion(orig, srcrepo, dstrepo):
sidedatacompanion = orig(srcrepo, dstrepo)
addedreqs = dstrepo.requirements - srcrepo.requirements
if requirements.SIDEDATA_REQUIREMENT in addedreqs:
assert sidedatacompanion is None # deal with composition later
def sidedatacompanion(revlog, rev):
update = {}
revlog.sidedatanocheck = True
try:
text = revlog.revision(rev) text = revlog.revision(rev)
finally:
del revlog.sidedatanocheck
## let's store some arbitrary data just for testing
# text length
update[sidedata.SD_TEST1] = struct.pack('>I', len(text))
# and sha2 hashes
sha256 = hashlib.sha256(text).digest() sha256 = hashlib.sha256(text).digest()
update[sidedata.SD_TEST2] = struct.pack('>32s', sha256) sidedata[sidedatamod.SD_TEST2] = struct.pack('>32s', sha256)
return False, (), update, 0, 0 return sidedata
return sidedatacompanion
def reposetup(ui, repo):
def extsetup(ui): # Sidedata keys happen to be the same as the categories, easier for testing.
extensions.wrapfunction(revlog.revlog, 'addrevision', wrapaddrevision) for kind in (b'changelog', b'manifest', b'filelog'):
extensions.wrapfunction(revlog.revlog, '_revisiondata', wrap_revisiondata) repo.register_sidedata_computer(
extensions.wrapfunction( kind,
upgrade_engine, 'getsidedatacompanion', wrapgetsidedatacompanion sidedatamod.SD_TEST1,
(sidedatamod.SD_TEST1,),
compute_sidedata_1,
)
repo.register_sidedata_computer(
kind,
sidedatamod.SD_TEST2,
(sidedatamod.SD_TEST2,),
compute_sidedata_2,
) )

tests/testlib/ext-sidedata-3.py

  • This file was copied from tests/testlib/ext-sidedata.py.
# ext-sidedata.py - small extension to test the sidedata logic # coding: utf8
# ext-sidedata-3.py - small extension to test (differently still) the sidedata
# logic
# #
# Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net) # Simulates a client for a complex sidedata exchange.
#
# Copyright 2021 Raphaël Gomès <rgomes@octobus.net>
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
from __future__ import absolute_import from __future__ import absolute_import
import hashlib import hashlib
import struct import struct
from mercurial.node import (
nullid,
nullrev,
)
from mercurial import ( from mercurial import (
extensions, extensions,
requirements,
revlog, revlog,
) )
from mercurial.upgrade_utils import engine as upgrade_engine from mercurial.revlogutils import sidedata as sidedatamod
def compute_sidedata_1(repo, revlog, rev, sidedata, text=None):
sidedata = sidedata.copy()
if text is None:
text = revlog.revision(rev)
sidedata[sidedatamod.SD_TEST1] = struct.pack('>I', len(text))
return sidedata
from mercurial.revlogutils import sidedata def compute_sidedata_2(repo, revlog, rev, sidedata, text=None):
sidedata = sidedata.copy()
if text is None:
text = revlog.revision(rev)
sha256 = hashlib.sha256(text).digest()
sidedata[sidedatamod.SD_TEST2] = struct.pack('>32s', sha256)
return sidedata
def compute_sidedata_3(repo, revlog, rev, sidedata, text=None):
sidedata = sidedata.copy()
if text is None:
text = revlog.revision(rev)
sha384 = hashlib.sha384(text).digest()
sidedata[sidedatamod.SD_TEST3] = struct.pack('>48s', sha384)
return sidedata
def wrapaddrevision( def wrapaddrevision(
orig, self, text, transaction, link, p1, p2, *args, **kwargs orig, self, text, transaction, link, p1, p2, *args, **kwargs
): ):
if kwargs.get('sidedata') is None: if kwargs.get('sidedata') is None:
kwargs['sidedata'] = {} kwargs['sidedata'] = {}
sd = kwargs['sidedata'] sd = kwargs['sidedata']
## let's store some arbitrary data just for testing sd = compute_sidedata_1(None, self, None, sd, text=text)
# text length kwargs['sidedata'] = compute_sidedata_2(None, self, None, sd, text=text)
sd[sidedata.SD_TEST1] = struct.pack('>I', len(text))
# and sha2 hashes
sha256 = hashlib.sha256(text).digest()
sd[sidedata.SD_TEST2] = struct.pack('>32s', sha256)
return orig(self, text, transaction, link, p1, p2, *args, **kwargs) return orig(self, text, transaction, link, p1, p2, *args, **kwargs)
def wrap_revisiondata(orig, self, nodeorrev, *args, **kwargs):
text, sd = orig(self, nodeorrev, *args, **kwargs)
if getattr(self, 'sidedatanocheck', False):
return text, sd
if self.version & 0xFFFF != 2:
return text, sd
if nodeorrev != nullrev and nodeorrev != nullid:
if len(text) != struct.unpack('>I', sd[sidedata.SD_TEST1])[0]:
raise RuntimeError('text size mismatch')
expected = sd[sidedata.SD_TEST2]
got = hashlib.sha256(text).digest()
if got != expected:
raise RuntimeError('sha256 mismatch')
return text, sd
def wrapgetsidedatacompanion(orig, srcrepo, dstrepo):
sidedatacompanion = orig(srcrepo, dstrepo)
addedreqs = dstrepo.requirements - srcrepo.requirements
if requirements.SIDEDATA_REQUIREMENT in addedreqs:
assert sidedatacompanion is None # deal with composition later
def sidedatacompanion(revlog, rev):
update = {}
revlog.sidedatanocheck = True
try:
text = revlog.revision(rev)
finally:
del revlog.sidedatanocheck
## let's store some arbitrary data just for testing
# text length
update[sidedata.SD_TEST1] = struct.pack('>I', len(text))
# and sha2 hashes
sha256 = hashlib.sha256(text).digest()
update[sidedata.SD_TEST2] = struct.pack('>32s', sha256)
return False, (), update, 0, 0
return sidedatacompanion
def extsetup(ui): def extsetup(ui):
extensions.wrapfunction(revlog.revlog, 'addrevision', wrapaddrevision) extensions.wrapfunction(revlog.revlog, 'addrevision', wrapaddrevision)
extensions.wrapfunction(revlog.revlog, '_revisiondata', wrap_revisiondata)
extensions.wrapfunction(
upgrade_engine, 'getsidedatacompanion', wrapgetsidedatacompanion def reposetup(ui, repo):
# Sidedata keys happen to be the same as the categories, easier for testing.
for kind in (b'changelog', b'manifest', b'filelog'):
repo.register_sidedata_computer(
kind,
sidedatamod.SD_TEST1,
(sidedatamod.SD_TEST1,),
compute_sidedata_1,
)
repo.register_sidedata_computer(
kind,
sidedatamod.SD_TEST2,
(sidedatamod.SD_TEST2,),
compute_sidedata_2,
)
repo.register_sidedata_computer(
kind,
sidedatamod.SD_TEST3,
(sidedatamod.SD_TEST3,),
compute_sidedata_3,
) )
repo.register_wanted_sidedata(sidedatamod.SD_TEST1)
repo.register_wanted_sidedata(sidedatamod.SD_TEST2)

tests/testlib/ext-sidedata-4.py

  • This file was copied from tests/testlib/ext-sidedata.py.
# ext-sidedata.py - small extension to test the sidedata logic # coding: utf8
# ext-sidedata-4.py - small extension to test (differently still) the sidedata
# logic
# #
# Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net) # Simulates a server for a complex sidedata exchange.
#
# Copyright 2021 Raphaël Gomès <rgomes@octobus.net>
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
from __future__ import absolute_import from __future__ import absolute_import
import hashlib
import struct
from mercurial.node import (
nullid,
nullrev,
)
from mercurial import (
extensions,
requirements,
revlog,
)
from mercurial.upgrade_utils import engine as upgrade_engine
from mercurial.revlogutils import sidedata from mercurial.revlogutils import sidedata
def wrapaddrevision( def reposetup(ui, repo):
orig, self, text, transaction, link, p1, p2, *args, **kwargs repo.register_wanted_sidedata(sidedata.SD_TEST2)
): repo.register_wanted_sidedata(sidedata.SD_TEST3)
if kwargs.get('sidedata') is None:
kwargs['sidedata'] = {}
sd = kwargs['sidedata']
## let's store some arbitrary data just for testing
# text length
sd[sidedata.SD_TEST1] = struct.pack('>I', len(text))
# and sha2 hashes
sha256 = hashlib.sha256(text).digest()
sd[sidedata.SD_TEST2] = struct.pack('>32s', sha256)
return orig(self, text, transaction, link, p1, p2, *args, **kwargs)
def wrap_revisiondata(orig, self, nodeorrev, *args, **kwargs):
text, sd = orig(self, nodeorrev, *args, **kwargs)
if getattr(self, 'sidedatanocheck', False):
return text, sd
if self.version & 0xFFFF != 2:
return text, sd
if nodeorrev != nullrev and nodeorrev != nullid:
if len(text) != struct.unpack('>I', sd[sidedata.SD_TEST1])[0]:
raise RuntimeError('text size mismatch')
expected = sd[sidedata.SD_TEST2]
got = hashlib.sha256(text).digest()
if got != expected:
raise RuntimeError('sha256 mismatch')
return text, sd
def wrapgetsidedatacompanion(orig, srcrepo, dstrepo):
sidedatacompanion = orig(srcrepo, dstrepo)
addedreqs = dstrepo.requirements - srcrepo.requirements
if requirements.SIDEDATA_REQUIREMENT in addedreqs:
assert sidedatacompanion is None # deal with composition later
def sidedatacompanion(revlog, rev):
update = {}
revlog.sidedatanocheck = True
try:
text = revlog.revision(rev)
finally:
del revlog.sidedatanocheck
## let's store some arbitrary data just for testing
# text length
update[sidedata.SD_TEST1] = struct.pack('>I', len(text))
# and sha2 hashes
sha256 = hashlib.sha256(text).digest()
update[sidedata.SD_TEST2] = struct.pack('>32s', sha256)
return False, (), update, 0, 0
return sidedatacompanion
def extsetup(ui):
extensions.wrapfunction(revlog.revlog, 'addrevision', wrapaddrevision)
extensions.wrapfunction(revlog.revlog, '_revisiondata', wrap_revisiondata)
extensions.wrapfunction(
upgrade_engine, 'getsidedatacompanion', wrapgetsidedatacompanion
)

tests/testlib/ext-sidedata-5.py

  • This file was added.
# coding: utf8
# ext-sidedata-5.py - small extension to test (differently still) the sidedata
# logic
#
# Simulates a server for a simple sidedata exchange.
#
# Copyright 2021 Raphaël Gomès <rgomes@octobus.net>
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version.
from __future__ import absolute_import
import hashlib
import struct
from mercurial import (
extensions,
revlog,
)
from mercurial.revlogutils import sidedata as sidedatamod
def compute_sidedata_1(repo, revlog, rev, sidedata, text=None):
sidedata = sidedata.copy()
if text is None:
text = revlog.revision(rev)
sidedata[sidedatamod.SD_TEST1] = struct.pack('>I', len(text))
return sidedata
def compute_sidedata_2(repo, revlog, rev, sidedata, text=None):
sidedata = sidedata.copy()
if text is None:
text = revlog.revision(rev)
sha256 = hashlib.sha256(text).digest()
sidedata[sidedatamod.SD_TEST2] = struct.pack('>32s', sha256)
return sidedata
def reposetup(ui, repo):
# Sidedata keys happen to be the same as the categories, easier for testing.
for kind in (b'changelog', b'manifest', b'filelog'):
repo.register_sidedata_computer(
kind,
sidedatamod.SD_TEST1,
(sidedatamod.SD_TEST1,),
compute_sidedata_1,
)
repo.register_sidedata_computer(
kind,
sidedatamod.SD_TEST2,
(sidedatamod.SD_TEST2,),
compute_sidedata_2,
)
# We don't register sidedata computers because we don't care within these
# tests
repo.register_wanted_sidedata(sidedatamod.SD_TEST1)
repo.register_wanted_sidedata(sidedatamod.SD_TEST2)
def wrapaddrevision(
orig, self, text, transaction, link, p1, p2, *args, **kwargs
):
if kwargs.get('sidedata') is None:
kwargs['sidedata'] = {}
sd = kwargs['sidedata']
## let's store some arbitrary data just for testing
# text length
sd[sidedatamod.SD_TEST1] = struct.pack('>I', len(text))
# and sha2 hashes
sha256 = hashlib.sha256(text).digest()
sd[sidedatamod.SD_TEST2] = struct.pack('>32s', sha256)
return orig(self, text, transaction, link, p1, p2, *args, **kwargs)
def extsetup(ui):
extensions.wrapfunction(revlog.revlog, 'addrevision', wrapaddrevision)

tests/testlib/ext-sidedata.py

# ext-sidedata.py - small extension to test the sidedata logic # ext-sidedata.py - small extension to test the sidedata logic
# #
# Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net) # Copyright 2019 Pierre-Yves David <pierre-yves.david@octobus.net>
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
from __future__ import absolute_import from __future__ import absolute_import
import hashlib import hashlib
import struct import struct
def wrap_revisiondata(orig, self, nodeorrev, *args, **kwargs): def wrap_revisiondata(orig, self, nodeorrev, *args, **kwargs):
text, sd = orig(self, nodeorrev, *args, **kwargs) text, sd = orig(self, nodeorrev, *args, **kwargs)
if getattr(self, 'sidedatanocheck', False): if getattr(self, 'sidedatanocheck', False):
return text, sd return text, sd
if self.version & 0xFFFF != 2: if self.version & 0xFFFF != 2:
return text, sd return text, sd
if nodeorrev != nullrev and nodeorrev != nullid: if nodeorrev != nullrev and nodeorrev != nullid:
if len(text) != struct.unpack('>I', sd[sidedata.SD_TEST1])[0]: cat1 = sd.get(sidedata.SD_TEST1)
if cat1 is not None and len(text) != struct.unpack('>I', cat1)[0]:
raise RuntimeError('text size mismatch') raise RuntimeError('text size mismatch')
expected = sd[sidedata.SD_TEST2] expected = sd.get(sidedata.SD_TEST2)
got = hashlib.sha256(text).digest() got = hashlib.sha256(text).digest()
if got != expected: if expected is not None and got != expected:
raise RuntimeError('sha256 mismatch') raise RuntimeError('sha256 mismatch')
return text, sd return text, sd
def wrapgetsidedatacompanion(orig, srcrepo, dstrepo): def wrapgetsidedatacompanion(orig, srcrepo, dstrepo):
sidedatacompanion = orig(srcrepo, dstrepo) sidedatacompanion = orig(srcrepo, dstrepo)
addedreqs = dstrepo.requirements - srcrepo.requirements addedreqs = dstrepo.requirements - srcrepo.requirements
if requirements.SIDEDATA_REQUIREMENT in addedreqs: if requirements.SIDEDATA_REQUIREMENT in addedreqs:
def extsetup(ui): def extsetup(ui):
extensions.wrapfunction(revlog.revlog, 'addrevision', wrapaddrevision) extensions.wrapfunction(revlog.revlog, 'addrevision', wrapaddrevision)
extensions.wrapfunction(revlog.revlog, '_revisiondata', wrap_revisiondata) extensions.wrapfunction(revlog.revlog, '_revisiondata', wrap_revisiondata)
extensions.wrapfunction( extensions.wrapfunction(
upgrade_engine, 'getsidedatacompanion', wrapgetsidedatacompanion upgrade_engine, 'getsidedatacompanion', wrapgetsidedatacompanion
) )
def reposetup(ui, repo):
# We don't register sidedata computers because we don't care within these
# tests
repo.register_wanted_sidedata(sidedata.SD_TEST1)
repo.register_wanted_sidedata(sidedata.SD_TEST2)