( )⚙ D10343 cg4: introduce protocol flag to signify the presence of sidedata

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10343

cg4: introduce protocol flag to signify the presence of sidedata
ClosedPublic

Authored by Alphare on Apr 9 2021, 4:49 AM.

Details

Reviewers
marmoute
indygreg
Group Reviewers
hg-reviewers
Commits
rHG119790e1c67c: cg4: introduce protocol flag to signify the presence of sidedata
Summary

We need a way of signaling whether the current revision has sidedata or not,
and re-using the revision flags would waste potential revlog flags and mix two
normally independent layers.

In this change, we add a single byte at the start of the ch4 delta header to
set potential protocol flags. We also reclaim the revlog flag for sidedata,
since it is no longer used, in its place now lives the (also experimental)
copytracing flag.

When generating deltas, apply the CG_FLAG_SIDEDATA flag if there is sidedata.
When applying the deltas, if said flag is present, the next chunk contains the
sidedata.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

Alphare created this revision.Apr 9 2021, 4:49 AM
marmoute requested changes to this revision.Apr 9 2021, 6:09 AM
marmoute added a subscriber: marmoute.
marmoute added inline comments.
mercurial/changegroup.py
659–660

The revlog flag is no longer useful since the sidedata are now stored separately. Are we still using it in the revlog index ?

We should use another flag/variable defined for changegroup only for this purpose.

mercurial/helptext/internals/changegroups.txt
156–158

Maybe add a not about the fact it does not affect changegroup handling and will be relevant for other Mercurial operation ?

mercurial/revlogutils/constants.py
88–89

So, this should not be in relogutils.constants ;-) (see previous message)

This revision now requires changes to proceed.Apr 9 2021, 6:09 AM
Alphare retitled this revision from cg4: use revision flag to signify the presence of sidedata to cg4: introduce protocol flag to signify the presence of sidedata.Apr 10 2021, 3:14 PM
Alphare edited the summary of this revision. (Show Details)
Alphare updated this revision to Diff 26778.
Alphare marked 3 inline comments as done.Apr 10 2021, 3:20 PM
Alphare planned changes to this revision.
Alphare updated this revision to Diff 27032.Apr 19 2021, 6:09 AM
Alphare updated this revision to Diff 27099.Apr 20 2021, 9:38 AM
marmoute accepted this revision.Apr 22 2021, 1:45 PM
marmoute added inline comments.
hgext/remotefilelog/remotefilelog.py
339–340

nits: The rest of the patchs seems to imply they are used. Am I missing something ?

marmoute requested changes to this revision.May 2 2021, 8:49 AM

This patch is currently failing tests/test-upgrade-repo.t

This revision now requires changes to proceed.May 2 2021, 8:49 AM
baymax updated this revision to Diff 27300.May 3 2021, 6:06 AM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

marmoute accepted this revision.May 3 2021, 7:50 AM

Ci fixed, thanks.

baymax updated this revision to Diff 27450.May 3 2021, 9:01 PM

✅ refresh by Heptapod after a successful CI run (🐙 💚)

Alphare added a commit: rHG119790e1c67c: cg4: introduce protocol flag to signify the presence of sidedata.May 5 2021, 6:40 PM
This revision was not accepted when it landed; it landed in state Needs Review.
Closed by commit rHG119790e1c67c: cg4: introduce protocol flag to signify the presence of sidedata (authored by Alphare). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 27032

hgext/remotefilelog/remotefilelog.py

linknode=self.linknode(node), linknode=self.linknode(node),
basenode=basenode, basenode=basenode,
flags=self.flags(node), flags=self.flags(node),
baserevisionsize=None, baserevisionsize=None,
revision=revision, revision=revision,
delta=delta, delta=delta,
# Sidedata is not supported yet # Sidedata is not supported yet
sidedata=None, sidedata=None,
# Protocol flags are not used yet
protocol_flags=0,
marmouteUnsubmitted
Not Done

nits: The rest of the patchs seems to imply they are used. Am I missing something ?

marmoute: nits: The rest of the patchs seems to imply they are used. Am I missing something ?
) )
def revdiff(self, node1, node2): def revdiff(self, node1, node2):
return mdiff.textdiff(self.rawdata(node1), self.rawdata(node2)) return mdiff.textdiff(self.rawdata(node1), self.rawdata(node2))
def lookup(self, node): def lookup(self, node):
if len(node) == 40: if len(node) == 40:
node = bin(node) node = bin(node)

hgext/sqlitestore.py

p1node = attr.ib() p1node = attr.ib()
p2node = attr.ib() p2node = attr.ib()
basenode = attr.ib() basenode = attr.ib()
flags = attr.ib() flags = attr.ib()
baserevisionsize = attr.ib() baserevisionsize = attr.ib()
revision = attr.ib() revision = attr.ib()
delta = attr.ib() delta = attr.ib()
sidedata = attr.ib() sidedata = attr.ib()
protocol_flags = attr.ib()
linknode = attr.ib(default=None) linknode = attr.ib(default=None)
@interfaceutil.implementer(repository.iverifyproblem) @interfaceutil.implementer(repository.iverifyproblem)
@attr.s(frozen=True) @attr.s(frozen=True)
class sqliteproblem(object): class sqliteproblem(object):
warning = attr.ib(default=None) warning = attr.ib(default=None)
error = attr.ib(default=None) error = attr.ib(default=None)

mercurial/changegroup.py

pycompat, pycompat,
requirements, requirements,
scmutil, scmutil,
util, util,
) )
from .interfaces import repository from .interfaces import repository
from .revlogutils import sidedata as sidedatamod from .revlogutils import sidedata as sidedatamod
from .utils import storageutil
_CHANGEGROUPV1_DELTA_HEADER = struct.Struct(b"20s20s20s20s") _CHANGEGROUPV1_DELTA_HEADER = struct.Struct(b"20s20s20s20s")
_CHANGEGROUPV2_DELTA_HEADER = struct.Struct(b"20s20s20s20s20s") _CHANGEGROUPV2_DELTA_HEADER = struct.Struct(b"20s20s20s20s20s")
_CHANGEGROUPV3_DELTA_HEADER = struct.Struct(b">20s20s20s20s20sH") _CHANGEGROUPV3_DELTA_HEADER = struct.Struct(b">20s20s20s20s20sH")
_CHANGEGROUPV4_DELTA_HEADER = struct.Struct(b">B20s20s20s20s20sH")
LFS_REQUIREMENT = b'lfs' LFS_REQUIREMENT = b'lfs'
readexactly = util.readexactly readexactly = util.readexactly
def getchunk(stream): def getchunk(stream):
"""return the next chunk from stream as a string""" """return the next chunk from stream as a string"""
def _deltaheader(self, headertuple, prevnode): def _deltaheader(self, headertuple, prevnode):
node, p1, p2, cs = headertuple node, p1, p2, cs = headertuple
if prevnode is None: if prevnode is None:
deltabase = p1 deltabase = p1
else: else:
deltabase = prevnode deltabase = prevnode
flags = 0 flags = 0
return node, p1, p2, deltabase, cs, flags protocol_flags = 0
return node, p1, p2, deltabase, cs, flags, protocol_flags
def deltachunk(self, prevnode): def deltachunk(self, prevnode):
l = self._chunklength() l = self._chunklength()
if not l: if not l:
return {} return {}
headerdata = readexactly(self._stream, self.deltaheadersize) headerdata = readexactly(self._stream, self.deltaheadersize)
header = self.deltaheader.unpack(headerdata) header = self.deltaheader.unpack(headerdata)
delta = readexactly(self._stream, l - self.deltaheadersize) delta = readexactly(self._stream, l - self.deltaheadersize)
node, p1, p2, deltabase, cs, flags = self._deltaheader(header, prevnode) header = self._deltaheader(header, prevnode)
# cg4 forward-compat node, p1, p2, deltabase, cs, flags, protocol_flags = header
sidedata = {} return node, p1, p2, cs, deltabase, delta, flags, protocol_flags
return (node, p1, p2, cs, deltabase, delta, flags, sidedata)
def getchunks(self): def getchunks(self):
"""returns all the chunks contains in the bundle """returns all the chunks contains in the bundle
Used when you need to forward the binary stream to a file or another Used when you need to forward the binary stream to a file or another
network API. To do so, it parse the changegroup data, otherwise it will network API. To do so, it parse the changegroup data, otherwise it will
block in case of sshrepo because it don't know the end of the stream. block in case of sshrepo because it don't know the end of the stream.
""" """
deltaheader = _CHANGEGROUPV2_DELTA_HEADER deltaheader = _CHANGEGROUPV2_DELTA_HEADER
deltaheadersize = deltaheader.size deltaheadersize = deltaheader.size
version = b'02' version = b'02'
def _deltaheader(self, headertuple, prevnode): def _deltaheader(self, headertuple, prevnode):
node, p1, p2, deltabase, cs = headertuple node, p1, p2, deltabase, cs = headertuple
flags = 0 flags = 0
return node, p1, p2, deltabase, cs, flags protocol_flags = 0
return node, p1, p2, deltabase, cs, flags, protocol_flags
class cg3unpacker(cg2unpacker): class cg3unpacker(cg2unpacker):
"""Unpacker for cg3 streams. """Unpacker for cg3 streams.
cg3 streams add support for exchanging treemanifests and revlog cg3 streams add support for exchanging treemanifests and revlog
flags. It adds the revlog flags to the delta header and an empty chunk flags. It adds the revlog flags to the delta header and an empty chunk
separating manifests and files. separating manifests and files.
""" """
deltaheader = _CHANGEGROUPV3_DELTA_HEADER deltaheader = _CHANGEGROUPV3_DELTA_HEADER
deltaheadersize = deltaheader.size deltaheadersize = deltaheader.size
version = b'03' version = b'03'
_grouplistcount = 2 # One list of manifests and one list of files _grouplistcount = 2 # One list of manifests and one list of files
def _deltaheader(self, headertuple, prevnode): def _deltaheader(self, headertuple, prevnode):
node, p1, p2, deltabase, cs, flags = headertuple node, p1, p2, deltabase, cs, flags = headertuple
return node, p1, p2, deltabase, cs, flags protocol_flags = 0
return node, p1, p2, deltabase, cs, flags, protocol_flags
def _unpackmanifests(self, repo, revmap, trp, prog, addrevisioncb=None): def _unpackmanifests(self, repo, revmap, trp, prog, addrevisioncb=None):
super(cg3unpacker, self)._unpackmanifests( super(cg3unpacker, self)._unpackmanifests(
repo, revmap, trp, prog, addrevisioncb=addrevisioncb repo, revmap, trp, prog, addrevisioncb=addrevisioncb
) )
for chunkdata in iter(self.filelogheader, {}): for chunkdata in iter(self.filelogheader, {}):
# If we get here, there are directory manifests in the changegroup # If we get here, there are directory manifests in the changegroup
d = chunkdata[b"filename"] d = chunkdata[b"filename"]
repo.ui.debug(b"adding %s revisions\n" % d) repo.ui.debug(b"adding %s revisions\n" % d)
deltas = self.deltaiter() deltas = self.deltaiter()
if not repo.manifestlog.getstorage(d).addgroup( if not repo.manifestlog.getstorage(d).addgroup(
deltas, revmap, trp, addrevisioncb=addrevisioncb deltas, revmap, trp, addrevisioncb=addrevisioncb
): ):
raise error.Abort(_(b"received dir revlog group is empty")) raise error.Abort(_(b"received dir revlog group is empty"))
class cg4unpacker(cg3unpacker): class cg4unpacker(cg3unpacker):
"""Unpacker for cg4 streams. """Unpacker for cg4 streams.
cg4 streams add support for exchanging sidedata. cg4 streams add support for exchanging sidedata.
""" """
deltaheader = _CHANGEGROUPV4_DELTA_HEADER
deltaheadersize = deltaheader.size
version = b'04' version = b'04'
def _deltaheader(self, headertuple, prevnode):
protocol_flags, node, p1, p2, deltabase, cs, flags = headertuple
return node, p1, p2, deltabase, cs, flags, protocol_flags
def deltachunk(self, prevnode): def deltachunk(self, prevnode):
res = super(cg4unpacker, self).deltachunk(prevnode) res = super(cg4unpacker, self).deltachunk(prevnode)
if not res: if not res:
return res return res
(node, p1, p2, cs, deltabase, delta, flags, _sidedata) = res (node, p1, p2, cs, deltabase, delta, flags, protocol_flags) = res
sidedata_raw = getchunk(self._stream)
sidedata = {} sidedata = {}
if len(sidedata_raw) > 0: if protocol_flags & storageutil.CG_FLAG_SIDEDATA:
sidedata_raw = getchunk(self._stream)
marmouteUnsubmitted
Done

The revlog flag is no longer useful since the sidedata are now stored separately. Are we still using it in the revlog index ?

We should use another flag/variable defined for changegroup only for this purpose.

marmoute: The revlog flag is no longer useful since the sidedata are now stored separately. Are we still…
sidedata = sidedatamod.deserialize_sidedata(sidedata_raw) sidedata = sidedatamod.deserialize_sidedata(sidedata_raw)
return node, p1, p2, cs, deltabase, delta, flags, sidedata return node, p1, p2, cs, deltabase, delta, flags, sidedata
class headerlessfixup(object): class headerlessfixup(object):
def __init__(self, fh, h): def __init__(self, fh, h):
self._h = h self._h = h
meta = headerfn(delta) meta = headerfn(delta)
yield chunkheader(len(meta) + len(prefix) + len(data)) yield chunkheader(len(meta) + len(prefix) + len(data))
yield meta yield meta
if prefix: if prefix:
yield prefix yield prefix
yield data yield data
sidedata = delta.sidedata if delta.protocol_flags & storageutil.CG_FLAG_SIDEDATA:
if sidedata is not None:
# Need a separate chunk for sidedata to be able to differentiate # Need a separate chunk for sidedata to be able to differentiate
# "raw delta" length and sidedata length # "raw delta" length and sidedata length
sidedata = delta.sidedata
yield chunkheader(len(sidedata)) yield chunkheader(len(sidedata))
yield sidedata yield sidedata
def _sortnodesellipsis(store, nodes, cl, lookup): def _sortnodesellipsis(store, nodes, cl, lookup):
"""Sort nodes for changegroup generation.""" """Sort nodes for changegroup generation."""
# Ellipses serving mode. # Ellipses serving mode.
# #
matcher, matcher,
bundlecaps, bundlecaps,
ellipses=False, ellipses=False,
shallow=False, shallow=False,
ellipsisroots=None, ellipsisroots=None,
fullnodes=None, fullnodes=None,
remote_sidedata=None, remote_sidedata=None,
): ):
# Same header func as cg3. Sidedata is in a separate chunk from the delta to # Sidedata is in a separate chunk from the delta to differentiate
# differenciate "raw delta" and sidedata. # "raw delta" and sidedata.
builddeltaheader = lambda d: _CHANGEGROUPV3_DELTA_HEADER.pack( def builddeltaheader(d):
d.node, d.p1node, d.p2node, d.basenode, d.linknode, d.flags return _CHANGEGROUPV4_DELTA_HEADER.pack(
d.protocol_flags,
d.node,
d.p1node,
d.p2node,
d.basenode,
d.linknode,
d.flags,
) )
return cgpacker( return cgpacker(
repo, repo,
oldmatcher, oldmatcher,
matcher, matcher,
b'04', b'04',
builddeltaheader=builddeltaheader, builddeltaheader=builddeltaheader,
manifestsend=closechunk(), manifestsend=closechunk(),
return revisions, files return revisions, files
def get_sidedata_helpers(repo, remote_sd_categories, pull=False): def get_sidedata_helpers(repo, remote_sd_categories, pull=False):
# Computers for computing sidedata on-the-fly # Computers for computing sidedata on-the-fly
sd_computers = collections.defaultdict(list) sd_computers = collections.defaultdict(list)
# Computers for categories to remove from sidedata # Computers for categories to remove from sidedata
sd_removers = collections.defaultdict(list) sd_removers = collections.defaultdict(list)
to_generate = remote_sd_categories - repo._wanted_sidedata to_generate = remote_sd_categories - repo._wanted_sidedata
to_remove = repo._wanted_sidedata - remote_sd_categories to_remove = repo._wanted_sidedata - remote_sd_categories
if pull: if pull:
to_generate, to_remove = to_remove, to_generate to_generate, to_remove = to_remove, to_generate
for revlog_kind, computers in repo._sidedata_computers.items(): for revlog_kind, computers in repo._sidedata_computers.items():
for category, computer in computers.items(): for category, computer in computers.items():
if category in to_generate: if category in to_generate:
sd_computers[revlog_kind].append(computer) sd_computers[revlog_kind].append(computer)
if category in to_remove: if category in to_remove:
sd_removers[revlog_kind].append(computer) sd_removers[revlog_kind].append(computer)
sidedata_helpers = (repo, sd_computers, sd_removers) sidedata_helpers = (repo, sd_computers, sd_removers)
return sidedata_helpers return sidedata_helpers

mercurial/helptext/internals/changegroups.txt

Changegroups are representations of repository revlog data, specifically Changegroups are representations of repository revlog data, specifically
the changelog data, root/flat manifest data, treemanifest data, and the changelog data, root/flat manifest data, treemanifest data, and
filelogs. filelogs.
There are 3 versions of changegroups: ``1``, ``2``, and ``3``. From a There are 4 versions of changegroups: ``1``, ``2``, ``3`` and ``4``. From a
high-level, versions ``1`` and ``2`` are almost exactly the same, with the high-level, versions ``1`` and ``2`` are almost exactly the same, with the
only difference being an additional item in the *delta header*. Version only difference being an additional item in the *delta header*. Version
``3`` adds support for storage flags in the *delta header* and optionally ``3`` adds support for storage flags in the *delta header* and optionally
exchanging treemanifests (enabled by setting an option on the exchanging treemanifests (enabled by setting an option on the
``changegroup`` part in the bundle2). ``changegroup`` part in the bundle2). Version ``4`` adds support for exchanging
sidedata (additional revision metadata not part of the digest).
Changegroups when not exchanging treemanifests consist of 3 logical Changegroups when not exchanging treemanifests consist of 3 logical
segments:: segments::
+---------------------------------+ +---------------------------------+
| | | | | | | |
| changeset | manifest | filelogs | | changeset | manifest | filelogs |
| | | | | | | |
| (various by version) | (various) | | (various by version) | (various) |
| | | | | |
+---------------------------------------+ +---------------------------------------+
The *delta data* is a series of *delta*s that describe a diff from an existing The *delta data* is a series of *delta*s that describe a diff from an existing
entry (either that the recipient already has, or previously specified in the entry (either that the recipient already has, or previously specified in the
bundle/changegroup). bundle/changegroup).
The *delta header* is different between versions ``1``, ``2``, and The *delta header* is different between versions ``1``, ``2``, ``3`` and ``4``
``3`` of the changegroup format. of the changegroup format.
Version 1 (headerlen=80):: Version 1 (headerlen=80)::
+------------------------------------------------------+ +------------------------------------------------------+
| | | | | | | | | |
| node | p1 node | p2 node | link node | | node | p1 node | p2 node | link node |
| (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | | (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) |
| | | | | | | | | |
+------------------------------------------------------------------------------+ +------------------------------------------------------------------------------+
| | | | | | | | | | | | | |
| node | p1 node | p2 node | base node | link node | flags | | node | p1 node | p2 node | base node | link node | flags |
| (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (2 bytes) | | (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (2 bytes) |
| | | | | | | | | | | | | |
+------------------------------------------------------------------------------+ +------------------------------------------------------------------------------+
Version 4 (headerlen=103)::
+------------------------------------------------------------------------------+----------+
| | | | | | | |
| node | p1 node | p2 node | base node | link node | flags | pflags |
| (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (2 bytes) | (1 byte) |
| | | | | | | |
+------------------------------------------------------------------------------+----------+
The *delta data* consists of ``chunklen - 4 - headerlen`` bytes, which contain a The *delta data* consists of ``chunklen - 4 - headerlen`` bytes, which contain a
series of *delta*s, densely packed (no separators). These deltas describe a diff series of *delta*s, densely packed (no separators). These deltas describe a diff
from an existing entry (either that the recipient already has, or previously from an existing entry (either that the recipient already has, or previously
specified in the bundle/changegroup). The format is described more fully in specified in the bundle/changegroup). The format is described more fully in
``hg help internals.bdiff``, but briefly:: ``hg help internals.bdiff``, but briefly::
+---------------------------------------------------------------+ +---------------------------------------------------------------+
| | | | | | | | | |
metadata. May only occur on file revisions. metadata. May only occur on file revisions.
16384 16384
Ellipsis revision. Revision hash does not match data (likely due to rewritten Ellipsis revision. Revision hash does not match data (likely due to rewritten
parents). parents).
8192 8192
Externally stored. The revision fulltext contains ``key:value`` ``\n`` Externally stored. The revision fulltext contains ``key:value`` ``\n``
delimited metadata defining an object stored elsewhere. Used by the LFS delimited metadata defining an object stored elsewhere. Used by the LFS
extension. extension.
4096
Contains copy information. This revision changes files in a way that could
affect copy tracing. This does *not* affect changegroup handling, but is
relevant for other parts of Mercurial.
For historical reasons, the integer values are identical to revlog version 1 For historical reasons, the integer values are identical to revlog version 1
marmouteUnsubmitted
Done

Maybe add a not about the fact it does not affect changegroup handling and will be relevant for other Mercurial operation ?

marmoute: Maybe add a not about the fact it does not affect changegroup handling and will be relevant for…
per-revision storage flags and correspond to bits being set in this 2-byte per-revision storage flags and correspond to bits being set in this 2-byte
field. Bits were allocated starting from the most-significant bit, hence the field. Bits were allocated starting from the most-significant bit, hence the
reverse ordering and allocation of these flags. reverse ordering and allocation of these flags.
The *pflags* (protocol flags) field holds bitwise flags affecting the protocol
itself. They are first in the header since they may affect the handling of the
rest of the fields in a future version. They are defined as such:
1 indicates whether to read a chunk of sidedata (of variable length) right
after the revision flags.
Changeset Segment Changeset Segment
================= =================
The *changeset segment* consists of a single *delta group* holding The *changeset segment* consists of a single *delta group* holding
changelog data. The *empty chunk* at the end of the *delta group* denotes changelog data. The *empty chunk* at the end of the *delta group* denotes
the boundary to the *manifest segment*. the boundary to the *manifest segment*.
Manifest Segment Manifest Segment
================ ================
The *manifest segment* consists of a single *delta group* holding manifest The *manifest segment* consists of a single *delta group* holding manifest
data. If treemanifests are in use, it contains only the manifest for the data. If treemanifests are in use, it contains only the manifest for the
root directory of the repository. Otherwise, it contains the entire root directory of the repository. Otherwise, it contains the entire
manifest data. The *empty chunk* at the end of the *delta group* denotes manifest data. The *empty chunk* at the end of the *delta group* denotes
the boundary to the next segment (either the *treemanifests segment* or the the boundary to the next segment (either the *treemanifests segment* or the
*filelogs segment*, depending on version and the request options). *filelogs segment*, depending on version and the request options).
Treemanifests Segment Treemanifests Segment
--------------------- ---------------------
The *treemanifests segment* only exists in changegroup version ``3``, and The *treemanifests segment* only exists in changegroup version ``3`` and ``4``,
only if the 'treemanifest' param is part of the bundle2 changegroup part and only if the 'treemanifest' param is part of the bundle2 changegroup part
(it is not possible to use changegroup version 3 outside of bundle2). (it is not possible to use changegroup version 3 or 4 outside of bundle2).
Aside from the filenames in the *treemanifests segment* containing a Aside from the filenames in the *treemanifests segment* containing a
trailing ``/`` character, it behaves identically to the *filelogs segment* trailing ``/`` character, it behaves identically to the *filelogs segment*
(see below). The final sub-segment is followed by an *empty chunk* (logically, (see below). The final sub-segment is followed by an *empty chunk* (logically,
a sub-segment with filename size 0). This denotes the boundary to the a sub-segment with filename size 0). This denotes the boundary to the
*filelogs segment*. *filelogs segment*.
Filelogs Segment Filelogs Segment
================ ================

mercurial/interfaces/repository.py

# Repository supports being stream cloned. # Repository supports being stream cloned.
REPO_FEATURE_STREAM_CLONE = b'streamclone' REPO_FEATURE_STREAM_CLONE = b'streamclone'
# Files storage may lack data for all ancestors. # Files storage may lack data for all ancestors.
REPO_FEATURE_SHALLOW_FILE_STORAGE = b'shallowfilestorage' REPO_FEATURE_SHALLOW_FILE_STORAGE = b'shallowfilestorage'
REVISION_FLAG_CENSORED = 1 << 15 REVISION_FLAG_CENSORED = 1 << 15
REVISION_FLAG_ELLIPSIS = 1 << 14 REVISION_FLAG_ELLIPSIS = 1 << 14
REVISION_FLAG_EXTSTORED = 1 << 13 REVISION_FLAG_EXTSTORED = 1 << 13
REVISION_FLAG_SIDEDATA = 1 << 12 REVISION_FLAG_HASCOPIESINFO = 1 << 12
REVISION_FLAG_HASCOPIESINFO = 1 << 11
REVISION_FLAGS_KNOWN = ( REVISION_FLAGS_KNOWN = (
REVISION_FLAG_CENSORED REVISION_FLAG_CENSORED
| REVISION_FLAG_ELLIPSIS | REVISION_FLAG_ELLIPSIS
| REVISION_FLAG_EXTSTORED | REVISION_FLAG_EXTSTORED
| REVISION_FLAG_SIDEDATA
| REVISION_FLAG_HASCOPIESINFO | REVISION_FLAG_HASCOPIESINFO
) )
CG_DELTAMODE_STD = b'default' CG_DELTAMODE_STD = b'default'
CG_DELTAMODE_PREV = b'previous' CG_DELTAMODE_PREV = b'previous'
CG_DELTAMODE_FULL = b'fulltext' CG_DELTAMODE_FULL = b'fulltext'
CG_DELTAMODE_P1 = b'p1' CG_DELTAMODE_P1 = b'p1'
Stored in the bdiff delta format. Stored in the bdiff delta format.
""" """
) )
sidedata = interfaceutil.Attribute( sidedata = interfaceutil.Attribute(
"""Raw sidedata bytes for the given revision.""" """Raw sidedata bytes for the given revision."""
) )
protocol_flags = interfaceutil.Attribute(
"""Single byte of integer flags that can influence the protocol.
This is a bitwise composition of the ``storageutil.CG_FLAG*`` constants.
"""
)
class ifilerevisionssequence(interfaceutil.Interface): class ifilerevisionssequence(interfaceutil.Interface):
"""Contains index data for all revisions of a file. """Contains index data for all revisions of a file.
Types implementing this behave like lists of tuples. The index Types implementing this behave like lists of tuples. The index
in the list corresponds to the revision number. The values contain in the list corresponds to the revision number. The values contain
index metadata. index metadata.

mercurial/revlog.py

from .revlogutils.flagutil import ( from .revlogutils.flagutil import (
REVIDX_DEFAULT_FLAGS, REVIDX_DEFAULT_FLAGS,
REVIDX_ELLIPSIS, REVIDX_ELLIPSIS,
REVIDX_EXTSTORED, REVIDX_EXTSTORED,
REVIDX_FLAGS_ORDER, REVIDX_FLAGS_ORDER,
REVIDX_HASCOPIESINFO, REVIDX_HASCOPIESINFO,
REVIDX_ISCENSORED, REVIDX_ISCENSORED,
REVIDX_RAWTEXT_CHANGING_FLAGS, REVIDX_RAWTEXT_CHANGING_FLAGS,
REVIDX_SIDEDATA,
) )
from .thirdparty import attr from .thirdparty import attr
from . import ( from . import (
ancestor, ancestor,
dagop, dagop,
error, error,
mdiff, mdiff,
policy, policy,
FLAG_GENERALDELTA FLAG_GENERALDELTA
REVLOG_DEFAULT_FLAGS REVLOG_DEFAULT_FLAGS
REVLOG_DEFAULT_FORMAT REVLOG_DEFAULT_FORMAT
REVLOG_DEFAULT_VERSION REVLOG_DEFAULT_VERSION
REVLOGV1_FLAGS REVLOGV1_FLAGS
REVLOGV2_FLAGS REVLOGV2_FLAGS
REVIDX_ISCENSORED REVIDX_ISCENSORED
REVIDX_ELLIPSIS REVIDX_ELLIPSIS
REVIDX_SIDEDATA
REVIDX_HASCOPIESINFO REVIDX_HASCOPIESINFO
REVIDX_EXTSTORED REVIDX_EXTSTORED
REVIDX_DEFAULT_FLAGS REVIDX_DEFAULT_FLAGS
REVIDX_FLAGS_ORDER REVIDX_FLAGS_ORDER
REVIDX_RAWTEXT_CHANGING_FLAGS REVIDX_RAWTEXT_CHANGING_FLAGS
parsers = policy.importmod('parsers') parsers = policy.importmod('parsers')
rustancestor = policy.importrust('ancestor') rustancestor = policy.importrust('ancestor')
p1node = attr.ib() p1node = attr.ib()
p2node = attr.ib() p2node = attr.ib()
basenode = attr.ib() basenode = attr.ib()
flags = attr.ib() flags = attr.ib()
baserevisionsize = attr.ib() baserevisionsize = attr.ib()
revision = attr.ib() revision = attr.ib()
delta = attr.ib() delta = attr.ib()
sidedata = attr.ib() sidedata = attr.ib()
protocol_flags = attr.ib()
linknode = attr.ib(default=None) linknode = attr.ib(default=None)
@interfaceutil.implementer(repository.iverifyproblem) @interfaceutil.implementer(repository.iverifyproblem)
@attr.s(frozen=True) @attr.s(frozen=True)
class revlogproblem(object): class revlogproblem(object):
warning = attr.ib(default=None) warning = attr.ib(default=None)
error = attr.ib(default=None) error = attr.ib(default=None)

mercurial/revlogutils/constants.py

# wire protocol and are even exposed in parts of the storage APIs. # wire protocol and are even exposed in parts of the storage APIs.
# revision has censor metadata, must be verified # revision has censor metadata, must be verified
REVIDX_ISCENSORED = repository.REVISION_FLAG_CENSORED REVIDX_ISCENSORED = repository.REVISION_FLAG_CENSORED
# revision hash does not match data (narrowhg) # revision hash does not match data (narrowhg)
REVIDX_ELLIPSIS = repository.REVISION_FLAG_ELLIPSIS REVIDX_ELLIPSIS = repository.REVISION_FLAG_ELLIPSIS
# revision data is stored externally # revision data is stored externally
REVIDX_EXTSTORED = repository.REVISION_FLAG_EXTSTORED REVIDX_EXTSTORED = repository.REVISION_FLAG_EXTSTORED
# revision data contains extra metadata not part of the official digest
REVIDX_SIDEDATA = repository.REVISION_FLAG_SIDEDATA
# revision changes files in a way that could affect copy tracing. # revision changes files in a way that could affect copy tracing.
REVIDX_HASCOPIESINFO = repository.REVISION_FLAG_HASCOPIESINFO REVIDX_HASCOPIESINFO = repository.REVISION_FLAG_HASCOPIESINFO
marmouteUnsubmitted
Done

So, this should not be in relogutils.constants ;-) (see previous message)

marmoute: So, this should not be in relogutils.constants ;-) (see previous message)
REVIDX_DEFAULT_FLAGS = 0 REVIDX_DEFAULT_FLAGS = 0
# stable order in which flags need to be processed and their processors applied # stable order in which flags need to be processed and their processors applied
REVIDX_FLAGS_ORDER = [ REVIDX_FLAGS_ORDER = [
REVIDX_ISCENSORED, REVIDX_ISCENSORED,
REVIDX_ELLIPSIS, REVIDX_ELLIPSIS,
REVIDX_EXTSTORED, REVIDX_EXTSTORED,
REVIDX_SIDEDATA,
REVIDX_HASCOPIESINFO, REVIDX_HASCOPIESINFO,
] ]
# bitmark for flags that could cause rawdata content change # bitmark for flags that could cause rawdata content change
REVIDX_RAWTEXT_CHANGING_FLAGS = ( REVIDX_RAWTEXT_CHANGING_FLAGS = REVIDX_ISCENSORED | REVIDX_EXTSTORED
REVIDX_ISCENSORED | REVIDX_EXTSTORED | REVIDX_SIDEDATA
)
SPARSE_REVLOG_MAX_CHAIN_LENGTH = 1000 SPARSE_REVLOG_MAX_CHAIN_LENGTH = 1000

mercurial/revlogutils/flagutil.py

from .constants import ( from .constants import (
REVIDX_DEFAULT_FLAGS, REVIDX_DEFAULT_FLAGS,
REVIDX_ELLIPSIS, REVIDX_ELLIPSIS,
REVIDX_EXTSTORED, REVIDX_EXTSTORED,
REVIDX_FLAGS_ORDER, REVIDX_FLAGS_ORDER,
REVIDX_HASCOPIESINFO, REVIDX_HASCOPIESINFO,
REVIDX_ISCENSORED, REVIDX_ISCENSORED,
REVIDX_RAWTEXT_CHANGING_FLAGS, REVIDX_RAWTEXT_CHANGING_FLAGS,
REVIDX_SIDEDATA,
) )
from .. import error, util from .. import error, util
# blanked usage of all the name to prevent pyflakes constraints # blanked usage of all the name to prevent pyflakes constraints
# We need these name available in the module for extensions. # We need these name available in the module for extensions.
REVIDX_ISCENSORED REVIDX_ISCENSORED
REVIDX_ELLIPSIS REVIDX_ELLIPSIS
REVIDX_EXTSTORED REVIDX_EXTSTORED
REVIDX_SIDEDATA
REVIDX_HASCOPIESINFO, REVIDX_HASCOPIESINFO,
REVIDX_DEFAULT_FLAGS REVIDX_DEFAULT_FLAGS
REVIDX_FLAGS_ORDER REVIDX_FLAGS_ORDER
REVIDX_RAWTEXT_CHANGING_FLAGS REVIDX_RAWTEXT_CHANGING_FLAGS
REVIDX_KNOWN_FLAGS = util.bitsfrom(REVIDX_FLAGS_ORDER) REVIDX_KNOWN_FLAGS = util.bitsfrom(REVIDX_FLAGS_ORDER)
# Store flag processors (cf. 'addflagprocessor()' to register) # Store flag processors (cf. 'addflagprocessor()' to register)

mercurial/utils/storageutil.py

pycompat, pycompat,
) )
from ..interfaces import repository from ..interfaces import repository
from ..revlogutils import sidedata as sidedatamod from ..revlogutils import sidedata as sidedatamod
from ..utils import hashutil from ..utils import hashutil
_nullhash = hashutil.sha1(nullid) _nullhash = hashutil.sha1(nullid)
# revision data contains extra metadata not part of the official digest
# Only used in changegroup >= v4.
CG_FLAG_SIDEDATA = 1
def hashrevisionsha1(text, p1, p2): def hashrevisionsha1(text, p1, p2):
"""Compute the SHA-1 for revision data and its parents. """Compute the SHA-1 for revision data and its parents.
This hash combines both the current file contents and its history This hash combines both the current file contents and its history
in a manner that makes it easy to distinguish nodes with the same in a manner that makes it easy to distinguish nodes with the same
content in the revision graph. content in the revision graph.
""" """
delta = revdifffn(baserev, rev) delta = revdifffn(baserev, rev)
else: else:
delta = mdiff.textdiff( delta = mdiff.textdiff(
store.rawdata(baserev), store.rawdata(rev) store.rawdata(baserev), store.rawdata(rev)
) )
available.add(rev) available.add(rev)
sidedata = None serialized_sidedata = None
if sidedata_helpers: if sidedata_helpers:
sidedata = store.sidedata(rev) sidedata = store.sidedata(rev)
sidedata = run_sidedata_helpers( sidedata = run_sidedata_helpers(
store=store, store=store,
sidedata_helpers=sidedata_helpers, sidedata_helpers=sidedata_helpers,
sidedata=sidedata, sidedata=sidedata,
rev=rev, rev=rev,
) )
sidedata = sidedatamod.serialize_sidedata(sidedata) if sidedata:
serialized_sidedata = sidedatamod.serialize_sidedata(sidedata)
flags = flagsfn(rev) if flagsfn else 0
protocol_flags = 0
if serialized_sidedata:
# Advertise that sidedata exists to the other side
protocol_flags |= CG_FLAG_SIDEDATA
yield resultcls( yield resultcls(
node=node, node=node,
p1node=fnode(p1rev), p1node=fnode(p1rev),
p2node=fnode(p2rev), p2node=fnode(p2rev),
basenode=fnode(baserev), basenode=fnode(baserev),
flags=flagsfn(rev) if flagsfn else 0, flags=flags,
baserevisionsize=baserevisionsize, baserevisionsize=baserevisionsize,
revision=revision, revision=revision,
delta=delta, delta=delta,
sidedata=sidedata, sidedata=serialized_sidedata,
protocol_flags=protocol_flags,
) )
prevrev = rev prevrev = rev
def run_sidedata_helpers(store, sidedata_helpers, sidedata, rev): def run_sidedata_helpers(store, sidedata_helpers, sidedata, rev):
"""Returns the sidedata for the given revision after running through """Returns the sidedata for the given revision after running through
the given helpers. the given helpers.

tests/test-check-interfaces.py

p2node=b'', p2node=b'',
basenode=b'', basenode=b'',
linknode=b'', linknode=b'',
flags=b'', flags=b'',
baserevisionsize=None, baserevisionsize=None,
revision=b'', revision=b'',
sidedata=b'', sidedata=b'',
delta=None, delta=None,
protocol_flags=b'',
) )
checkzobject(rd) checkzobject(rd)
ziverify.verifyClass(repository.iverifyproblem, revlog.revlogproblem) ziverify.verifyClass(repository.iverifyproblem, revlog.revlogproblem)
checkzobject(revlog.revlogproblem()) checkzobject(revlog.revlogproblem())
main() main()

tests/test-help.t

$ hg help internals.changegroups $ hg help internals.changegroups
Changegroups Changegroups
"""""""""""" """"""""""""
Changegroups are representations of repository revlog data, specifically Changegroups are representations of repository revlog data, specifically
the changelog data, root/flat manifest data, treemanifest data, and the changelog data, root/flat manifest data, treemanifest data, and
filelogs. filelogs.
There are 3 versions of changegroups: "1", "2", and "3". From a high- There are 4 versions of changegroups: "1", "2", "3" and "4". From a high-
level, versions "1" and "2" are almost exactly the same, with the only level, versions "1" and "2" are almost exactly the same, with the only
difference being an additional item in the *delta header*. Version "3" difference being an additional item in the *delta header*. Version "3"
adds support for storage flags in the *delta header* and optionally adds support for storage flags in the *delta header* and optionally
exchanging treemanifests (enabled by setting an option on the exchanging treemanifests (enabled by setting an option on the
"changegroup" part in the bundle2). "changegroup" part in the bundle2). Version "4" adds support for
exchanging sidedata (additional revision metadata not part of the digest).
Changegroups when not exchanging treemanifests consist of 3 logical Changegroups when not exchanging treemanifests consist of 3 logical
segments: segments:
+---------------------------------+ +---------------------------------+
| | | | | | | |
| changeset | manifest | filelogs | | changeset | manifest | filelogs |
| | | | | | | |
| (various by version) | (various) | | (various by version) | (various) |
| | | | | |
+---------------------------------------+ +---------------------------------------+
The *delta data* is a series of *delta*s that describe a diff from an The *delta data* is a series of *delta*s that describe a diff from an
existing entry (either that the recipient already has, or previously existing entry (either that the recipient already has, or previously
specified in the bundle/changegroup). specified in the bundle/changegroup).
The *delta header* is different between versions "1", "2", and "3" of the The *delta header* is different between versions "1", "2", "3" and "4" of
changegroup format. the changegroup format.
Version 1 (headerlen=80): Version 1 (headerlen=80):
+------------------------------------------------------+ +------------------------------------------------------+
| | | | | | | | | |
| node | p1 node | p2 node | link node | | node | p1 node | p2 node | link node |
| (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | | (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) |
| | | | | | | | | |
+------------------------------------------------------------------------------+ +------------------------------------------------------------------------------+
| | | | | | | | | | | | | |
| node | p1 node | p2 node | base node | link node | flags | | node | p1 node | p2 node | base node | link node | flags |
| (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (2 bytes) | | (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (2 bytes) |
| | | | | | | | | | | | | |
+------------------------------------------------------------------------------+ +------------------------------------------------------------------------------+
Version 4 (headerlen=103):
+------------------------------------------------------------------------------+----------+
| | | | | | | |
| node | p1 node | p2 node | base node | link node | flags | pflags |
| (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (2 bytes) | (1 byte) |
| | | | | | | |
+------------------------------------------------------------------------------+----------+
The *delta data* consists of "chunklen - 4 - headerlen" bytes, which The *delta data* consists of "chunklen - 4 - headerlen" bytes, which
contain a series of *delta*s, densely packed (no separators). These deltas contain a series of *delta*s, densely packed (no separators). These deltas
describe a diff from an existing entry (either that the recipient already describe a diff from an existing entry (either that the recipient already
has, or previously specified in the bundle/changegroup). The format is has, or previously specified in the bundle/changegroup). The format is
described more fully in "hg help internals.bdiff", but briefly: described more fully in "hg help internals.bdiff", but briefly:
+---------------------------------------------------------------+ +---------------------------------------------------------------+
| | | | | | | | | |
Ellipsis revision. Revision hash does not match data (likely due to Ellipsis revision. Revision hash does not match data (likely due to
rewritten parents). rewritten parents).
8192 8192
Externally stored. The revision fulltext contains "key:value" "\n" Externally stored. The revision fulltext contains "key:value" "\n"
delimited metadata defining an object stored elsewhere. Used by the LFS delimited metadata defining an object stored elsewhere. Used by the LFS
extension. extension.
4096
Contains copy information. This revision changes files in a way that
could affect copy tracing. This does *not* affect changegroup handling,
but is relevant for other parts of Mercurial.
For historical reasons, the integer values are identical to revlog version For historical reasons, the integer values are identical to revlog version
1 per-revision storage flags and correspond to bits being set in this 1 per-revision storage flags and correspond to bits being set in this
2-byte field. Bits were allocated starting from the most-significant bit, 2-byte field. Bits were allocated starting from the most-significant bit,
hence the reverse ordering and allocation of these flags. hence the reverse ordering and allocation of these flags.
The *pflags* (protocol flags) field holds bitwise flags affecting the
protocol itself. They are first in the header since they may affect the
handling of the rest of the fields in a future version. They are defined
as such:
1 indicates whether to read a chunk of sidedata (of variable length) right
after the revision flags.
Changeset Segment Changeset Segment
================= =================
The *changeset segment* consists of a single *delta group* holding The *changeset segment* consists of a single *delta group* holding
changelog data. The *empty chunk* at the end of the *delta group* denotes changelog data. The *empty chunk* at the end of the *delta group* denotes
the boundary to the *manifest segment*. the boundary to the *manifest segment*.
Manifest Segment Manifest Segment
================ ================
The *manifest segment* consists of a single *delta group* holding manifest The *manifest segment* consists of a single *delta group* holding manifest
data. If treemanifests are in use, it contains only the manifest for the data. If treemanifests are in use, it contains only the manifest for the
root directory of the repository. Otherwise, it contains the entire root directory of the repository. Otherwise, it contains the entire
manifest data. The *empty chunk* at the end of the *delta group* denotes manifest data. The *empty chunk* at the end of the *delta group* denotes
the boundary to the next segment (either the *treemanifests segment* or the boundary to the next segment (either the *treemanifests segment* or
the *filelogs segment*, depending on version and the request options). the *filelogs segment*, depending on version and the request options).
Treemanifests Segment Treemanifests Segment
--------------------- ---------------------
The *treemanifests segment* only exists in changegroup version "3", and The *treemanifests segment* only exists in changegroup version "3" and
only if the 'treemanifest' param is part of the bundle2 changegroup part "4", and only if the 'treemanifest' param is part of the bundle2
(it is not possible to use changegroup version 3 outside of bundle2). changegroup part (it is not possible to use changegroup version 3 or 4
Aside from the filenames in the *treemanifests segment* containing a outside of bundle2). Aside from the filenames in the *treemanifests
trailing "/" character, it behaves identically to the *filelogs segment* segment* containing a trailing "/" character, it behaves identically to
(see below). The final sub-segment is followed by an *empty chunk* the *filelogs segment* (see below). The final sub-segment is followed by
(logically, a sub-segment with filename size 0). This denotes the boundary an *empty chunk* (logically, a sub-segment with filename size 0). This
to the *filelogs segment*. denotes the boundary to the *filelogs segment*.
Filelogs Segment Filelogs Segment
================ ================
The *filelogs segment* consists of multiple sub-segments, each The *filelogs segment* consists of multiple sub-segments, each
corresponding to an individual file whose data is being described: corresponding to an individual file whose data is being described:
+--------------------------------------------------+ +--------------------------------------------------+
<div id="doc"> <div id="doc">
<h1>Changegroups</h1> <h1>Changegroups</h1>
<p> <p>
Changegroups are representations of repository revlog data, specifically Changegroups are representations of repository revlog data, specifically
the changelog data, root/flat manifest data, treemanifest data, and the changelog data, root/flat manifest data, treemanifest data, and
filelogs. filelogs.
</p> </p>
<p> <p>
There are 3 versions of changegroups: &quot;1&quot;, &quot;2&quot;, and &quot;3&quot;. From a There are 4 versions of changegroups: &quot;1&quot;, &quot;2&quot;, &quot;3&quot; and &quot;4&quot;. From a
high-level, versions &quot;1&quot; and &quot;2&quot; are almost exactly the same, with the high-level, versions &quot;1&quot; and &quot;2&quot; are almost exactly the same, with the
only difference being an additional item in the *delta header*. Version only difference being an additional item in the *delta header*. Version
&quot;3&quot; adds support for storage flags in the *delta header* and optionally &quot;3&quot; adds support for storage flags in the *delta header* and optionally
exchanging treemanifests (enabled by setting an option on the exchanging treemanifests (enabled by setting an option on the
&quot;changegroup&quot; part in the bundle2). &quot;changegroup&quot; part in the bundle2). Version &quot;4&quot; adds support for exchanging
sidedata (additional revision metadata not part of the digest).
</p> </p>
<p> <p>
Changegroups when not exchanging treemanifests consist of 3 logical Changegroups when not exchanging treemanifests consist of 3 logical
segments: segments:
</p> </p>
<pre> <pre>
+---------------------------------+ +---------------------------------+
| | | | | | | |
+---------------------------------------+ +---------------------------------------+
</pre> </pre>
<p> <p>
The *delta data* is a series of *delta*s that describe a diff from an existing The *delta data* is a series of *delta*s that describe a diff from an existing
entry (either that the recipient already has, or previously specified in the entry (either that the recipient already has, or previously specified in the
bundle/changegroup). bundle/changegroup).
</p> </p>
<p> <p>
The *delta header* is different between versions &quot;1&quot;, &quot;2&quot;, and The *delta header* is different between versions &quot;1&quot;, &quot;2&quot;, &quot;3&quot; and &quot;4&quot;
&quot;3&quot; of the changegroup format. of the changegroup format.
</p> </p>
<p> <p>
Version 1 (headerlen=80): Version 1 (headerlen=80):
</p> </p>
<pre> <pre>
+------------------------------------------------------+ +------------------------------------------------------+
| | | | | | | | | |
| node | p1 node | p2 node | link node | | node | p1 node | p2 node | link node |
+------------------------------------------------------------------------------+ +------------------------------------------------------------------------------+
| | | | | | | | | | | | | |
| node | p1 node | p2 node | base node | link node | flags | | node | p1 node | p2 node | base node | link node | flags |
| (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (2 bytes) | | (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (2 bytes) |
| | | | | | | | | | | | | |
+------------------------------------------------------------------------------+ +------------------------------------------------------------------------------+
</pre> </pre>
<p> <p>
Version 4 (headerlen=103):
</p>
<pre>
+------------------------------------------------------------------------------+----------+
| | | | | | | |
| node | p1 node | p2 node | base node | link node | flags | pflags |
| (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (20 bytes) | (2 bytes) | (1 byte) |
| | | | | | | |
+------------------------------------------------------------------------------+----------+
</pre>
<p>
The *delta data* consists of &quot;chunklen - 4 - headerlen&quot; bytes, which contain a The *delta data* consists of &quot;chunklen - 4 - headerlen&quot; bytes, which contain a
series of *delta*s, densely packed (no separators). These deltas describe a diff series of *delta*s, densely packed (no separators). These deltas describe a diff
from an existing entry (either that the recipient already has, or previously from an existing entry (either that the recipient already has, or previously
specified in the bundle/changegroup). The format is described more fully in specified in the bundle/changegroup). The format is described more fully in
&quot;hg help internals.bdiff&quot;, but briefly: &quot;hg help internals.bdiff&quot;, but briefly:
</p> </p>
<pre> <pre>
+---------------------------------------------------------------+ +---------------------------------------------------------------+
</p> </p>
<dl> <dl>
<dt>32768 <dt>32768
<dd>Censored revision. The revision's fulltext has been replaced by censor metadata. May only occur on file revisions. <dd>Censored revision. The revision's fulltext has been replaced by censor metadata. May only occur on file revisions.
<dt>16384 <dt>16384
<dd>Ellipsis revision. Revision hash does not match data (likely due to rewritten parents). <dd>Ellipsis revision. Revision hash does not match data (likely due to rewritten parents).
<dt>8192 <dt>8192
<dd>Externally stored. The revision fulltext contains &quot;key:value&quot; &quot;\n&quot; delimited metadata defining an object stored elsewhere. Used by the LFS extension. <dd>Externally stored. The revision fulltext contains &quot;key:value&quot; &quot;\n&quot; delimited metadata defining an object stored elsewhere. Used by the LFS extension.
<dt>4096
<dd>Contains copy information. This revision changes files in a way that could affect copy tracing. This does *not* affect changegroup handling, but is relevant for other parts of Mercurial.
</dl> </dl>
<p> <p>
For historical reasons, the integer values are identical to revlog version 1 For historical reasons, the integer values are identical to revlog version 1
per-revision storage flags and correspond to bits being set in this 2-byte per-revision storage flags and correspond to bits being set in this 2-byte
field. Bits were allocated starting from the most-significant bit, hence the field. Bits were allocated starting from the most-significant bit, hence the
reverse ordering and allocation of these flags. reverse ordering and allocation of these flags.
</p> </p>
<p>
The *pflags* (protocol flags) field holds bitwise flags affecting the protocol
itself. They are first in the header since they may affect the handling of the
rest of the fields in a future version. They are defined as such:
</p>
<dl>
<dt>1 indicates whether to read a chunk of sidedata (of variable length) right
<dd>after the revision flags.
</dl>
<h2>Changeset Segment</h2> <h2>Changeset Segment</h2>
<p> <p>
The *changeset segment* consists of a single *delta group* holding The *changeset segment* consists of a single *delta group* holding
changelog data. The *empty chunk* at the end of the *delta group* denotes changelog data. The *empty chunk* at the end of the *delta group* denotes
the boundary to the *manifest segment*. the boundary to the *manifest segment*.
</p> </p>
<h2>Manifest Segment</h2> <h2>Manifest Segment</h2>
<p> <p>
The *manifest segment* consists of a single *delta group* holding manifest The *manifest segment* consists of a single *delta group* holding manifest
data. If treemanifests are in use, it contains only the manifest for the data. If treemanifests are in use, it contains only the manifest for the
root directory of the repository. Otherwise, it contains the entire root directory of the repository. Otherwise, it contains the entire
manifest data. The *empty chunk* at the end of the *delta group* denotes manifest data. The *empty chunk* at the end of the *delta group* denotes
the boundary to the next segment (either the *treemanifests segment* or the the boundary to the next segment (either the *treemanifests segment* or the
*filelogs segment*, depending on version and the request options). *filelogs segment*, depending on version and the request options).
</p> </p>
<h3>Treemanifests Segment</h3> <h3>Treemanifests Segment</h3>
<p> <p>
The *treemanifests segment* only exists in changegroup version &quot;3&quot;, and The *treemanifests segment* only exists in changegroup version &quot;3&quot; and &quot;4&quot;,
only if the 'treemanifest' param is part of the bundle2 changegroup part and only if the 'treemanifest' param is part of the bundle2 changegroup part
(it is not possible to use changegroup version 3 outside of bundle2). (it is not possible to use changegroup version 3 or 4 outside of bundle2).
Aside from the filenames in the *treemanifests segment* containing a Aside from the filenames in the *treemanifests segment* containing a
trailing &quot;/&quot; character, it behaves identically to the *filelogs segment* trailing &quot;/&quot; character, it behaves identically to the *filelogs segment*
(see below). The final sub-segment is followed by an *empty chunk* (logically, (see below). The final sub-segment is followed by an *empty chunk* (logically,
a sub-segment with filename size 0). This denotes the boundary to the a sub-segment with filename size 0). This denotes the boundary to the
*filelogs segment*. *filelogs segment*.
</p> </p>
<h2>Filelogs Segment</h2> <h2>Filelogs Segment</h2>
<p> <p>

tests/test-lfs-serve.t

... bprint(b"\n# LFS not enabled- revlogs don't have 0x2000 flag") ... bprint(b"\n# LFS not enabled- revlogs don't have 0x2000 flag")
... stdout.flush() ... stdout.flush()
... runcommand(server, [b'debugprocessors', b'nonlfs3.txt']) ... runcommand(server, [b'debugprocessors', b'nonlfs3.txt'])
... runcommand(server, [b'config', b'extensions']) ... runcommand(server, [b'config', b'extensions'])
# LFS required- both lfs and non-lfs revlogs have 0x2000 flag # LFS required- both lfs and non-lfs revlogs have 0x2000 flag
*** runcommand debugprocessors lfs.bin -R ../server *** runcommand debugprocessors lfs.bin -R ../server
registered processor '0x8000' registered processor '0x8000'
registered processor '0x800' registered processor '0x1000'
registered processor '0x2000' registered processor '0x2000'
*** runcommand debugprocessors nonlfs2.txt -R ../server *** runcommand debugprocessors nonlfs2.txt -R ../server
registered processor '0x8000' registered processor '0x8000'
registered processor '0x800' registered processor '0x1000'
registered processor '0x2000' registered processor '0x2000'
*** runcommand config extensions --cwd ../server *** runcommand config extensions --cwd ../server
extensions.debugprocessors=$TESTTMP/debugprocessors.py extensions.debugprocessors=$TESTTMP/debugprocessors.py
extensions.lfs= extensions.lfs=
# LFS not enabled- revlogs don't have 0x2000 flag # LFS not enabled- revlogs don't have 0x2000 flag
*** runcommand debugprocessors nonlfs3.txt *** runcommand debugprocessors nonlfs3.txt
registered processor '0x8000' registered processor '0x8000'
registered processor '0x800' registered processor '0x1000'
*** runcommand config extensions *** runcommand config extensions
extensions.debugprocessors=$TESTTMP/debugprocessors.py extensions.debugprocessors=$TESTTMP/debugprocessors.py
$ rm $HGRCPATH $ rm $HGRCPATH
$ mv $HGRCPATH.tmp $HGRCPATH $ mv $HGRCPATH.tmp $HGRCPATH
$ hg clone $TESTTMP/client $TESTTMP/nonlfs -qr 0 --config extensions.lfs= $ hg clone $TESTTMP/client $TESTTMP/nonlfs -qr 0 --config extensions.lfs=
$ cat >> $TESTTMP/nonlfs/.hg/hgrc <<EOF $ cat >> $TESTTMP/nonlfs/.hg/hgrc <<EOF
... runcommand(server, [b'debugprocessors', b'nonlfs.txt', b'-R', ... runcommand(server, [b'debugprocessors', b'nonlfs.txt', b'-R',
... b'../nonlfs']) ... b'../nonlfs'])
... runcommand(server, [b'config', b'extensions', b'--cwd', ... runcommand(server, [b'config', b'extensions', b'--cwd',
... b'../nonlfs']) ... b'../nonlfs'])
# LFS enabled- both lfs and non-lfs revlogs have 0x2000 flag # LFS enabled- both lfs and non-lfs revlogs have 0x2000 flag
*** runcommand debugprocessors lfs.bin -R ../server *** runcommand debugprocessors lfs.bin -R ../server
registered processor '0x8000' registered processor '0x8000'
registered processor '0x800' registered processor '0x1000'
registered processor '0x2000' registered processor '0x2000'
*** runcommand debugprocessors nonlfs2.txt -R ../server *** runcommand debugprocessors nonlfs2.txt -R ../server
registered processor '0x8000' registered processor '0x8000'
registered processor '0x800' registered processor '0x1000'
registered processor '0x2000' registered processor '0x2000'
*** runcommand config extensions --cwd ../server *** runcommand config extensions --cwd ../server
extensions.debugprocessors=$TESTTMP/debugprocessors.py extensions.debugprocessors=$TESTTMP/debugprocessors.py
extensions.lfs= extensions.lfs=
# LFS enabled without requirement- revlogs have 0x2000 flag # LFS enabled without requirement- revlogs have 0x2000 flag
*** runcommand debugprocessors nonlfs3.txt *** runcommand debugprocessors nonlfs3.txt
registered processor '0x8000' registered processor '0x8000'
registered processor '0x800' registered processor '0x1000'
registered processor '0x2000' registered processor '0x2000'
*** runcommand config extensions *** runcommand config extensions
extensions.debugprocessors=$TESTTMP/debugprocessors.py extensions.debugprocessors=$TESTTMP/debugprocessors.py
extensions.lfs= extensions.lfs=
# LFS disabled locally- revlogs don't have 0x2000 flag # LFS disabled locally- revlogs don't have 0x2000 flag
*** runcommand debugprocessors nonlfs.txt -R ../nonlfs *** runcommand debugprocessors nonlfs.txt -R ../nonlfs
registered processor '0x8000' registered processor '0x8000'
registered processor '0x800' registered processor '0x1000'
*** runcommand config extensions --cwd ../nonlfs *** runcommand config extensions --cwd ../nonlfs
extensions.debugprocessors=$TESTTMP/debugprocessors.py extensions.debugprocessors=$TESTTMP/debugprocessors.py
extensions.lfs=! extensions.lfs=!
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
Case #6: client with lfs content and the extension enabled; server with Case #6: client with lfs content and the extension enabled; server with
lfs content, and the extension enabled. lfs content, and the extension enabled.