This is an archive of the discontinued Mercurial Phabricator instance.

Differential D2393

cleanup: say goodbye to manifestv2 format
ClosedPublic

Authored by durin42 on Feb 22 2018, 10:15 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG0147a4730420: cleanup: say goodbye to manifestv2 format
Summary

This experiment was a bust: we'd hoped for smaller repository sizes,
but things got larger. Google ended up rolling out tree manifests in a
format that's compatible with the original manifest format, and I
believe Facebook is doing the same. This code was never implemented as
native speedups, so I'm pretty comfortable saying nobody is using the
experimental feature. Let's rip it out.

I noticed this code still kicking around because I was investigating a
repo corruption issue for timeless.

.. bc::

Support for the experimental manifestv2 format has been removed, as
it was never completed and failed to meet expectations.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

durin42 created this revision.Feb 22 2018, 10:15 PM
martinvonz added a subscriber: martinvonz.Feb 23 2018, 12:28 AM

LGTM. Last time I was going to remove it, I left it in because Durham felt that it was useful for forcing him to not specialize the manifest API too much for how v1 works. I believe he's pretty much done with that now, so that should no longer be a reason to keep this around. Thanks for cleaning up.

mercurial/manifest.py
537

delete "for v1"?

durin42 updated this revision to Diff 6019.Feb 23 2018, 12:24 PM
durin42 marked an inline comment as done.Feb 23 2018, 12:25 PM
Closed by commit rHG0147a4730420: cleanup: say goodbye to manifestv2 format (authored by durin42). · Explain WhyFeb 23 2018, 12:46 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 6020

mercurial/configitems.py

default=dynamicdefault, default=dynamicdefault,
) )
coreconfigitem('experimental', 'hook-track-tags', coreconfigitem('experimental', 'hook-track-tags',
default=False, default=False,
) )
coreconfigitem('experimental', 'httppostargs', coreconfigitem('experimental', 'httppostargs',
default=False, default=False,
) )
coreconfigitem('experimental', 'manifestv2',
default=False,
)
coreconfigitem('experimental', 'mergedriver', coreconfigitem('experimental', 'mergedriver',
default=None, default=None,
) )
coreconfigitem('experimental', 'obsmarkers-exchange-debug', coreconfigitem('experimental', 'obsmarkers-exchange-debug',
default=False, default=False,
) )
coreconfigitem('experimental', 'remotenames', coreconfigitem('experimental', 'remotenames',
default=False, default=False,

mercurial/help/internals/requirements.txt

Repositories contain a file (``.hg/requires``) containing a list of Repositories contain a file (``.hg/requires``) containing a list of
features/capabilities that are *required* for clients to interface features/capabilities that are *required* for clients to interface
with the repository. This file has been present in Mercurial since with the repository. This file has been present in Mercurial since
version 0.9.2 (released December 2006). version 0.9.2 (released December 2006).
One of the first things clients do when opening a repository is read One of the first things clients do when opening a repository is read
``.hg/requires`` and verify that all listed requirements are supported, ``.hg/requires`` and verify that all listed requirements are supported,
aborting if not. Requirements are therefore a strong mechanism to aborting if not. Requirements are therefore a strong mechanism to
default until Mercurial 3.7 (released February 2016). default until Mercurial 3.7 (released February 2016).
manifestv2 manifestv2
========== ==========
Denotes that version 2 of manifests are being used. Denotes that version 2 of manifests are being used.
Support for this requirement was added in Mercurial 3.4 (released Support for this requirement was added in Mercurial 3.4 (released
May 2015). The requirement is currently experimental and is disabled May 2015). The new format failed to meet expectations and support
by default. for the format and requirement were removed in Mercurial 4.6
(released May 2018) since the feature never graduated frome experiment
status.
treemanifest treemanifest
============ ============
Denotes that tree manifests are being used. Tree manifests are Denotes that tree manifests are being used. Tree manifests are
one manifest per directory (as opposed to a single flat manifest). one manifest per directory (as opposed to a single flat manifest).
Support for this requirement was added in Mercurial 3.4 (released Support for this requirement was added in Mercurial 3.4 (released

mercurial/localrepo.py

# End of baselegacywirecommands interface. # End of baselegacywirecommands interface.
# Increment the sub-version when the revlog v2 format changes to lock out old # Increment the sub-version when the revlog v2 format changes to lock out old
# clients. # clients.
REVLOGV2_REQUIREMENT = 'exp-revlogv2.0' REVLOGV2_REQUIREMENT = 'exp-revlogv2.0'
class localrepository(object): class localrepository(object):
# obsolete experimental requirements:
# - manifestv2: An experimental new manifest format that allowed
# for stem compression of long paths. Experiment ended up not
# being successful (repository sizes went up due to worse delta
# chains), and the code was deleted in 4.6.
supportedformats = { supportedformats = {
'revlogv1', 'revlogv1',
'generaldelta', 'generaldelta',
'treemanifest', 'treemanifest',
'manifestv2',
REVLOGV2_REQUIREMENT, REVLOGV2_REQUIREMENT,
} }
_basesupported = supportedformats | { _basesupported = supportedformats | {
'store', 'store',
'fncache', 'fncache',
'shared', 'shared',
'relshared', 'relshared',
'dotencode', 'dotencode',
'exp-sparse', 'exp-sparse',
} }
openerreqs = { openerreqs = {
'revlogv1', 'revlogv1',
'generaldelta', 'generaldelta',
'treemanifest', 'treemanifest',
'manifestv2',
} }
# a list of (ui, featureset) functions. # a list of (ui, featureset) functions.
# only functions defined in module of enabled extensions are invoked # only functions defined in module of enabled extensions are invoked
featuresetupfuncs = set() featuresetupfuncs = set()
# list of prefix for file which can be written without 'wlock' # list of prefix for file which can be written without 'wlock'
# Extensions should extend this list when needed # Extensions should extend this list when needed
# zlib is the historical default and doesn't need an explicit requirement. # zlib is the historical default and doesn't need an explicit requirement.
if compengine != 'zlib': if compengine != 'zlib':
requirements.add('exp-compression-%s' % compengine) requirements.add('exp-compression-%s' % compengine)
if scmutil.gdinitconfig(ui): if scmutil.gdinitconfig(ui):
requirements.add('generaldelta') requirements.add('generaldelta')
if ui.configbool('experimental', 'treemanifest'): if ui.configbool('experimental', 'treemanifest'):
requirements.add('treemanifest') requirements.add('treemanifest')
if ui.configbool('experimental', 'manifestv2'):
requirements.add('manifestv2')
revlogv2 = ui.config('experimental', 'revlogv2') revlogv2 = ui.config('experimental', 'revlogv2')
if revlogv2 == 'enable-unstable-format-and-corrupt-my-data': if revlogv2 == 'enable-unstable-format-and-corrupt-my-data':
requirements.remove('revlogv1') requirements.remove('revlogv1')
# generaldelta is implied by revlogv2. # generaldelta is implied by revlogv2.
requirements.discard('generaldelta') requirements.discard('generaldelta')
requirements.add(REVLOGV2_REQUIREMENT) requirements.add(REVLOGV2_REQUIREMENT)
return requirements return requirements

mercurial/manifest.py

# manifest.py - manifest revision class for mercurial # manifest.py - manifest revision class for mercurial
# #
# Copyright 2005-2007 Matt Mackall <mpm@selenic.com> # Copyright 2005-2007 Matt Mackall <mpm@selenic.com>
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
from __future__ import absolute_import from __future__ import absolute_import
import heapq import heapq
import itertools import itertools
import os
import struct import struct
from .i18n import _ from .i18n import _
from .node import ( from .node import (
bin, bin,
hex, hex,
) )
from . import ( from . import (
error, error,
mdiff, mdiff,
policy, policy,
revlog, revlog,
util, util,
) )
parsers = policy.importmod(r'parsers') parsers = policy.importmod(r'parsers')
propertycache = util.propertycache propertycache = util.propertycache
def _parsev1(data): def _parse(data):
# This method does a little bit of excessive-looking # This method does a little bit of excessive-looking
# precondition checking. This is so that the behavior of this # precondition checking. This is so that the behavior of this
# class exactly matches its C counterpart to try and help # class exactly matches its C counterpart to try and help
# prevent surprise breakage for anyone that develops against # prevent surprise breakage for anyone that develops against
# the pure version. # the pure version.
if data and data[-1:] != '\n': if data and data[-1:] != '\n':
raise ValueError('Manifest did not end in a newline.') raise ValueError('Manifest did not end in a newline.')
prev = None prev = None
for l in data.splitlines(): for l in data.splitlines():
if prev is not None and prev > l: if prev is not None and prev > l:
raise ValueError('Manifest lines not in sorted order.') raise ValueError('Manifest lines not in sorted order.')
prev = l prev = l
f, n = l.split('\0') f, n = l.split('\0')
if len(n) > 40: if len(n) > 40:
yield f, bin(n[:40]), n[40:] yield f, bin(n[:40]), n[40:]
else: else:
yield f, bin(n), '' yield f, bin(n), ''
def _parsev2(data): def _text(it):
metadataend = data.find('\n')
# Just ignore metadata for now
pos = metadataend + 1
prevf = ''
while pos < len(data):
end = data.find('\n', pos + 1) # +1 to skip stem length byte
if end == -1:
raise ValueError('Manifest ended with incomplete file entry.')
stemlen = ord(data[pos:pos + 1])
items = data[pos + 1:end].split('\0')
f = prevf[:stemlen] + items[0]
if prevf > f:
raise ValueError('Manifest entries not in sorted order.')
fl = items[1]
# Just ignore metadata (items[2:] for now)
n = data[end + 1:end + 21]
yield f, n, fl
pos = end + 22
prevf = f
def _parse(data):
"""Generates (path, node, flags) tuples from a manifest text"""
if data.startswith('\0'):
return iter(_parsev2(data))
else:
return iter(_parsev1(data))
def _text(it, usemanifestv2):
"""Given an iterator over (path, node, flags) tuples, returns a manifest
text"""
if usemanifestv2:
return _textv2(it)
else:
return _textv1(it)
def _textv1(it):
files = [] files = []
lines = [] lines = []
_hex = revlog.hex _hex = revlog.hex
for f, n, fl in it: for f, n, fl in it:
files.append(f) files.append(f)
# if this is changed to support newlines in filenames, # if this is changed to support newlines in filenames,
# be sure to check the templates/ dir again (especially *-raw.tmpl) # be sure to check the templates/ dir again (especially *-raw.tmpl)
lines.append("%s\0%s%s\n" % (f, _hex(n), fl)) lines.append("%s\0%s%s\n" % (f, _hex(n), fl))
_checkforbidden(files) _checkforbidden(files)
return ''.join(lines) return ''.join(lines)
def _textv2(it):
files = []
lines = ['\0\n']
prevf = ''
for f, n, fl in it:
files.append(f)
stem = os.path.commonprefix([prevf, f])
stemlen = min(len(stem), 255)
lines.append("%c%s\0%s\n%s\n" % (stemlen, f[stemlen:], fl, n))
prevf = f
_checkforbidden(files)
return ''.join(lines)
class lazymanifestiter(object): class lazymanifestiter(object):
def __init__(self, lm): def __init__(self, lm):
self.pos = 0 self.pos = 0
self.lm = lm self.lm = lm
def __iter__(self): def __iter__(self):
return self return self
try: try:
_lazymanifest = parsers.lazymanifest _lazymanifest = parsers.lazymanifest
except AttributeError: except AttributeError:
pass pass
class manifestdict(object): class manifestdict(object):
def __init__(self, data=''): def __init__(self, data=''):
if data.startswith('\0'):
#_lazymanifest can not parse v2
self._lm = _lazymanifest('')
for f, n, fl in _parsev2(data):
self._lm[f] = n, fl
else:
self._lm = _lazymanifest(data) self._lm = _lazymanifest(data)
def __getitem__(self, key): def __getitem__(self, key):
return self._lm[key][0] return self._lm[key][0]
def find(self, key): def find(self, key):
return self._lm[key] return self._lm[key]
def __len__(self): def __len__(self):
def items(self): def items(self):
return (x[:2] for x in self._lm.iterentries()) return (x[:2] for x in self._lm.iterentries())
iteritems = items iteritems = items
def iterentries(self): def iterentries(self):
return self._lm.iterentries() return self._lm.iterentries()
def text(self, usemanifestv2=False): def text(self):
if usemanifestv2: # most likely uses native version
martinvonzUnsubmitted
Done

delete "for v1"?

martinvonz: delete "for v1"?
return _textv2(self._lm.iterentries())
else:
# use (probably) native version for v1
return self._lm.text() return self._lm.text()
def fastdelta(self, base, changes): def fastdelta(self, base, changes):
"""Given a base manifest text as a bytearray and a list of changes """Given a base manifest text as a bytearray and a list of changes
relative to that text, compute a delta that can be used by revlog. relative to that text, compute a delta that can be used by revlog.
""" """
delta = [] delta = []
dstart = None dstart = None
dend = None dend = None
self.setflag(f, fl) self.setflag(f, fl)
else: else:
# Assigning to _files and _flags avoids marking as dirty, # Assigning to _files and _flags avoids marking as dirty,
# and should be a little faster. # and should be a little faster.
self._files[f] = n self._files[f] = n
if fl: if fl:
self._flags[f] = fl self._flags[f] = fl
def text(self, usemanifestv2=False): def text(self):
"""Get the full data of this manifest as a bytestring.""" """Get the full data of this manifest as a bytestring."""
self._load() self._load()
return _text(self.iterentries(), usemanifestv2) return _text(self.iterentries())
def dirtext(self, usemanifestv2=False): def dirtext(self):
"""Get the full data of this directory as a bytestring. Make sure that """Get the full data of this directory as a bytestring. Make sure that
any submanifests have been written first, so their nodeids are correct. any submanifests have been written first, so their nodeids are correct.
""" """
self._load() self._load()
flags = self.flags flags = self.flags
dirs = [(d[:-1], self._dirs[d]._node, 't') for d in self._dirs] dirs = [(d[:-1], self._dirs[d]._node, 't') for d in self._dirs]
files = [(f, self._files[f], flags(f)) for f in self._files] files = [(f, self._files[f], flags(f)) for f in self._files]
return _text(sorted(dirs + files), usemanifestv2) return _text(sorted(dirs + files))
def read(self, gettext, readsubtree): def read(self, gettext, readsubtree):
def _load_for_read(s): def _load_for_read(s):
s.parse(gettext(), readsubtree) s.parse(gettext(), readsubtree)
s._dirty = False s._dirty = False
self._loadfunc = _load_for_read self._loadfunc = _load_for_read
def writesubtrees(self, m1, m2, writesubtree): def writesubtrees(self, m1, m2, writesubtree):
option takes precedence, so if it is set to True, we ignore whatever option takes precedence, so if it is set to True, we ignore whatever
value is passed in to the constructor. value is passed in to the constructor.
""" """
# During normal operations, we expect to deal with not more than four # During normal operations, we expect to deal with not more than four
# revs at a time (such as during commit --amend). When rebasing large # revs at a time (such as during commit --amend). When rebasing large
# stacks of commits, the number can go up, hence the config knob below. # stacks of commits, the number can go up, hence the config knob below.
cachesize = 4 cachesize = 4
optiontreemanifest = False optiontreemanifest = False
usemanifestv2 = False
opts = getattr(opener, 'options', None) opts = getattr(opener, 'options', None)
if opts is not None: if opts is not None:
cachesize = opts.get('manifestcachesize', cachesize) cachesize = opts.get('manifestcachesize', cachesize)
optiontreemanifest = opts.get('treemanifest', False) optiontreemanifest = opts.get('treemanifest', False)
usemanifestv2 = opts.get('manifestv2', usemanifestv2)
self._treeondisk = optiontreemanifest or treemanifest self._treeondisk = optiontreemanifest or treemanifest
self._usemanifestv2 = usemanifestv2
self._fulltextcache = util.lrucachedict(cachesize) self._fulltextcache = util.lrucachedict(cachesize)
if dir: if dir:
assert self._treeondisk, 'opts is %r' % opts assert self._treeondisk, 'opts is %r' % opts
if not dir.endswith('/'): if not dir.endswith('/'):
dir = dir + '/' dir = dir + '/'
if d not in self._dirlogcache: if d not in self._dirlogcache:
mfrevlog = manifestrevlog(self.opener, d, mfrevlog = manifestrevlog(self.opener, d,
self._dirlogcache, self._dirlogcache,
treemanifest=self._treeondisk) treemanifest=self._treeondisk)
self._dirlogcache[d] = mfrevlog self._dirlogcache[d] = mfrevlog
return self._dirlogcache[d] return self._dirlogcache[d]
def add(self, m, transaction, link, p1, p2, added, removed, readtree=None): def add(self, m, transaction, link, p1, p2, added, removed, readtree=None):
if (p1 in self.fulltextcache and util.safehasattr(m, 'fastdelta') if p1 in self.fulltextcache and util.safehasattr(m, 'fastdelta'):
and not self._usemanifestv2):
# If our first parent is in the manifest cache, we can # If our first parent is in the manifest cache, we can
# compute a delta here using properties we know about the # compute a delta here using properties we know about the
# manifest up-front, which may save time later for the # manifest up-front, which may save time later for the
# revlog layer. # revlog layer.
_checkforbidden(added) _checkforbidden(added)
# combine the changed lists into one sorted iterator # combine the changed lists into one sorted iterator
work = heapq.merge([(x, False) for x in added], work = heapq.merge([(x, False) for x in added],
# process. # process.
if self._treeondisk: if self._treeondisk:
assert readtree, "readtree must be set for treemanifest writes" assert readtree, "readtree must be set for treemanifest writes"
m1 = readtree(self._dir, p1) m1 = readtree(self._dir, p1)
m2 = readtree(self._dir, p2) m2 = readtree(self._dir, p2)
n = self._addtree(m, transaction, link, m1, m2, readtree) n = self._addtree(m, transaction, link, m1, m2, readtree)
arraytext = None arraytext = None
else: else:
text = m.text(self._usemanifestv2) text = m.text()
n = self.addrevision(text, transaction, link, p1, p2) n = self.addrevision(text, transaction, link, p1, p2)
arraytext = bytearray(text) arraytext = bytearray(text)
if arraytext is not None: if arraytext is not None:
self.fulltextcache[n] = arraytext self.fulltextcache[n] = arraytext
return n return n
def _addtree(self, m, transaction, link, m1, m2, readtree): def _addtree(self, m, transaction, link, m1, m2, readtree):
# If the manifest is unchanged compared to one parent, # If the manifest is unchanged compared to one parent,
# don't write a new revision # don't write a new revision
if self._dir != '' and (m.unmodifiedsince(m1) or m.unmodifiedsince(m2)): if self._dir != '' and (m.unmodifiedsince(m1) or m.unmodifiedsince(m2)):
return m.node() return m.node()
def writesubtree(subm, subp1, subp2): def writesubtree(subm, subp1, subp2):
sublog = self.dirlog(subm.dir()) sublog = self.dirlog(subm.dir())
sublog.add(subm, transaction, link, subp1, subp2, None, None, sublog.add(subm, transaction, link, subp1, subp2, None, None,
readtree=readtree) readtree=readtree)
m.writesubtrees(m1, m2, writesubtree) m.writesubtrees(m1, m2, writesubtree)
text = m.dirtext(self._usemanifestv2) text = m.dirtext()
n = None n = None
if self._dir != '': if self._dir != '':
# Double-check whether contents are unchanged to one parent # Double-check whether contents are unchanged to one parent
if text == m1.dirtext(self._usemanifestv2): if text == m1.dirtext():
n = m1.node() n = m1.node()
elif text == m2.dirtext(self._usemanifestv2): elif text == m2.dirtext():
n = m2.node() n = m2.node()
if not n: if not n:
n = self.addrevision(text, transaction, link, m1.node(), m2.node()) n = self.addrevision(text, transaction, link, m1.node(), m2.node())
# Save nodeid so parent manifest can calculate its nodeid # Save nodeid so parent manifest can calculate its nodeid
m.setnode(n) m.setnode(n)
return n return n
def readdelta(self, shallow=False): def readdelta(self, shallow=False):
'''Returns a manifest containing just the entries that are present '''Returns a manifest containing just the entries that are present
in this manifest, but not in its p1 manifest. This is efficient to read in this manifest, but not in its p1 manifest. This is efficient to read
if the revlog delta is already p1. if the revlog delta is already p1.
Changing the value of `shallow` has no effect on flat manifests. Changing the value of `shallow` has no effect on flat manifests.
''' '''
revlog = self._revlog() revlog = self._revlog()
if revlog._usemanifestv2:
# Need to perform a slow delta
r0 = revlog.deltaparent(revlog.rev(self._node))
m0 = self._manifestlog[revlog.node(r0)].read()
m1 = self.read()
md = manifestdict()
for f, ((n0, fl0), (n1, fl1)) in m0.diff(m1).iteritems():
if n1:
md[f] = n1
if fl1:
md.setflag(f, fl1)
return md
r = revlog.rev(self._node) r = revlog.rev(self._node)
d = mdiff.patchtext(revlog.revdiff(revlog.deltaparent(r), r)) d = mdiff.patchtext(revlog.revdiff(revlog.deltaparent(r), r))
return manifestdict(d) return manifestdict(d)
def find(self, key): def find(self, key):
return self.read().find(key) return self.read().find(key)
class memtreemanifestctx(object): class memtreemanifestctx(object):
If `shallow` is True, this will read the delta for this directory, If `shallow` is True, this will read the delta for this directory,
without recursively reading subdirectory manifests. Instead, any without recursively reading subdirectory manifests. Instead, any
subdirectory entry will be reported as it appears in the manifest, i.e. subdirectory entry will be reported as it appears in the manifest, i.e.
the subdirectory will be reported among files and distinguished only by the subdirectory will be reported among files and distinguished only by
its 't' flag. its 't' flag.
''' '''
revlog = self._revlog() revlog = self._revlog()
if shallow and not revlog._usemanifestv2: if shallow:
r = revlog.rev(self._node) r = revlog.rev(self._node)
d = mdiff.patchtext(revlog.revdiff(revlog.deltaparent(r), r)) d = mdiff.patchtext(revlog.revdiff(revlog.deltaparent(r), r))
return manifestdict(d) return manifestdict(d)
else: else:
# Need to perform a slow delta # Need to perform a slow delta
r0 = revlog.deltaparent(revlog.rev(self._node)) r0 = revlog.deltaparent(revlog.rev(self._node))
m0 = self._manifestlog.get(self._dir, revlog.node(r0)).read() m0 = self._manifestlog.get(self._dir, revlog.node(r0)).read()
m1 = self.read() m1 = self.read()

mercurial/upgrade.py

"""Obtain requirements that will prevent an upgrade from occurring. """Obtain requirements that will prevent an upgrade from occurring.
An upgrade cannot be performed if the source repository contains a An upgrade cannot be performed if the source repository contains a
requirements in the returned set. requirements in the returned set.
""" """
return { return {
# The upgrade code does not yet support these experimental features. # The upgrade code does not yet support these experimental features.
# This is an artificial limitation. # This is an artificial limitation.
'manifestv2',
'treemanifest', 'treemanifest',
# This was a precursor to generaldelta and was never enabled by default. # This was a precursor to generaldelta and was never enabled by default.
# It should (hopefully) not exist in the wild. # It should (hopefully) not exist in the wild.
'parentdelta', 'parentdelta',
# Upgrade should operate on the actual store, not the shared link. # Upgrade should operate on the actual store, not the shared link.
'shared', 'shared',
} }

tests/test-manifest.py

from __future__ import absolute_import from __future__ import absolute_import
import binascii import binascii
import itertools import itertools
import silenttestrunner import silenttestrunner
import unittest import unittest
from mercurial import ( from mercurial import (
manifest as manifestmod, manifest as manifestmod,
match as matchmod, match as matchmod,
) )
EMTPY_MANIFEST = b'' EMTPY_MANIFEST = b''
EMTPY_MANIFEST_V2 = b'\0\n'
HASH_1 = b'1' * 40 HASH_1 = b'1' * 40
BIN_HASH_1 = binascii.unhexlify(HASH_1) BIN_HASH_1 = binascii.unhexlify(HASH_1)
HASH_2 = b'f' * 40 HASH_2 = b'f' * 40
BIN_HASH_2 = binascii.unhexlify(HASH_2) BIN_HASH_2 = binascii.unhexlify(HASH_2)
HASH_3 = b'1234567890abcdef0987654321deadbeef0fcafe' HASH_3 = b'1234567890abcdef0987654321deadbeef0fcafe'
BIN_HASH_3 = binascii.unhexlify(HASH_3) BIN_HASH_3 = binascii.unhexlify(HASH_3)
A_SHORT_MANIFEST = ( A_SHORT_MANIFEST = (
b'bar/baz/qux.py\0%(hash2)s%(flag2)s\n' b'bar/baz/qux.py\0%(hash2)s%(flag2)s\n'
b'foo\0%(hash1)s%(flag1)s\n' b'foo\0%(hash1)s%(flag1)s\n'
) % {b'hash1': HASH_1, ) % {b'hash1': HASH_1,
b'flag1': b'', b'flag1': b'',
b'hash2': HASH_2, b'hash2': HASH_2,
b'flag2': b'l', b'flag2': b'l',
} }
# Same data as A_SHORT_MANIFEST
A_SHORT_MANIFEST_V2 = (
b'\0\n'
b'\x00bar/baz/qux.py\0%(flag2)s\n%(hash2)s\n'
b'\x00foo\0%(flag1)s\n%(hash1)s\n'
) % {b'hash1': BIN_HASH_1,
b'flag1': b'',
b'hash2': BIN_HASH_2,
b'flag2': b'l',
}
# Same data as A_SHORT_MANIFEST
A_METADATA_MANIFEST = (
b'\0foo\0bar\n'
b'\x00bar/baz/qux.py\0%(flag2)s\0foo\0bar\n%(hash2)s\n' # flag and metadata
b'\x00foo\0%(flag1)s\0foo\n%(hash1)s\n' # no flag, but metadata
) % {b'hash1': BIN_HASH_1,
b'flag1': b'',
b'hash2': BIN_HASH_2,
b'flag2': b'l',
}
A_STEM_COMPRESSED_MANIFEST = (
b'\0\n'
b'\x00bar/baz/qux.py\0%(flag2)s\n%(hash2)s\n'
b'\x04qux/foo.py\0%(flag1)s\n%(hash1)s\n' # simple case of 4 stem chars
b'\x0az.py\0%(flag1)s\n%(hash1)s\n' # tricky newline = 10 stem characters
b'\x00%(verylongdir)sx/x\0\n%(hash1)s\n'
b'\xffx/y\0\n%(hash2)s\n' # more than 255 stem chars
) % {b'hash1': BIN_HASH_1,
b'flag1': b'',
b'hash2': BIN_HASH_2,
b'flag2': b'l',
b'verylongdir': 255 * b'x',
}
A_DEEPER_MANIFEST = ( A_DEEPER_MANIFEST = (
b'a/b/c/bar.py\0%(hash3)s%(flag1)s\n' b'a/b/c/bar.py\0%(hash3)s%(flag1)s\n'
b'a/b/c/bar.txt\0%(hash1)s%(flag1)s\n' b'a/b/c/bar.txt\0%(hash1)s%(flag1)s\n'
b'a/b/c/foo.py\0%(hash3)s%(flag1)s\n' b'a/b/c/foo.py\0%(hash3)s%(flag1)s\n'
b'a/b/c/foo.txt\0%(hash2)s%(flag2)s\n' b'a/b/c/foo.txt\0%(hash2)s%(flag2)s\n'
b'a/b/d/baz.py\0%(hash3)s%(flag1)s\n' b'a/b/d/baz.py\0%(hash3)s%(flag1)s\n'
b'a/b/d/qux.py\0%(hash1)s%(flag2)s\n' b'a/b/d/qux.py\0%(hash1)s%(flag2)s\n'
b'a/b/d/ten.txt\0%(hash3)s%(flag2)s\n' b'a/b/d/ten.txt\0%(hash3)s%(flag2)s\n'
def parsemanifest(self, text): def parsemanifest(self, text):
raise NotImplementedError('parsemanifest not implemented by test case') raise NotImplementedError('parsemanifest not implemented by test case')
def testEmptyManifest(self): def testEmptyManifest(self):
m = self.parsemanifest(EMTPY_MANIFEST) m = self.parsemanifest(EMTPY_MANIFEST)
self.assertEqual(0, len(m)) self.assertEqual(0, len(m))
self.assertEqual([], list(m)) self.assertEqual([], list(m))
def testEmptyManifestv2(self):
m = self.parsemanifest(EMTPY_MANIFEST_V2)
self.assertEqual(0, len(m))
self.assertEqual([], list(m))
def testManifest(self): def testManifest(self):
m = self.parsemanifest(A_SHORT_MANIFEST) m = self.parsemanifest(A_SHORT_MANIFEST)
self.assertEqual([b'bar/baz/qux.py', b'foo'], list(m)) self.assertEqual([b'bar/baz/qux.py', b'foo'], list(m))
self.assertEqual(BIN_HASH_2, m[b'bar/baz/qux.py']) self.assertEqual(BIN_HASH_2, m[b'bar/baz/qux.py'])
self.assertEqual(b'l', m.flags(b'bar/baz/qux.py')) self.assertEqual(b'l', m.flags(b'bar/baz/qux.py'))
self.assertEqual(BIN_HASH_1, m[b'foo']) self.assertEqual(BIN_HASH_1, m[b'foo'])
self.assertEqual(b'', m.flags(b'foo')) self.assertEqual(b'', m.flags(b'foo'))
with self.assertRaises(KeyError): with self.assertRaises(KeyError):
m[b'wat'] m[b'wat']
def testParseManifestV2(self):
m1 = self.parsemanifest(A_SHORT_MANIFEST)
m2 = self.parsemanifest(A_SHORT_MANIFEST_V2)
# Should have same content as A_SHORT_MANIFEST
self.assertEqual(m1.text(), m2.text())
def testParseManifestMetadata(self):
# Metadata is for future-proofing and should be accepted but ignored
m = self.parsemanifest(A_METADATA_MANIFEST)
self.assertEqual(A_SHORT_MANIFEST, m.text())
def testParseManifestStemCompression(self):
m = self.parsemanifest(A_STEM_COMPRESSED_MANIFEST)
self.assertIn(b'bar/baz/qux.py', m)
self.assertIn(b'bar/qux/foo.py', m)
self.assertIn(b'bar/qux/foz.py', m)
self.assertIn(256 * b'x' + b'/x', m)
self.assertIn(256 * b'x' + b'/y', m)
self.assertEqual(A_STEM_COMPRESSED_MANIFEST, m.text(usemanifestv2=True))
def testTextV2(self):
m1 = self.parsemanifest(A_SHORT_MANIFEST)
v2text = m1.text(usemanifestv2=True)
self.assertEqual(A_SHORT_MANIFEST_V2, v2text)
def testSetItem(self): def testSetItem(self):
want = BIN_HASH_1 want = BIN_HASH_1
m = self.parsemanifest(EMTPY_MANIFEST) m = self.parsemanifest(EMTPY_MANIFEST)
m[b'a'] = want m[b'a'] = want
self.assertIn(b'a', m) self.assertIn(b'a', m)
self.assertEqual(want, m[b'a']) self.assertEqual(want, m[b'a'])
self.assertEqual(b'a\0' + HASH_1 + b'\n', m.text()) self.assertEqual(b'a\0' + HASH_1 + b'\n', m.text())

tests/test-manifestv2.t

  • This file was deleted.
Create repo with old manifest
$ cat << EOF >> $HGRCPATH
> [format]
> usegeneraldelta=yes
> EOF
$ hg init existing
$ cd existing
$ echo footext > foo
$ hg add foo
$ hg commit -m initial
We're using v1, so no manifestv2 entry is in requires yet.
$ grep manifestv2 .hg/requires
[1]
Let's clone this with manifestv2 enabled to switch to the new format for
future commits.
$ cd ..
$ hg clone --pull existing new --config experimental.manifestv2=1
requesting all changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
new changesets 0fc9a4fafa44
updating to branch default
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd new
Check that entry was added to .hg/requires.
$ grep manifestv2 .hg/requires
manifestv2
Make a new commit.
$ echo newfootext > foo
$ hg commit -m new
Check that the manifest actually switched to v2.
$ hg debugdata -m 0
foo\x0021e958b1dca695a60ee2e9cf151753204ee0f9e9 (esc)
$ hg debugdata -m 1
\x00 (esc)
\x00foo\x00 (esc)
I\xab\x7f\xb8(\x83\xcas\x15\x9d\xc2\xd3\xd3:5\x08\xbad5_ (esc)
Check that manifestv2 is used if the requirement is present, even if it's
disabled in the config.
$ echo newerfootext > foo
$ hg --config experimental.manifestv2=False commit -m newer
$ hg debugdata -m 2
\x00 (esc)
\x00foo\x00 (esc)
\xa6\xb1\xfb\xef]\x91\xa1\x19`\xf3.#\x90S\xf8\x06 \xe2\x19\x00 (esc)
Check that we can still read v1 manifests.
$ hg files -r 0
foo
$ cd ..
Check that entry is added to .hg/requires on repo creation
$ hg --config experimental.manifestv2=True init repo
$ cd repo
$ grep manifestv2 .hg/requires
manifestv2
Set up simple repo
$ echo a > file1
$ echo b > file2
$ echo c > file3
$ hg ci -Aqm 'initial'
$ echo d > file2
$ hg ci -m 'modify file2'
Check that 'hg verify', which uses manifest.readdelta(), works
$ hg verify
checking changesets
checking manifests
crosschecking files in changesets and manifests
checking files
3 files, 2 changesets, 4 total revisions
Check that manifest revlog is smaller than for v1
$ hg debugindex -m
rev offset length delta linkrev nodeid p1 p2
0 0 81 -1 0 57361477c778 000000000000 000000000000
1 81 33 0 1 aeaab5a2ef74 57361477c778 000000000000

tests/test-upgrade-repo.t

$ hg init share-parent $ hg init share-parent
$ hg -q share share-parent share-child $ hg -q share share-parent share-child
$ hg -R share-child debugupgraderepo $ hg -R share-child debugupgraderepo
abort: cannot upgrade repository; unsupported source requirement: shared abort: cannot upgrade repository; unsupported source requirement: shared
[255] [255]
Do not yet support upgrading manifestv2 and treemanifest repos Do not yet support upgrading treemanifest repos
$ hg --config experimental.manifestv2=true init manifestv2
$ hg -R manifestv2 debugupgraderepo
abort: cannot upgrade repository; unsupported source requirement: manifestv2
[255]
$ hg --config experimental.treemanifest=true init treemanifest $ hg --config experimental.treemanifest=true init treemanifest
$ hg -R treemanifest debugupgraderepo $ hg -R treemanifest debugupgraderepo
abort: cannot upgrade repository; unsupported source requirement: treemanifest abort: cannot upgrade repository; unsupported source requirement: treemanifest
[255] [255]
Cannot add manifestv2 or treemanifest requirement during upgrade Cannot add treemanifest requirement during upgrade
$ hg init disallowaddedreq $ hg init disallowaddedreq
$ hg -R disallowaddedreq --config experimental.manifestv2=true --config experimental.treemanifest=true debugupgraderepo $ hg -R disallowaddedreq --config experimental.treemanifest=true debugupgraderepo
abort: cannot upgrade repository; do not support adding requirement: manifestv2, treemanifest abort: cannot upgrade repository; do not support adding requirement: treemanifest
[255] [255]
An upgrade of a repository created with recommended settings only suggests optimizations An upgrade of a repository created with recommended settings only suggests optimizations
$ hg init empty $ hg init empty
$ cd empty $ cd empty
$ hg debugformat $ hg debugformat
format-variant repo format-variant repo
tests/test-manifest.py