( )⚙ D6416 changelog: optionally store added and removed files in changeset extras

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D6416

changelog: optionally store added and removed files in changeset extras
ClosedPublic

Authored by martinvonz on May 21 2019, 8:32 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHGf385ba70e4af: changelog: optionally store added and removed files in changeset extras
Summary

As mentioned in an earlier patch, copies._chain() is used a lot in the
changeset-centric version of pathcopies(). It is expensive because it
needs to look at the manifest in order to filter out copies whose
target file has since been removed. I want to store the sets of added
and removed files in the changeset in order to speed that up. This
patch does the writing part of that. It could easily be a separate
config, but it's currently tied to experimental.copies.write-to since
that's the only real use case (it will also make the {file_*} template
keywords faster, but I doubt that anyone cares enough about those to
write extra metadata for them).

The new information is stored in the changeset extras. Since they're
always subsets of the changeset's "files" list, they're stored as
indexes into that list. I've stored the indexes as stringified ints
separated by NUL bytes. The size of 00changelog.d for the hg repo
increased in size by 0.28% percent (compared to the size with only
copy information in the changesets, which in turn is 0.17% larger than
without copy information). We could store only the delta between the
indexes and we could store them in binary, but the chosen format is
more readable.

We could also have implemented this as a cache outside the
changelog. One advantage of doing it that way is that we would get the
speedups from the {file_*} template keywords also on old
repos. Another advantage is that it we can rewrite the cache if we
find a bug in how we calculate the set of files. A disadvantage is
that it would be more complex. Another is that it would surely use
more space. We already write the copy information to the changeset
extras, so it seems like a small step to also write these file sets.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Revision Contents
Changeset List

Diff 15323

mercurial/changelog.py

k, v = l.split('\0') k, v = l.split('\0')
copies[k] = v copies[k] = v
return copies return copies
except ValueError: except ValueError:
# Perhaps someone had chosen the same key name (e.g. "p1copies") and # Perhaps someone had chosen the same key name (e.g. "p1copies") and
# used different syntax for the value. # used different syntax for the value.
return None return None
def encodefileindices(files, subset):
subset = set(subset)
indices = []
for i, f in enumerate(files):
if f in subset:
indices.append('%d' % i)
return '\0'.join(indices)
def stripdesc(desc): def stripdesc(desc):
"""strip trailing whitespace and leading and trailing empty lines""" """strip trailing whitespace and leading and trailing empty lines"""
return '\n'.join([l.rstrip() for l in desc.splitlines()]).strip('\n') return '\n'.join([l.rstrip() for l in desc.splitlines()]).strip('\n')
class appender(object): class appender(object):
'''the changelog index must be updated last on disk, so we use this class '''the changelog index must be updated last on disk, so we use this class
to delay writes to it''' to delay writes to it'''
def __init__(self, vfs, name, mode, buf): def __init__(self, vfs, name, mode, buf):
text = self.revision(node) text = self.revision(node)
if not text: if not text:
return [] return []
last = text.index("\n\n") last = text.index("\n\n")
l = text[:last].split('\n') l = text[:last].split('\n')
return l[3:] return l[3:]
def add(self, manifest, files, desc, transaction, p1, p2, def add(self, manifest, files, desc, transaction, p1, p2,
user, date=None, extra=None, p1copies=None, p2copies=None): user, date=None, extra=None, p1copies=None, p2copies=None,
filesadded=None, filesremoved=None):
# Convert to UTF-8 encoded bytestrings as the very first # Convert to UTF-8 encoded bytestrings as the very first
# thing: calling any method on a localstr object will turn it # thing: calling any method on a localstr object will turn it
# into a str object and the cached UTF-8 string is thus lost. # into a str object and the cached UTF-8 string is thus lost.
user, desc = encoding.fromlocal(user), encoding.fromlocal(desc) user, desc = encoding.fromlocal(user), encoding.fromlocal(desc)
user = user.strip() user = user.strip()
# An empty username or a username with a "\n" will make the # An empty username or a username with a "\n" will make the
# revision text contain two "\n\n" sequences -> corrupt # revision text contain two "\n\n" sequences -> corrupt
parseddate = "%d %d" % dateutil.makedate() parseddate = "%d %d" % dateutil.makedate()
if extra: if extra:
branch = extra.get("branch") branch = extra.get("branch")
if branch in ("default", ""): if branch in ("default", ""):
del extra["branch"] del extra["branch"]
elif branch in (".", "null", "tip"): elif branch in (".", "null", "tip"):
raise error.StorageError(_('the name \'%s\' is reserved') raise error.StorageError(_('the name \'%s\' is reserved')
% branch) % branch)
if (p1copies is not None or p2copies is not None) and extra is None: extrasentries = p1copies, p2copies, filesadded, filesremoved
if extra is None and any(x is not None for x in extrasentries):
extra = {} extra = {}
if p1copies is not None: if p1copies is not None:
extra['p1copies'] = encodecopies(p1copies) extra['p1copies'] = encodecopies(p1copies)
if p2copies is not None: if p2copies is not None:
extra['p2copies'] = encodecopies(p2copies) extra['p2copies'] = encodecopies(p2copies)
sortedfiles = sorted(files)
if filesadded is not None:
extra['filesadded'] = encodefileindices(sortedfiles, filesadded)
if filesremoved is not None:
extra['filesremoved'] = encodefileindices(sortedfiles, filesremoved)
if extra: if extra:
extra = encodeextra(extra) extra = encodeextra(extra)
parseddate = "%s %s" % (parseddate, extra) parseddate = "%s %s" % (parseddate, extra)
l = [hex(manifest), user, parseddate] + sorted(files) + ["", desc] l = [hex(manifest), user, parseddate] + sortedfiles + ["", desc]
text = "\n".join(l) text = "\n".join(l)
return self.addrevision(text, transaction, len(self), p1, p2) return self.addrevision(text, transaction, len(self), p1, p2)
def branchinfo(self, rev): def branchinfo(self, rev):
"""return the branch name and open/close state of a revision """return the branch name and open/close state of a revision
This function exists because creating a changectx object This function exists because creating a changectx object
just to access this is costly.""" just to access this is costly."""

mercurial/localrepo.py

from p1 or p2 are excluded from the committed ctx.files(). from p1 or p2 are excluded from the committed ctx.files().
""" """
p1, p2 = ctx.p1(), ctx.p2() p1, p2 = ctx.p1(), ctx.p2()
user = ctx.user() user = ctx.user()
writecopiesto = self.ui.config('experimental', 'copies.write-to') writecopiesto = self.ui.config('experimental', 'copies.write-to')
writefilecopymeta = writecopiesto != 'changeset-only' writefilecopymeta = writecopiesto != 'changeset-only'
writechangesetcopy = (writecopiesto in
('changeset-only', 'compatibility'))
p1copies, p2copies = None, None p1copies, p2copies = None, None
if writecopiesto in ('changeset-only', 'compatibility'): if writechangesetcopy:
p1copies = ctx.p1copies() p1copies = ctx.p1copies()
p2copies = ctx.p2copies() p2copies = ctx.p2copies()
filesadded, filesremoved = None, None
with self.lock(), self.transaction("commit") as tr: with self.lock(), self.transaction("commit") as tr:
trp = weakref.proxy(tr) trp = weakref.proxy(tr)
if ctx.manifestnode(): if ctx.manifestnode():
# reuse an existing manifest revision # reuse an existing manifest revision
self.ui.debug('reusing known manifest\n') self.ui.debug('reusing known manifest\n')
mn = ctx.manifestnode() mn = ctx.manifestnode()
files = ctx.files() files = ctx.files()
if writechangesetcopy:
filesadded = ctx.filesadded()
filesremoved = ctx.filesremoved()
elif ctx.files(): elif ctx.files():
m1ctx = p1.manifestctx() m1ctx = p1.manifestctx()
m2ctx = p2.manifestctx() m2ctx = p2.manifestctx()
mctx = m1ctx.copy() mctx = m1ctx.copy()
m = mctx.read() m = mctx.read()
m1 = m1ctx.read() m1 = m1ctx.read()
m2 = m2ctx.read() m2 = m2ctx.read()
# ignoring things outside of narrowspec in most cases. The # ignoring things outside of narrowspec in most cases. The
# one case where we might have files outside the narrowspec # one case where we might have files outside the narrowspec
# at this point is merges, and we already error out in the # at this point is merges, and we already error out in the
# case where the merge has files outside of the narrowspec, # case where the merge has files outside of the narrowspec,
# so this is safe. # so this is safe.
mn = mctx.write(trp, linkrev, mn = mctx.write(trp, linkrev,
p1.manifestnode(), p2.manifestnode(), p1.manifestnode(), p2.manifestnode(),
added, drop, match=self.narrowmatch()) added, drop, match=self.narrowmatch())
if writechangesetcopy:
filesadded = [f for f in changed
if not (f in m1 or f in m2)]
filesremoved = removed
else: else:
self.ui.debug('reusing manifest from p1 (listed files ' self.ui.debug('reusing manifest from p1 (listed files '
'actually unchanged)\n') 'actually unchanged)\n')
mn = p1.manifestnode() mn = p1.manifestnode()
else: else:
self.ui.debug('reusing manifest from p1 (no file change)\n') self.ui.debug('reusing manifest from p1 (no file change)\n')
mn = p1.manifestnode() mn = p1.manifestnode()
files = [] files = []
if writecopiesto == 'changeset-only': if writecopiesto == 'changeset-only':
# If writing only to changeset extras, use None to indicate that # If writing only to changeset extras, use None to indicate that
# no entry should be written. If writing to both, write an empty # no entry should be written. If writing to both, write an empty
# entry to prevent the reader from falling back to reading # entry to prevent the reader from falling back to reading
# filelogs. # filelogs.
p1copies = p1copies or None p1copies = p1copies or None
p2copies = p2copies or None p2copies = p2copies or None
filesadded = filesadded or None
filesremoved = filesremoved or None
# update changelog # update changelog
self.ui.note(_("committing changelog\n")) self.ui.note(_("committing changelog\n"))
self.changelog.delayupdate(tr) self.changelog.delayupdate(tr)
n = self.changelog.add(mn, files, ctx.description(), n = self.changelog.add(mn, files, ctx.description(),
trp, p1.node(), p2.node(), trp, p1.node(), p2.node(),
user, ctx.date(), ctx.extra().copy(), user, ctx.date(), ctx.extra().copy(),
p1copies, p2copies) p1copies, p2copies, filesadded, filesremoved)
xp1, xp2 = p1.hex(), p2 and p2.hex() or '' xp1, xp2 = p1.hex(), p2 and p2.hex() or ''
self.hook('pretxncommit', throw=True, node=hex(n), parent1=xp1, self.hook('pretxncommit', throw=True, node=hex(n), parent1=xp1,
parent2=xp2) parent2=xp2)
# set the new commit is proper phase # set the new commit is proper phase
targetphase = subrepoutil.newcommitphase(self.ui, ctx) targetphase = subrepoutil.newcommitphase(self.ui, ctx)
if targetphase: if targetphase:
# retract boundary do not alter parent changeset. # retract boundary do not alter parent changeset.
# if a parent have higher the resulting phase will # if a parent have higher the resulting phase will

tests/test-copies-in-changeset.t

$ cat >> $HGRCPATH << EOF $ cat >> $HGRCPATH << EOF
> [experimental] > [experimental]
> copies.write-to=changeset-only > copies.write-to=changeset-only
> copies.read-from=changeset-only > copies.read-from=changeset-only
> [alias] > [alias]
> changesetcopies = log -r . -T 'files: {files} > changesetcopies = log -r . -T 'files: {files}
> {extras % "{ifcontains("files", key, "{key}: {value}\n")}"}
> {extras % "{ifcontains("copies", key, "{key}: {value}\n")}"}' > {extras % "{ifcontains("copies", key, "{key}: {value}\n")}"}'
> showcopies = log -r . -T '{file_copies % "{source} -> {name}\n"}' > showcopies = log -r . -T '{file_copies % "{source} -> {name}\n"}'
> [extensions] > [extensions]
> rebase = > rebase =
> EOF > EOF
Check that copies are recorded correctly Check that copies are recorded correctly
$ hg init repo $ hg init repo
$ cd repo $ cd repo
$ echo a > a $ echo a > a
$ hg add a $ hg add a
$ hg ci -m initial $ hg ci -m initial
$ hg cp a b $ hg cp a b
$ hg cp a c $ hg cp a c
$ hg cp a d $ hg cp a d
$ hg ci -m 'copy a to b, c, and d' $ hg ci -m 'copy a to b, c, and d'
$ hg changesetcopies $ hg changesetcopies
files: b c d files: b c d
filesadded: 0\x001\x002 (esc)
p1copies: b\x00a (esc) p1copies: b\x00a (esc)
c\x00a (esc) c\x00a (esc)
d\x00a (esc) d\x00a (esc)
$ hg showcopies $ hg showcopies
a -> b a -> b
a -> c a -> c
a -> d a -> d
$ hg showcopies --config experimental.copies.read-from=compatibility $ hg showcopies --config experimental.copies.read-from=compatibility
a -> b a -> b
a -> c a -> c
a -> d a -> d
$ hg showcopies --config experimental.copies.read-from=filelog-only $ hg showcopies --config experimental.copies.read-from=filelog-only
Check that renames are recorded correctly Check that renames are recorded correctly
$ hg mv b b2 $ hg mv b b2
$ hg ci -m 'rename b to b2' $ hg ci -m 'rename b to b2'
$ hg changesetcopies $ hg changesetcopies
files: b b2 files: b b2
filesadded: 1
filesremoved: 0
p1copies: b2\x00b (esc) p1copies: b2\x00b (esc)
$ hg showcopies $ hg showcopies
b -> b2 b -> b2
Rename onto existing file. This should get recorded in the changeset files list and in the extras, Rename onto existing file. This should get recorded in the changeset files list and in the extras,
even though there is no filelog entry. even though there is no filelog entry.
$ hg cp b2 c --force $ hg cp b2 c --force
$ hg st --copies $ hg st --copies
M c M c
b2 b2
$ hg debugindex c $ hg debugindex c
rev linkrev nodeid p1 p2 rev linkrev nodeid p1 p2
0 1 b789fdd96dc2 000000000000 000000000000 0 1 b789fdd96dc2 000000000000 000000000000
$ hg ci -m 'move b onto d' $ hg ci -m 'move b onto d'
$ hg changesetcopies $ hg changesetcopies
files: c files: c
p1copies: c\x00b2 (esc) p1copies: c\x00b2 (esc)
$ hg showcopies $ hg showcopies
b2 -> c b2 -> c
$ hg debugindex c $ hg debugindex c
rev linkrev nodeid p1 p2 rev linkrev nodeid p1 p2
0 1 b789fdd96dc2 000000000000 000000000000 0 1 b789fdd96dc2 000000000000 000000000000
Create a merge commit with copying done during merge. Create a merge commit with copying done during merge.
$ hg cp a g $ hg cp a g
File 'd' exists only in p2, so 'h' should be from p2 File 'd' exists only in p2, so 'h' should be from p2
$ hg cp d h $ hg cp d h
File 'f' exists only in p1, so 'i' should be from p1 File 'f' exists only in p1, so 'i' should be from p1
$ hg cp f i $ hg cp f i
$ hg ci -m 'merge' $ hg ci -m 'merge'
$ hg changesetcopies $ hg changesetcopies
files: g h i files: g h i
filesadded: 0\x001\x002 (esc)
p1copies: g\x00a (esc) p1copies: g\x00a (esc)
i\x00f (esc) i\x00f (esc)
p2copies: h\x00d (esc) p2copies: h\x00d (esc)
$ hg showcopies $ hg showcopies
a -> g a -> g
d -> h d -> h
f -> i f -> i
Test writing to both changeset and filelog Test writing to both changeset and filelog
$ hg cp a j $ hg cp a j
$ hg ci -m 'copy a to j' --config experimental.copies.write-to=compatibility $ hg ci -m 'copy a to j' --config experimental.copies.write-to=compatibility
$ hg changesetcopies $ hg changesetcopies
files: j files: j
filesadded: 0
filesremoved:
p1copies: j\x00a (esc) p1copies: j\x00a (esc)
p2copies: p2copies:
$ hg debugdata j 0 $ hg debugdata j 0
\x01 (esc) \x01 (esc)
copy: a copy: a
copyrev: b789fdd96dc2f3bd229c1dd8eedf0fc60e2b68e3 copyrev: b789fdd96dc2f3bd229c1dd8eedf0fc60e2b68e3
\x01 (esc) \x01 (esc)
a a
$ hg showcopies $ hg showcopies
a -> j a -> j
$ hg showcopies --config experimental.copies.read-from=compatibility $ hg showcopies --config experimental.copies.read-from=compatibility
a -> j a -> j
$ hg showcopies --config experimental.copies.read-from=filelog-only $ hg showcopies --config experimental.copies.read-from=filelog-only
a -> j a -> j
The entries should be written to extras even if they're empty (so the client The entries should be written to extras even if they're empty (so the client
won't have to fall back to reading from filelogs) won't have to fall back to reading from filelogs)
$ echo x >> j $ echo x >> j
$ hg ci -m 'modify j' --config experimental.copies.write-to=compatibility $ hg ci -m 'modify j' --config experimental.copies.write-to=compatibility
$ hg changesetcopies $ hg changesetcopies
files: j files: j
filesadded:
filesremoved:
p1copies: p1copies:
p2copies: p2copies:
Test writing only to filelog Test writing only to filelog
$ hg cp a k $ hg cp a k
$ hg ci -m 'copy a to k' --config experimental.copies.write-to=filelog-only $ hg ci -m 'copy a to k' --config experimental.copies.write-to=filelog-only
$ hg changesetcopies $ hg changesetcopies
files: k files: k
$ hg debugdata k 0 $ hg debugdata k 0
\x01 (esc) \x01 (esc)
copy: a copy: a
copyrev: b789fdd96dc2f3bd229c1dd8eedf0fc60e2b68e3 copyrev: b789fdd96dc2f3bd229c1dd8eedf0fc60e2b68e3
\x01 (esc) \x01 (esc)
a a
$ hg showcopies $ hg showcopies
$ hg showcopies --config experimental.copies.read-from=compatibility $ hg showcopies --config experimental.copies.read-from=compatibility
$ echo a > a $ echo a > a
$ hg ci -Aqm 'add a' $ hg ci -Aqm 'add a'
$ echo a2 > a $ echo a2 > a
$ hg ci -m 'modify a' $ hg ci -m 'modify a'
$ hg co -q 0 $ hg co -q 0
$ hg mv a b $ hg mv a b
$ hg ci -qm 'rename a to b' $ hg ci -qm 'rename a to b'
$ hg rebase -d 1 --config rebase.experimental.inmemory=yes $ hg rebase -d 1 --config rebase.experimental.inmemory=yes
rebasing 2:55d0b405c1b2 "rename a to b" (tip) rebasing 2:acfc33f3aa6d "rename a to b" (tip)
merging a and b to b merging a and b to b
saved backup bundle to $TESTTMP/rebase-rename/.hg/strip-backup/55d0b405c1b2-78df867e-rebase.hg saved backup bundle to $TESTTMP/rebase-rename/.hg/strip-backup/acfc33f3aa6d-81d0180d-rebase.hg
$ hg st --change . --copies $ hg st --change . --copies
A b A b
a a
R a R a
$ cd .. $ cd ..

tests/test-copies.t

| b | b
o 0 added a o 0 added a
a a
Grafting revision 4 on top of revision 2, showing that it respect the rename: Grafting revision 4 on top of revision 2, showing that it respect the rename:
$ hg up 2 -q $ hg up 2 -q
$ hg graft -r 4 --base 3 --hidden $ hg graft -r 4 --base 3 --hidden
grafting 4:af28412ec03c "added d, modified b" (tip) grafting 4:af28412ec03c "added d, modified b" (tip) (no-changeset !)
grafting 4:6325ca0b7a1c "added d, modified b" (tip) (changeset !)
merging b1 and b to b1 merging b1 and b to b1
$ hg l -l1 -p $ hg l -l1 -p
@ 5 added d, modified b @ 5 added d, modified b
| b1 | b1
~ diff -r 5a4825cc2926 -r 94a2f1a0e8e2 b1 (no-changeset !) ~ diff -r 5a4825cc2926 -r 94a2f1a0e8e2 b1 (no-changeset !)
~ diff -r f5474f5023a8 -r ef7c02d69f3d b1 (changeset !) ~ diff -r df722b7fe2d5 -r ba3ddbbdfd96 b1 (changeset !)
--- a/b1 Thu Jan 01 00:00:00 1970 +0000 --- a/b1 Thu Jan 01 00:00:00 1970 +0000
+++ b/b1 Thu Jan 01 00:00:00 1970 +0000 +++ b/b1 Thu Jan 01 00:00:00 1970 +0000
@@ -1,1 +1,2 @@ @@ -1,1 +1,2 @@
b b
+baba +baba
Test to make sure that fullcopytracing algorithm doesn't fail when neither of the Test to make sure that fullcopytracing algorithm doesn't fail when neither of the
merging csets is a descendant of the base. merging csets is a descendant of the base.
| o 2 added willconflict | o 2 added willconflict
|/ willconflict |/ willconflict
o 1 added b o 1 added b
| b | b
o 0 added a o 0 added a
a a
$ hg rebase -r . -d 2 -t :other $ hg rebase -r . -d 2 -t :other
rebasing 5:5018b1509e94 "added willconflict and d" (tip) rebasing 5:5018b1509e94 "added willconflict and d" (tip) (no-changeset !)
rebasing 5:619047c26bf8 "added willconflict and d" (tip) (changeset !)
$ hg up 3 -q $ hg up 3 -q
$ hg l --hidden $ hg l --hidden
o 6 added willconflict and d o 6 added willconflict and d
| d willconflict | d willconflict
| x 5 added willconflict and d | x 5 added willconflict and d
| | d willconflict | | d willconflict
| | x 4 added d | | x 4 added d
| |/ d | |/ d
+---@ 3 added c +---@ 3 added c
| | c | | c
o | 2 added willconflict o | 2 added willconflict
|/ willconflict |/ willconflict
o 1 added b o 1 added b
| b | b
o 0 added a o 0 added a
a a
Now if we trigger a merge between revision 3 and 6 using base revision 4, Now if we trigger a merge between revision 3 and 6 using base revision 4,
neither of the merging csets will be a descendant of the base revision: neither of the merging csets will be a descendant of the base revision:
$ hg graft -r 6 --base 4 --hidden -t :other $ hg graft -r 6 --base 4 --hidden -t :other
grafting 6:99802e4f1e46 "added willconflict and d" (tip) grafting 6:99802e4f1e46 "added willconflict and d" (tip) (no-changeset !)
grafting 6:9ddc6fb3b691 "added willconflict and d" (tip) (changeset !)