This is an archive of the discontinued Mercurial Phabricator instance.

Differential D10715

revlog: make transaction handle revlog splits better (issue6423)
AbandonedPublic

Authored by valentin.gatienbaron on May 16 2021, 3:31 PM.

Details

Reviewers
indygreg
Group Reviewers
hg-reviewers
Summary

Currently, any transaction that includes a revlog split and does not
successfully commit prevents any further write to the repository
(short of performing manual surgery), because it leaves behind a
transaction that hg doesn't know how to rollback.

This teaches hg how to handle such transactions. Specifically, any
interruption outside of a tight window (rename + two writes) results
in transactions that can be rolled back correctly. An interruption in
that window inside that window leaves a broken repository, similarly
to before this change.

One awkward thing is that if an old hg version leaves behind one of
these transactions it can't rollback, an hg with this change would
manage to roll it back, but wrongly (it would end up with non inline
revlog and .d file of size 0). That seems acceptable to me, because
this is a strange scenario (and either way, the solution is to
reclone).

Another awkward thing is that the fncache fails to be updated. This
seems acceptable, because it's not used much, and I'll take the
occcasional call to hg debugrebuildfncache over a broken repository.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Revision Contents
Changeset List

Diff 27957

mercurial/revlog.py

return return
troffset = tr.findoffset(self._indexfile) troffset = tr.findoffset(self._indexfile)
if troffset is None: if troffset is None:
raise error.RevlogError( raise error.RevlogError(
_(b"%s not found in the transaction") % self._indexfile _(b"%s not found in the transaction") % self._indexfile
) )
trindex = 0 trindex = 0
tr.add(self._datafile, 0) trdataoffset = 0
if fp: if fp:
fp.flush() fp.flush()
fp.close() fp.close()
# We can't use the cached file handle after close(). So prevent # We can't use the cached file handle after close(). So prevent
# its usage. # its usage.
self._writinghandles = None self._writinghandles = None
with self._indexfp(b'r') as ifh, self._datafp(b'w') as dfh: with self._indexfp(b'r') as ifh, self._datafp(b'w') as dfh:
if dfh.tell() != 0:
# Should never happen, but may reduce confusion if something
# went wrong in a rollback or manual fix.
raise error.RevlogError(
_(b"%s is an inline revlog but %s is nonempty")
% (self._indexfile, self._datafile)
)
# write this after the dfh.tell check, otherwise the rollback caused
# by a failure of the check would truncate whatever is the datafile
tr.add(self._datafile, 0)
for r in self: for r in self:
dfh.write(self._getsegmentforrevs(r, r, df=ifh)[1]) dfh.write(self._getsegmentforrevs(r, r, df=ifh)[1])
if troffset <= self.start(r) + r * self.index.entry_size: if troffset <= self.start(r) + r * self.index.entry_size:
# trindex is the first revision to be excluded on rollback
trindex = r trindex = r
trdataoffset = self.start(r)
with self._indexfp(b'w') as fp: with self._indexfp(b'w') as fp:
self._format_flags &= ~FLAG_INLINE_DATA self._format_flags &= ~FLAG_INLINE_DATA
self._inline = False self._inline = False
for i in self: for i in self:
e = self.index.entry_binary(i) e = self.index.entry_binary(i)
if i == 0: if i == 0:
header = self._format_flags | self._format_version header = self._format_flags | self._format_version
header = self.index.pack_header(header) header = self.index.pack_header(header)
e = header + e e = header + e
fp.write(e) fp.write(e)
# the temp file replace the real index when we exit the context # The temp file replace the real index when we exit the context
# manager # manager. When that happens, transaction rollback is temporarily
# broken, because the rollback journal entry for the old .i is
# larger than the size of the new .i (because they include the same
# revisions, but offset for the old .i accounts for the inline
# data. Except when the data is of size 0, but that should be both
# impossible and harmless). This gets corrected very immediately by
# the tr.replace below. Note that we *must* write the offset for the
# (new) datafile before the offset for the new .i. In this order, if
# we get interrupted after the first tr.replace, the rollback
# fails. In the other order, if we got interrupted after the
# tr.replace for the index, rollback would succeed but leave the
# repository broken (non-inline index + datafile of size 0). One
# problem we still have is that since this change is done outside
# the transaction, the fncache should get updated even on
# transaction rollback, but doesn't.
tr.replace(self._datafile, trdataoffset)
tr.replace(self._indexfile, trindex * self.index.entry_size) tr.replace(self._indexfile, trindex * self.index.entry_size)
nodemaputil.setup_persistent_nodemap(tr, self) nodemaputil.setup_persistent_nodemap(tr, self)
self._chunkclear() self._chunkclear()
def _nodeduplicatecallback(self, transaction, node): def _nodeduplicatecallback(self, transaction, node):
"""called when trying to add a node already stored.""" """called when trying to add a node already stored."""
def addrevision( def addrevision(

mercurial/transaction.py

raise error.ProgrammingError( raise error.ProgrammingError(
b'cannot use transaction when it is already committed/aborted' b'cannot use transaction when it is already committed/aborted'
) )
return func(self, *args, **kwds) return func(self, *args, **kwds)
return _active return _active
def _entries_to_playback(entries):
# On revlog split, we end up with two journal entries for the .i
# and .d. In these cases we want the second entries to override
# the first ones. This filters out the first entries.
usable_entries = []
seen_f = {}
for f, o in reversed(entries):
if f in seen_f:
if seen_f[f] != 1:
raise error.Abort(
_(b"unexpectedly many rollback journal entries for %s") % f
)
seen_f[f] = 2
else:
usable_entries.append((f, o))
seen_f[f] = 1
usable_entries.reverse()
return usable_entries
def _playback( def _playback(
journal, journal,
report, report,
opener, opener,
vfsmap, vfsmap,
entries, entries,
backupentries, backupentries,
unlink=True, unlink=True,
checkambigfiles=None, checkambigfiles=None,
): ):
for f, o in entries: for f, o in _entries_to_playback(entries):
if o or not unlink: if o or not unlink:
checkambig = checkambigfiles and (f, b'') in checkambigfiles checkambig = checkambigfiles and (f, b'') in checkambigfiles
try: try:
fp = opener(f, b'a', checkambig=checkambig) fp = opener(f, b'a', checkambig=checkambig)
if fp.tell() < o: if fp.tell() < o:
raise error.Abort( raise error.Abort(
_( _(
b"attempted to truncate %s to %d bytes, but it was " b"attempted to truncate %s to %d bytes, but it was "

tests/test-transaction-rollback-on-revlog-split.t

> EOF > EOF
>>> import os >>> import os
>>> open('file', 'wb').write(os.urandom(300000)) >>> open('file', 'wb').write(os.urandom(300000))
300000 300000
$ hg commit -m_ $ hg commit -m_
transaction abort! transaction abort!
rollback failed - please run hg recover rollback completed
(failure reason: attempted to truncate data/file.i to 1065 bytes, but it was already 128 bytes)
abort: pretxncommit hook exited with status 1 abort: pretxncommit hook exited with status 1
[40] [40]
$ cat .hg/store/journal | tr -s '\000' ' ' Verify that rollback worked properly (the fncache being stale is a
data/file.i 1065 bug):
data/file.d 0
data/file.i 64 $ hg verify -q
00manifest.i 111 warning: revlog 'data/file.d' not in fncache!
00changelog.i 122 1 warnings encountered!
hint: run "hg debugrebuildfncache" to recover from corrupt fncache
$ hg recover $ hg debugrebuildfncache
rolling back interrupted transaction adding data/file.d
abort: attempted to truncate data/file.i to 1065 bytes, but it was already 128 bytes 1 items added, 0 removed from fncache
[255] $ hg verify -q