( )⚙ D11236 streamclone: ensure the server sends the right amount of data

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D11236

streamclone: ensure the server sends the right amount of data
ClosedPublic

Authored by valentin.gatienbaron on Aug 1 2021, 10:59 AM.

Details

Reviewers
pulkit
Group Reviewers
hg-reviewers
Commits
rHG48f07adbda98: streamclone: ensure the server sends the right amount of data
Summary

Otherwise, the client would fail with some confusing error. I have
seen an error which I think is this, perhaps due to a concurrent
revlog split, which streamclones do not handle correctly and would
result in a short read of the index of the revlog being split.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

pulkit accepted this revision.Aug 1 2021, 3:02 PM
This revision is now accepted and ready to land.Aug 1 2021, 3:02 PM
Alphare added a subscriber: Alphare.Aug 2 2021, 5:00 AM

@pulkit I think this would be a good contender for stable and should go in during the freeze.

Revision Contents
Changeset List

PathPackages
Mmercurial/streamclone.py (18 lines)

Diff 29769

mercurial/streamclone.py

progress = repo.ui.makeprogress( progress = repo.ui.makeprogress(
_(b'bundle'), total=totalfilesize, unit=_(b'bytes') _(b'bundle'), total=totalfilesize, unit=_(b'bytes')
) )
progress.update(0) progress.update(0)
with maketempcopies() as copy, progress: with maketempcopies() as copy, progress:
# copy is delayed until we are in the try # copy is delayed until we are in the try
entries = [_filterfull(e, copy, vfsmap) for e in entries] entries = [_filterfull(e, copy, vfsmap) for e in entries]
yield None # this release the lock on the repository yield None # this release the lock on the repository
seen = 0 totalbytecount = 0
for src, name, ftype, data in entries: for src, name, ftype, data in entries:
vfs = vfsmap[src] vfs = vfsmap[src]
yield src yield src
yield util.uvarintencode(len(name)) yield util.uvarintencode(len(name))
if ftype == _fileappend: if ftype == _fileappend:
fp = vfs(name) fp = vfs(name)
size = data size = data
elif ftype == _filefull: elif ftype == _filefull:
fp = open(data, b'rb') fp = open(data, b'rb')
size = util.fstat(fp).st_size size = util.fstat(fp).st_size
bytecount = 0
try: try:
yield util.uvarintencode(size) yield util.uvarintencode(size)
yield name yield name
if size <= 65536: if size <= 65536:
chunks = (fp.read(size),) chunks = (fp.read(size),)
else: else:
chunks = util.filechunkiter(fp, limit=size) chunks = util.filechunkiter(fp, limit=size)
for chunk in chunks: for chunk in chunks:
seen += len(chunk) bytecount += len(chunk)
progress.update(seen) totalbytecount += len(chunk)
progress.update(totalbytecount)
yield chunk yield chunk
if bytecount != size:
# Would most likely be caused by a race due to `hg strip` or
# a revlog split
raise error.Abort(
_(
b'clone could only read %d bytes from %s, but '
b'expected %d bytes'
)
% (bytecount, name, size)
)
finally: finally:
fp.close() fp.close()
def _test_sync_point_walk_1(repo): def _test_sync_point_walk_1(repo):
"""a function for synchronisation during tests""" """a function for synchronisation during tests"""