( )⚙ D11236 streamclone: ensure the server sends the right amount of data

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D11236

streamclone: ensure the server sends the right amount of data
ClosedPublic

Authored by valentin.gatienbaron on Aug 1 2021, 10:59 AM.

Details

Reviewers
pulkit
Group Reviewers
hg-reviewers
Commits
rHG48f07adbda98: streamclone: ensure the server sends the right amount of data
Summary

Otherwise, the client would fail with some confusing error. I have
seen an error which I think is this, perhaps due to a concurrent
revlog split, which streamclones do not handle correctly and would
result in a short read of the index of the revlog being split.

Diff Detail

Repository
rHG Mercurial
Branch
default
Lint
No Linters Available
Unit
No Unit Test Coverage

Event Timeline

pulkit accepted this revision.Aug 1 2021, 3:02 PM
This revision is now accepted and ready to land.Aug 1 2021, 3:02 PM
Alphare added a subscriber: Alphare.Aug 2 2021, 5:00 AM

@pulkit I think this would be a good contender for stable and should go in during the freeze.

Revision Contents
Changeset List

PathPackages
Mmercurial/streamclone.py (18 lines)

Diff 29766

mercurial/streamclone.py

progress = repo.ui.makeprogress( progress = repo.ui.makeprogress(
_(b'bundle'), total=totalfilesize, unit=_(b'bytes') _(b'bundle'), total=totalfilesize, unit=_(b'bytes')
) )
progress.update(0) progress.update(0)
with maketempcopies() as copy, progress: with maketempcopies() as copy, progress:
# copy is delayed until we are in the try # copy is delayed until we are in the try
entries = [_filterfull(e, copy, vfsmap) for e in entries] entries = [_filterfull(e, copy, vfsmap) for e in entries]
yield None # this release the lock on the repository yield None # this release the lock on the repository
seen = 0 totalbytecount = 0
for src, name, ftype, data in entries: for src, name, ftype, data in entries:
vfs = vfsmap[src] vfs = vfsmap[src]
yield src yield src
yield util.uvarintencode(len(name)) yield util.uvarintencode(len(name))
if ftype == _fileappend: if ftype == _fileappend:
fp = vfs(name) fp = vfs(name)
size = data size = data
elif ftype == _filefull: elif ftype == _filefull:
fp = open(data, b'rb') fp = open(data, b'rb')
size = util.fstat(fp).st_size size = util.fstat(fp).st_size
bytecount = 0
try: try:
yield util.uvarintencode(size) yield util.uvarintencode(size)
yield name yield name
if size <= 65536: if size <= 65536:
chunks = (fp.read(size),) chunks = (fp.read(size),)
else: else:
chunks = util.filechunkiter(fp, limit=size) chunks = util.filechunkiter(fp, limit=size)
for chunk in chunks: for chunk in chunks:
seen += len(chunk) bytecount += len(chunk)
progress.update(seen) totalbytecount += len(chunk)
progress.update(totalbytecount)
yield chunk yield chunk
if bytecount != size:
# Would most likely be caused by a race due to `hg strip` or
# a revlog split
raise error.Abort(
_(
b'clone could only read %d bytes from %s, but '
b'expected %d bytes'
)
% (bytecount, name, size)
)
finally: finally:
fp.close() fp.close()
def _test_sync_point_walk_1(repo): def _test_sync_point_walk_1(repo):
"""a function for synchronisation during tests""" """a function for synchronisation during tests"""