This is an archive of the discontinued Mercurial Phabricator instance.

Differential D1511

remotefilelog: error gracefuly when the file is too big
AbandonedPublic

Authored by mitrandir on Nov 24 2017, 12:05 PM.
Tags
None
Subscribers
durham, mbthomas

Details

Reviewers
mbthomas
Group Reviewers
Restricted Project
Summary

We don't really support files larger than 2G with files larger than
4G remotefilelog crashes badly.

Test Plan

Tested with 4G file, doens't crash now. No tests as test would involve
creating such a large file.

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

mitrandir created this revision.Nov 24 2017, 12:05 PM
Herald added a reviewer: Restricted Project. · View Herald TranscriptNov 24 2017, 12:05 PM
mbthomas added a subscriber: mbthomas.Nov 24 2017, 12:24 PM

Can you test the logic by monkeypatching _maxentrysize to a smaller number?

mbthomas requested changes to this revision.Nov 24 2017, 12:25 PM
This revision now requires changes to proceed.Nov 24 2017, 12:25 PM
durham added a subscriber: durham.Nov 28 2017, 11:13 AM
durham added inline comments.
remotefilelog/remotefilelog.py
129

I think there's a util.bytecount() function that can turn a number of bytes into a human readable string (like 2.1GB)

mitrandir abandoned this revision.Feb 23 2018, 10:58 AM

Revision Contents
Changeset List

PathPackages
Mremotefilelog/remotefilelog.py (10 lines)

Diff 3837

remotefilelog/remotefilelog.py

fileserverclient, fileserverclient,
shallowutil, shallowutil,
) )
import collections, os import collections, os
from mercurial.node import bin, nullid from mercurial.node import bin, nullid
from mercurial import filelog, revlog, mdiff, ancestor, error from mercurial import filelog, revlog, mdiff, ancestor, error
from mercurial.i18n import _ from mercurial.i18n import _
# corresponds to uncompressed length of revlog's indexformatng (2 gigs, 4-byte
# signed integer)
_maxentrysize = 0x7fffffff
class remotefilelognodemap(object): class remotefilelognodemap(object):
def __init__(self, filename, store): def __init__(self, filename, store):
self._filename = filename self._filename = filename
self._store = store self._store = store
def __contains__(self, node): def __contains__(self, node):
missing = self._store.getmissing([(self._filename, node)]) missing = self._store.getmissing([(self._filename, node)])
return not bool(missing) return not bool(missing)
def addrevision(self, text, transaction, linknode, p1, p2, cachedelta=None, def addrevision(self, text, transaction, linknode, p1, p2, cachedelta=None,
node=None, flags=revlog.REVIDX_DEFAULT_FLAGS): node=None, flags=revlog.REVIDX_DEFAULT_FLAGS):
# text passed to "addrevision" includes hg filelog metadata header # text passed to "addrevision" includes hg filelog metadata header
if node is None: if node is None:
node = revlog.hash(text, p1, p2) node = revlog.hash(text, p1, p2)
meta, metaoffset = filelog.parsemeta(text) meta, metaoffset = filelog.parsemeta(text)
rawtext, validatehash = self._processflags(text, flags, 'write') rawtext, validatehash = self._processflags(text, flags, 'write')
if len(rawtext) > _maxentrysize:
raise revlog.RevlogError(
_("%s: size of %d bytes exceeds maximum size of 2GiB")
% (self.filename, len(rawtext)))
durhamUnsubmitted
Not Done

I think there's a util.bytecount() function that can turn a number of bytes into a human readable string (like 2.1GB)

durham: I think there's a util.bytecount() function that can turn a number of bytes into a human…
return self.addrawrevision(rawtext, transaction, linknode, p1, p2, return self.addrawrevision(rawtext, transaction, linknode, p1, p2,
node, flags, cachedelta, node, flags, cachedelta,
_metatuple=(meta, metaoffset)) _metatuple=(meta, metaoffset))
def addrawrevision(self, rawtext, transaction, linknode, p1, p2, node, def addrawrevision(self, rawtext, transaction, linknode, p1, p2, node,
flags, cachedelta=None, _metatuple=None): flags, cachedelta=None, _metatuple=None):
if _metatuple: if _metatuple:
# _metatuple: used by "addrevision" internally by remotefilelog # _metatuple: used by "addrevision" internally by remotefilelog