This is an archive of the discontinued Mercurial Phabricator instance.

Differential D81

infinitepush: re-enable lfs support
ClosedPublic

Authored by stash on Jul 14 2017, 6:45 AM.
Tags
None
Subscribers
durham

Details

Reviewers
durham
Group Reviewers
Restricted Project
Commits
rFBHGX501c3c98bbc9: infinitepush: re-enable lfs support
Summary

D17 was reverted because it broke creating bundle for non-general delta repos.
The reason is the following: D17 made infinitepush extension override
changegroup.supportedoutgoingversion() function and discard '01' version.
For non-general delta repositories it resulted in broken hg bundle ...
command.
abort: repository does not support bundle version 01

This diff fixes it by not overriding supportedoutgoingversion(). Instead
getscratchbranchpart() has its own logic of selecting changegroup version.

Test Plan

Run unit-test
Run hg bundle -r . --base .^ somefile in non-general delta repo,
make sure it works fine.

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

stash created this revision.Jul 14 2017, 6:45 AM
durham accepted this revision.Jul 14 2017, 3:39 PM
durham added a subscriber: durham.
durham added inline comments.
tests/library-infinitepush.sh
44 ↗(On Diff #131)

Does this mean the tests are testing the case where changegroup3 is disabled? Seems like we want that coverage since our normal devserver/hg server communication doesn't use changegroup3 right?

This revision is now accepted and ready to land.Jul 14 2017, 3:39 PM
stash updated this revision to Diff 202.Jul 17 2017, 8:24 AM

addressed comments

Closed by commit rFBHGX501c3c98bbc9: infinitepush: re-enable lfs support (authored by stash). · Explain WhyJul 20 2017, 6:44 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 339

hgext3rd/lfs/wrapper.py

for name in ['lfslocalblobstore', 'lfsremoteblobstore']: for name in ['lfslocalblobstore', 'lfsremoteblobstore']:
if util.safehasattr(othervfs, name): if util.safehasattr(othervfs, name):
setattr(self, name, getattr(othervfs, name)) setattr(self, name, getattr(othervfs, name))
def _canskipupload(repo): def _canskipupload(repo):
# if remotestore is a null store, upload is a no-op and can be skipped # if remotestore is a null store, upload is a no-op and can be skipped
return isinstance(repo.svfs.lfsremoteblobstore, blobstore._nullremote) return isinstance(repo.svfs.lfsremoteblobstore, blobstore._nullremote)
def uploadblobsfromrevs(repo, revs):
'''upload lfs blobs introduced by revs
Note: also used by other extensions e. g. infinitepush. avoid renaming.
'''
if _canskipupload(repo):
return
pointers = extractpointers(repo, revs)
uploadblobs(repo, pointers)
def prepush(pushop): def prepush(pushop):
"""Prepush hook. """Prepush hook.
Read through the revisions to push, looking for filelog entries that can be Read through the revisions to push, looking for filelog entries that can be
deserialized into metadata so that we can block the push on their upload to deserialized into metadata so that we can block the push on their upload to
the remote blobstore. the remote blobstore.
""" """
if _canskipupload(pushop.repo): return uploadblobsfromrevs(pushop.repo, pushop.outgoing.missing)
return
pointers = extractpointers(pushop.repo, pushop.outgoing.missing)
uploadblobs(pushop.repo, pointers)
def writenewbundle(orig, ui, repo, source, filename, bundletype, outgoing, def writenewbundle(orig, ui, repo, source, filename, bundletype, outgoing,
*args, **kwargs): *args, **kwargs):
"""upload LFS blobs added by outgoing revisions on 'hg bundle'""" """upload LFS blobs added by outgoing revisions on 'hg bundle'"""
if _canskipupload(repo): uploadblobsfromrevs(repo, outgoing.missing)
return
pointers = extractpointers(repo, outgoing.missing)
uploadblobs(repo, pointers)
return orig(ui, repo, source, filename, bundletype, outgoing, *args, return orig(ui, repo, source, filename, bundletype, outgoing, *args,
**kwargs) **kwargs)
def extractpointers(repo, revs): def extractpointers(repo, revs):
"""return a list of lfs pointers added by given revs""" """return a list of lfs pointers added by given revs"""
ui = repo.ui ui = repo.ui
if ui.debugflag: if ui.debugflag:
ui.write(_('lfs: computing set of blobs to upload\n')) ui.write(_('lfs: computing set of blobs to upload\n'))

infinitepush/bundleparts.py

# Copyright 2017 Facebook, Inc. # Copyright 2017 Facebook, Inc.
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
from .common import ( from .common import (
encodebookmarks, encodebookmarks,
isremotebooksenabled, isremotebooksenabled,
) )
from mercurial import ( from mercurial import (
bundle2, bundle2,
changegroup, changegroup,
error, error,
extensions,
revsetlang, revsetlang,
) )
from mercurial.i18n import _ from mercurial.i18n import _
scratchbranchparttype = 'b2x:infinitepush' scratchbranchparttype = 'b2x:infinitepush'
scratchbookmarksparttype = 'b2x:infinitepushscratchbookmarks' scratchbookmarksparttype = 'b2x:infinitepushscratchbookmarks'
def getscratchbranchpart(repo, peer, outgoing, confignonforwardmove, def getscratchbranchpart(repo, peer, outgoing, confignonforwardmove,
ui, bookmark, create): ui, bookmark, create):
if not outgoing.missing: if not outgoing.missing:
raise error.Abort(_('no commits to push')) raise error.Abort(_('no commits to push'))
if scratchbranchparttype not in bundle2.bundle2caps(peer): if scratchbranchparttype not in bundle2.bundle2caps(peer):
raise error.Abort(_('no server support for %r') % scratchbranchparttype) raise error.Abort(_('no server support for %r') % scratchbranchparttype)
_validaterevset(repo, revsetlang.formatspec('%ln', outgoing.missing), _validaterevset(repo, revsetlang.formatspec('%ln', outgoing.missing),
bookmark) bookmark)
cgversion = '02' supportedversions = changegroup.supportedoutgoingversions(repo)
# Explicitly avoid using '01' changegroup version in infinitepush to
# support general delta
supportedversions.discard('01')
cgversion = min(supportedversions)
_handlelfs(repo, outgoing.missing)
cg = changegroup.getlocalchangegroupraw(repo, 'push', cg = changegroup.getlocalchangegroupraw(repo, 'push',
outgoing, version=cgversion) outgoing, version=cgversion)
params = {} params = {}
params['cgversion'] = cgversion params['cgversion'] = cgversion
if bookmark: if bookmark:
params['bookmark'] = bookmark params['bookmark'] = bookmark
# 'prevbooknode' is necessary for pushkey reply part # 'prevbooknode' is necessary for pushkey reply part
if not repo.revs(revset): if not repo.revs(revset):
raise error.Abort(_('nothing to push')) raise error.Abort(_('nothing to push'))
if bookmark: if bookmark:
# Allow bundle with many heads only if no bookmark is specified # Allow bundle with many heads only if no bookmark is specified
heads = repo.revs('heads(%r)', revset) heads = repo.revs('heads(%r)', revset)
if len(heads) > 1: if len(heads) > 1:
raise error.Abort( raise error.Abort(
_('cannot push more than one head to a scratch branch')) _('cannot push more than one head to a scratch branch'))
def _handlelfs(repo, missing):
'''Special case if lfs is enabled
If lfs is enabled then we need to call prepush hook
to make sure large files are uploaded to lfs
'''
try:
lfsmod = extensions.find('lfs')
lfsmod.wrapper.uploadblobsfromrevs(repo, missing)
except KeyError:
# Ignore if lfs extension is not enabled
return

tests/test-infinitepush-lfs.t

  • This file was added.
Setup common infinitepush
$ . "$TESTDIR/library.sh"
$ . "$TESTDIR/library-infinitepush.sh"
$ setupcommon
Setup lfs
$ cat >> $HGRCPATH << EOF
> [experimental]
> changegroup3=True
> [extensions]
> lfs=$TESTDIR/../hgext3rd/lfs/
> [lfs]
> threshold=10B
> url=file:$TESTTMP/dummy-remote/
> EOF
Setup server repo
$ hg init repo
$ cd repo
$ setupserver
$ echo 1 > 1
$ hg add 1
$ hg ci -m initial
Setup client
$ cd ..
$ hg clone ssh://user@dummy/repo client -q
$ cd client
$ echo aaaaaaaaaaa > largefile
$ hg ci -Aqm commit
$ hg debugdata largefile 0
version https://git-lfs.github.com/spec/v1
oid sha256:ab483e1d855ad0ea27a68eeea02a04c1de6ccd2dc2c05e3a48c9a1ebb8af5f99
size 12
x-is-binary 0
$ hg push -r . --to scratch/lfscommit --create
pushing to ssh://user@dummy/repo
searching for changes
remote: pushing 1 commit:
remote: 0da81a72db1a commit
$ scratchbookmarks
scratch/lfscommit 0da81a72db1a2d8256845e3808971f33e73d24c4
$ cd ..
Setup another client
$ hg clone ssh://user@dummy/repo client2 -q
$ cd client2
$ hg update scratch/lfscommit
'scratch/lfscommit' does not exist locally - looking for it remotely...
pulling from ssh://user@dummy/repo
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
(run 'hg update' to get a working copy)
'scratch/lfscommit' found remotely
1 files updated, 0 files merged, 0 files removed, 0 files unresolved
(activating bookmark scratch/lfscommit)