This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4765

fastcheckout: extension for obtaining a checkout quickly
AbandonedPublic

Authored by indygreg on Sep 26 2018, 5:44 PM.

Download Raw Diff

Details

Reviewers

None

Group Reviewers

hg-reviewers

Summary

One of the use cases for version control is in automated environments
that just want to obtain a working directory for a specific revision
as fast as possible. They don't necessarily care about repository
history or even the ability to commit to a local repo or do anything
beyond basic filesystem operations on the working directory. For these
consumers, a svn co equivalent local repo / working directory is
all they need.

This commit introduces an extension with the aim of fulfilling this
market niche. hg fastcheckout can be used to construct a working
directory against a revision in a remote repo, optionally purging
and cleaning the working directory along the way.

While this command might be useful as a general Mercurial feature,
the main reason I'm writing it is to provide a command where we can
start flushing out partial clone in core APIs and in end-user UI.
We already do some of this in support of the narrow extension. However,
narrow has a certain view of the world and I'm not convinced that its
view will align with the APIs available in the new wire protocol. I'd
like to start from scratch and see where that takes us. Since my top
priority for partial clone is to enable fast single revision checkouts
in automated environments, it makes sense to provide a command that
does this while simultaneously allowing us to test partial clone and
measure its performance.

Diff Detail

Repository

rHG Mercurial

Lint

Lint Skipped

Unit

Unit Tests Skipped

Event Timeline

indygreg created this revision.Sep 26 2018, 5:44 PM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptSep 26 2018, 5:44 PM

Herald added subscribers: mercurial-devel, mjpieters. · View Herald Transcript

indygreg added a child revision: D4766: fastcheckout: arguments to make checkout narrow.Sep 26 2018, 5:44 PM

Per discussions at 4.8 sprint, we're going to hold off on this.

Revision Contents
Changeset List

			Path	Packages
A	M		hgext/fastcheckout.py (166 lines)
A	M		tests/test-fastcheckout.t (123 lines)

Commit	Parents	Author	Summary	Date
		Gregory Szorc		Sep 7 2018, 5:22 PM

Status	Author	Revision
Abandoned	indygreg	D4767 exchangev2: recognize narrow patterns when pulling
Abandoned	indygreg	D4766 fastcheckout: arguments to make checkout narrow
Abandoned	indygreg	D4765 fastcheckout: extension for obtaining a checkout quickly

Diff 11436

hgext/fastcheckout.py

This file was added.

				# fastcheckout.py - obtain working directories quickly
				#
				# Copyright 2018 Gregory Szorc <gregory.szorc@gmail.com>
				#
				# This software may be used and distributed according to the terms of the
				# GNU General Public License version 2 or any later version.

				"""create working directories as fast as possible (EXPERIMENTAL)

				This extension provides the `hg fastcheckout` command, which is used to
				create or update a working directory to a specific revision as fast as
				possible.

				The extension is intended to be an incubator for "partial clone" features
				until they are stable enough to be migrated to core Mercurial. As such,
				there is no guarantee the extension will ever become non-experimental or
				retained in future Mercurial versions.
				"""

				from __future__ import absolute_import

				from mercurial.i18n import _

				from mercurial import (
				cmdutil,
				error,
				exchange,
				hg,
				merge as mergemod,
				pycompat,
				registrar,
				scmutil,
				util,
				vfs as vfsmod,
				)

				# Note for extension authors: ONLY specify testedwith = 'ships-with-hg-core' for
				# extensions which SHIP WITH MERCURIAL. Non-mainline extensions should
				# be specifying the version(s) of Mercurial they are tested with, or
				# leave the attribute unspecified.
				testedwith = 'ships-with-hg-core'

				cmdtable = {}
				command = registrar.command(cmdtable)

				@command(
				'fastcheckout',
				[
				('C', 'clean', False, _('discard uncommitted changes (no backup)')),
				('', 'purge', False,
				_('purge the working directory of untracked files'),
				),
				('', 'purge-all', False,
				_('purge the working directory of untracked and ignored files')),
				],
				_('SOURCE REV [DEST]'),
				norepo=True)
				def fastcheckout(ui, source, rev, dest=None, **opts):
				"""obtain a working directory quickly

				Receives as arguments a repository source, revision, and optional
				destination directory. If the destination path is not specified, the
				basename of the source is used.

				After successful execution, the destination contains a working directory
				of the specified revision. The destination will be created and repository
				data cloned if necessary. Otherwise, an incremental pull and checkout
				is performed.

				``--clean`` will discard uncommitted changes and the working directory
				state will match the state of files in the requested revision.

				``--purge`` can be used to purge the working directory of untracked
				files, similar to running :hg:`purge`. ``--purge-all`` can be used to
				purge the working directory of both untracked and ignored files, similar
				to running :hg:`purge --all`.

				The source repository must support modern versions of the Mercurial
				wire protocol.
				"""
				opts = pycompat.byteskwargs(opts)

				if dest:
				dest = ui.expandpath(dest)
				else:
				dest = hg.defaultdest(source)
				if dest:
				ui.status(_('destination directory: %s\n') % dest)

				dest = util.urllocalpath(dest)

				if not dest:
				raise error.Abort(_('empty destination path is not valid'))

				destvfs = vfsmod.vfs(dest, expandpath=True)

				overrides = {
				('experimental', 'httppeer.advertise-v2'): True,
				}

				with ui.configoverride(overrides):
				return dofastcheckout(ui, source, rev, destvfs,
				purge=opts.get('purge', False),
				purgeall=opts.get('purgeall', False),
				clean=opts.get('clean', False))

				def getpeer(ui, source):
				peer = hg.peer(ui, {}, source)

				if not peer.capable('command-changesetdata'):
				raise error.Abort(_('source repository does not support modern wire '
				'protocol features needed for fast checkouts'))

				return peer

				def dofastcheckout(ui, source, rev, destvfs, purge=False, purgeall=False,
				clean=False):

				createdrepo = False
				peer = None

				if not destvfs.lexists():
				# TODO perform partial clone here.
				peer = getpeer(ui, source)
				ui.write(_('fetching data from %s\n') % peer.url())
				hg.clone(ui, {}, peer, destvfs.base, pull=True,
				update=False, revs=[rev])

				createdrepo = True

				repo = hg.repository(ui, destvfs.base)

				try:
				scmutil.revsingle(repo, rev)
				haverev = True
				except error.RepoLookupError:
				haverev = False

				if not haverev:
				if not peer:
				peer = getpeer(ui, source)

				ui.write(_('fetching data from %s\n') % peer.url())

				with peer.commandexecutor() as e:
				pullrev = e.callcommand('lookup', {'key': rev}).result()

				exchange.pull(repo, peer, heads=[pullrev])

				if not createdrepo and (purge or purgeall):
				ui.write(_('purging working directory\n'))
				matcher = scmutil.match(repo[None])
				purgecount = len(mergemod.purge(repo, matcher, ignored=purgeall,
				abortonerror=True))
				ui.write(_('purged %d items from working directory\n') % purgecount)

				with repo.wlock():
				cmdutil.clearunfinished(repo)

				ctx = scmutil.revsingle(repo, rev, rev)
				ui.write(_('updating to %s\n') % ctx)

				ret = hg.updatetotally(ui, repo, ctx.rev(), ctx.rev(),
				clean=clean, updatecheck='none')

				return ret

tests/test-fastcheckout.t

This file was added.

				$ . $TESTDIR/wireprotohelpers.sh

				$ cat >> $HGRCPATH << EOF
				> [extensions]
				> fastcheckout =
				> EOF

				$ hg init server
				$ cd server

				$ echo 0 > foo
				$ mkdir -p dir0/child0 dir0/child1 dir1
				$ echo 0 > dir0/file0.txt
				$ echo 0 > dir0/child0/py0.py
				$ echo 0 > dir0/child0/c.c
				$ echo 0 > dir0/child1/py1.py
				$ echo 0 > dir0/child1/py2.py
				$ echo 0 > dir1/file1.txt
				$ echo 0 > dir1/file2.txt
				$ echo 0 > dir1/py3.py

				$ hg -q commit -A -m 'commit 0'

				$ echo 1 > dir0/file0.txt
				$ echo 1 > dir0/child1/py1.py
				$ echo 1 > dir0/child1/py2.py
				$ hg commit -m 'commit 1'

				$ echo 2 > dir1/py3.py
				$ hg commit -m 'commit 2'
				$ echo 0 > dir0/file3.txt
				$ hg commit -A -m 'commit 3'
				adding dir0/file3.txt

				$ hg log -G -T '{rev}:{node} {desc}'
				@ 3:27c5539872f8d2cfd5b95cab1fcaaee9b9bf9459 commit 3
				\|
				o 2:15c769176aab806212712ee9df382ebc7b63ade1 commit 2
				\|
				o 1:805a128077322d4be238a5d74719670ef29969af commit 1
				\|
				o 0:6a99379240831e67873d9bf61a687f20b08e6b95 commit 0


				$ cd ..

				$ hg -R server serve -p $HGPORT -d --pid-file hg.pid -E error.log
				$ cat hg.pid > $DAEMON_PIDS

				`hg fastcheckout` only works if new wire protocol is available

				$ hg fastcheckout http://localhost:$HGPORT irrelevant-rev checkout-missing
				abort: source repository does not support modern wire protocol features needed for fast checkouts
				[255]

				$ test -d checkout-missing && echo 'checkout-missing exists'
				[1]

				Restart server with modern wire protocol enabled

				$ killdaemons.py
				$ enablehttpv2 server
				$ hg -R server serve -p $HGPORT -d --pid-file hg.pid -E error.log
				$ cat hg.pid > $DAEMON_PIDS

				Missing destination results in new checkout

				$ hg fastcheckout http://localhost:$HGPORT 805a128077 dest0
				fetching data from http://localhost:$HGPORT
				new changesets 6a9937924083:805a12807732
				updating to 805a12807732
				9 files updated, 0 files merged, 0 files removed, 0 files unresolved

				Incremental update for known revision works

				$ hg fastcheckout http://localhost:$HGPORT 6a9937924083 dest0
				updating to 6a9937924083
				3 files updated, 0 files merged, 0 files removed, 0 files unresolved

				Incremental update for unknown revision fetches new revision

				$ hg fastcheckout http://localhost:$HGPORT 15c769176aab dest0
				fetching data from http://localhost:$HGPORT
				searching for changes
				new changesets 15c769176aab
				updating to 15c769176aab
				4 files updated, 0 files merged, 0 files removed, 0 files unresolved

				--purge removes untracked files

				$ touch dest0/extra0
				$ mkdir dest0/extradir
				$ hg fastcheckout --purge http://localhost:$HGPORT 15c769176aab dest0
				purging working directory
				purged 2 items from working directory
				updating to 15c769176aab
				0 files updated, 0 files merged, 0 files removed, 0 files unresolved
				$ ls dest0 \| sort
				dir0
				dir1
				foo

				--clean restores pristine version of file

				$ echo modified > dest0/dir1/py3.py
				$ hg fastcheckout --clean http://localhost:$HGPORT 15c769176aab dest0
				updating to 15c769176aab
				1 files updated, 0 files merged, 0 files removed, 0 files unresolved

				$ cat dest0/dir1/py3.py
				2

				$ echo modified > dest0/dir1/py3.py
				$ hg fastcheckout --clean http://localhost:$HGPORT 6a9937924 dest0
				updating to 6a9937924083
				4 files updated, 0 files merged, 0 files removed, 0 files unresolved

				$ cat dest0/dir1/py3.py
				0

				Should not have any server-side errors

				$ cat error.log