This is an archive of the discontinued Mercurial Phabricator instance.

Differential D5122

remotefilelog: remove support for fetching packs
ClosedPublic

Authored by durin42 on Oct 16 2018, 3:18 PM.

Details

Reviewers
indygreg
Group Reviewers
hg-reviewers
Commits
rHG6d64e2abe8d3: remotefilelog: remove support for fetching packs
Summary

We've never used this at Google, and it's strictly an optimization
that indygreg and I are confident we can handle more elegantly
later. Let's rip it out to keep the surface area of the imported
remotefilelog small.

Some tests have changes because the tests for regular packs and
repacking were mixed with tests of fetching packs. I'm pretty sure
I've done the right thing here.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

durin42 created this revision.Oct 16 2018, 3:18 PM
indygreg accepted this revision.Oct 17 2018, 2:47 AM
This revision is now accepted and ready to land.Oct 17 2018, 2:47 AM
Closed by commit rHG6d64e2abe8d3: remotefilelog: remove support for fetching packs (authored by durin42). · Explain WhyNov 5 2018, 2:53 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 12414

hgext/remotefilelog/__init__.py

background background
``remotefilelog.cachepath`` path to cache ``remotefilelog.cachepath`` path to cache
``remotefilelog.cachegroup`` if set, make cache directory sgid to this ``remotefilelog.cachegroup`` if set, make cache directory sgid to this
group group
``remotefilelog.cacheprocess`` binary to invoke for fetching file data ``remotefilelog.cacheprocess`` binary to invoke for fetching file data
``remotefilelog.debug`` turn on remotefilelog-specific debug output ``remotefilelog.debug`` turn on remotefilelog-specific debug output
``remotefilelog.excludepattern`` pattern of files to exclude from pulls ``remotefilelog.excludepattern`` pattern of files to exclude from pulls
``remotefilelog.includepattern``pattern of files to include in pulls ``remotefilelog.includepattern``pattern of files to include in pulls
``remotefilelog.fetchpacks`` if set, fetch pre-packed files from the server
``remotefilelog.fetchwarning``: message to print when too many ``remotefilelog.fetchwarning``: message to print when too many
single-file fetches occur single-file fetches occur
``remotefilelog.getfilesstep`` number of files to request in a single RPC ``remotefilelog.getfilesstep`` number of files to request in a single RPC
``remotefilelog.getfilestype`` if set to 'threaded' use threads to fetch ``remotefilelog.getfilestype`` if set to 'threaded' use threads to fetch
files, otherwise use optimistic fetching files, otherwise use optimistic fetching
``remotefilelog.pullprefetch`` revset for selecting files that should be ``remotefilelog.pullprefetch`` revset for selecting files that should be
eagerly downloaded rather than lazily eagerly downloaded rather than lazily
``remotefilelog.reponame`` name of the repo. If set, used to partition ``remotefilelog.reponame`` name of the repo. If set, used to partition
configitem('remotefilelog', 'reponame', default='') configitem('remotefilelog', 'reponame', default='')
configitem('remotefilelog', 'cachepath', default=None) configitem('remotefilelog', 'cachepath', default=None)
configitem('remotefilelog', 'cachegroup', default=None) configitem('remotefilelog', 'cachegroup', default=None)
configitem('remotefilelog', 'cacheprocess', default=None) configitem('remotefilelog', 'cacheprocess', default=None)
configitem('remotefilelog', 'cacheprocess.includepath', default=None) configitem('remotefilelog', 'cacheprocess.includepath', default=None)
configitem("remotefilelog", "cachelimit", default="1000 GB") configitem("remotefilelog", "cachelimit", default="1000 GB")
configitem('remotefilelog', 'fetchpacks', default=False)
configitem('remotefilelog', 'fallbackpath', default=configitems.dynamicdefault, configitem('remotefilelog', 'fallbackpath', default=configitems.dynamicdefault,
alias=[('remotefilelog', 'fallbackrepo')]) alias=[('remotefilelog', 'fallbackrepo')])
configitem('remotefilelog', 'validatecachelog', default=None) configitem('remotefilelog', 'validatecachelog', default=None)
configitem('remotefilelog', 'validatecache', default='on') configitem('remotefilelog', 'validatecache', default='on')
configitem('remotefilelog', 'server', default=None) configitem('remotefilelog', 'server', default=None)
configitem('remotefilelog', 'servercachepath', default=None) configitem('remotefilelog', 'servercachepath', default=None)
configitem("remotefilelog", "serverexpiration", default=30) configitem("remotefilelog", "serverexpiration", default=30)

hgext/remotefilelog/fileserverclient.py

# fileserverclient.py - client for communicating with the cache process # fileserverclient.py - client for communicating with the cache process
# #
# Copyright 2013 Facebook, Inc. # Copyright 2013 Facebook, Inc.
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
from __future__ import absolute_import from __future__ import absolute_import
import hashlib import hashlib
import io import io
import os import os
import struct
import threading import threading
import time import time
from mercurial.i18n import _ from mercurial.i18n import _
from mercurial.node import bin, hex, nullid from mercurial.node import bin, hex, nullid
from mercurial import ( from mercurial import (
error, error,
revlog, revlog,
sshpeer, sshpeer,
util, util,
wireprotov1peer, wireprotov1peer,
) )
from mercurial.utils import procutil from mercurial.utils import procutil
from . import ( from . import (
constants, constants,
contentstore, contentstore,
lz4wrapper, lz4wrapper,
metadatastore, metadatastore,
shallowutil,
wirepack,
) )
_sshv1peer = sshpeer.sshv1peer _sshv1peer = sshpeer.sshv1peer
# Statistics for debugging # Statistics for debugging
fetchcost = 0 fetchcost = 0
fetches = 0 fetches = 0
fetched = 0 fetched = 0
A list of nodes that the server couldn't find is returned. A list of nodes that the server couldn't find is returned.
If the connection fails, an exception is raised. If the connection fails, an exception is raised.
""" """
if not self.remotecache.connected: if not self.remotecache.connected:
self.connect() self.connect()
cache = self.remotecache cache = self.remotecache
writedata = self.writedata writedata = self.writedata
if self.ui.configbool('remotefilelog', 'fetchpacks'):
self.requestpack(fileids)
return
repo = self.repo repo = self.repo
count = len(fileids) count = len(fileids)
request = "get\n%d\n" % count request = "get\n%d\n" % count
idmap = {} idmap = {}
reponame = repo.name reponame = repo.name
for file, id in fileids: for file, id in fileids:
fullid = getcachekey(reponame, file, id) fullid = getcachekey(reponame, file, id)
if self.cacheprocesspasspath: if self.cacheprocesspasspath:
if len(data) != size: if len(data) != size:
raise error.ResponseError(_("error downloading file contents:"), raise error.ResponseError(_("error downloading file contents:"),
_("only received %s of %s bytes") _("only received %s of %s bytes")
% (len(data), size)) % (len(data), size))
self.writedata.addremotefilelognode(filename, bin(node), self.writedata.addremotefilelognode(filename, bin(node),
lz4wrapper.lz4decompress(data)) lz4wrapper.lz4decompress(data))
def requestpack(self, fileids):
"""Requests the given file revisions from the server in a pack format.
See `remotefilelogserver.getpack` for the file format.
"""
try:
with self._connect() as conn:
total = len(fileids)
rcvd = 0
remote = conn.peer
remote._callstream("getpackv1")
self._sendpackrequest(remote, fileids)
packpath = shallowutil.getcachepackpath(
self.repo, constants.FILEPACK_CATEGORY)
pipei = remote._pipei
receiveddata, receivedhistory = wirepack.receivepack(
self.repo.ui, pipei, packpath)
rcvd = len(receiveddata)
self.ui.log("remotefilefetchlog",
"Success(pack)\n" if (rcvd==total) else "Fail(pack)\n",
fetched_files = rcvd,
total_to_fetch = total)
except Exception:
self.ui.log("remotefilefetchlog",
"Fail(pack)\n",
fetched_files = rcvd,
total_to_fetch = total)
raise
def _sendpackrequest(self, remote, fileids):
"""Formats and writes the given fileids to the remote as part of a
getpackv1 call.
"""
# Sort the requests by name, so we receive requests in batches by name
grouped = {}
for filename, node in fileids:
grouped.setdefault(filename, set()).add(node)
# Issue request
pipeo = remote._pipeo
for filename, nodes in grouped.iteritems():
filenamelen = struct.pack(constants.FILENAMESTRUCT, len(filename))
countlen = struct.pack(constants.PACKREQUESTCOUNTSTRUCT, len(nodes))
rawnodes = ''.join(bin(n) for n in nodes)
pipeo.write('%s%s%s%s' % (filenamelen, filename, countlen,
rawnodes))
pipeo.flush()
pipeo.write(struct.pack(constants.FILENAMESTRUCT, 0))
pipeo.flush()
def connect(self): def connect(self):
if self.cacheprocess: if self.cacheprocess:
cmd = "%s %s" % (self.cacheprocess, self.writedata._path) cmd = "%s %s" % (self.cacheprocess, self.writedata._path)
self.remotecache.connect(cmd) self.remotecache.connect(cmd)
else: else:
# If no cache process is specified, we fake one that always # If no cache process is specified, we fake one that always
# returns cache misses. This enables tests to run easily # returns cache misses. This enables tests to run easily
# and may eventually allow us to be a drop in replacement # and may eventually allow us to be a drop in replacement

hgext/remotefilelog/shallowrepo.py

packcontentstore, cachecontent, packcontentstore, cachecontent,
localcontent, remotecontent, writestore=localcontent) localcontent, remotecontent, writestore=localcontent)
repo.metadatastore = metadatastore.unionmetadatastore( repo.metadatastore = metadatastore.unionmetadatastore(
packmetadatastore, cachemetadata, localmetadata, remotemetadata, packmetadatastore, cachemetadata, localmetadata, remotemetadata,
writestore=localmetadata) writestore=localmetadata)
fileservicedatawrite = cachecontent fileservicedatawrite = cachecontent
fileservicehistorywrite = cachemetadata fileservicehistorywrite = cachemetadata
if repo.ui.configbool('remotefilelog', 'fetchpacks'):
fileservicedatawrite = packcontentstore
fileservicehistorywrite = packmetadatastore
repo.fileservice.setstore(repo.contentstore, repo.metadatastore, repo.fileservice.setstore(repo.contentstore, repo.metadatastore,
fileservicedatawrite, fileservicehistorywrite) fileservicedatawrite, fileservicehistorywrite)
shallowutil.reportpackmetrics(repo.ui, 'filestore', shallowutil.reportpackmetrics(repo.ui, 'filestore',
packcontentstore, packmetadatastore) packcontentstore, packmetadatastore)
def wraprepo(repo): def wraprepo(repo):
class shallowrepository(repo.__class__): class shallowrepository(repo.__class__):
@util.propertycache @util.propertycache

tests/test-remotefilelog-prefetch.t

$ hg up -q --clean tip $ hg up -q --clean tip
$ hg revert --all $ hg revert --all
$ mv x x2 $ mv x x2
$ mv y y2 $ mv y y2
$ mv z z2 $ mv z z2
$ clearcache $ clearcache
$ hg addremove -s 50 > /dev/null $ hg addremove -s 50 > /dev/null
3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over * (glob) 3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over * (glob)
$ hg revert --all
forgetting x2
forgetting y2
forgetting z2
undeleting x
undeleting y
undeleting z
$ cd ..
# Prefetch packs # Revert across double renames. Note: the scary "abort", error is because
$ hgcloneshallow ssh://user@dummy/master packprefetch # https://bz.mercurial-scm.org/5419 .
streaming all changes
2 files to transfer, 528 bytes of data
transferred 528 bytes in * seconds (*/sec) (glob)
searching for changes
no changes found
updating to branch default
3 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd packprefetch
$ cat >> .hg/hgrc <<EOF
> [remotefilelog]
> fetchpacks=True
> backgroundrepack=True
> EOF
$ clearcache
$ hg prefetch -r .
3 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob)
$ find $TESTTMP/hgcache -type f | sort
$TESTTMP/hgcache/master/packs/47d8f1b90a73af4ff8af19fcd10bdc027b6a881a.histidx
$TESTTMP/hgcache/master/packs/47d8f1b90a73af4ff8af19fcd10bdc027b6a881a.histpack
$TESTTMP/hgcache/master/packs/8c654541e4f20141a894bbfe428e36fc92202e39.dataidx
$TESTTMP/hgcache/master/packs/8c654541e4f20141a894bbfe428e36fc92202e39.datapack
$ hg cat -r . x
x2
$ hg cat -r . y
y
$ hg cat -r . z
z
# Prefetch packs that include renames
$ cd ../master $ cd ../master
$ hg mv z z2 $ hg mv z z2
$ hg commit -m 'move z -> z2' $ hg commit -m 'move z -> z2'
$ cd ../packprefetch $ cd ../shallow2
$ hg pull -q $ hg pull -q
(running background incremental repack)
$ hg prefetch -r tip
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob)
$ hg up tip -q
$ hg log -f z2 -T '{desc}\n'
move z -> z2
x
# Revert across double renames. Note: the scary "abort", error is because
# https://bz.mercurial-scm.org/5419 .
$ clearcache $ clearcache
$ hg mv y y2 $ hg mv y y2
y2: not overwriting - file exists
('hg rename --after' to record the rename)
[1]
$ hg mv x x2 $ hg mv x x2
x2: not overwriting - file exists
('hg rename --after' to record the rename)
[1]
$ hg mv z2 z3 $ hg mv z2 z3
z2: not copying - file is not managed
abort: no files to copy
[255]
$ hg revert -a -r 1 || true $ hg revert -a -r 1 || true
forgetting x2 3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over * (glob)
forgetting y2
forgetting z3
adding z
undeleting x
undeleting y
3 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob)
abort: z2@109c3a557a73: not found in manifest! (?) abort: z2@109c3a557a73: not found in manifest! (?)

tests/test-remotefilelog-repack-fast.t

x x
Incremental repack Incremental repack
$ rm -rf $CACHEDIR/master/packs/* $ rm -rf $CACHEDIR/master/packs/*
$ cat >> .hg/hgrc <<EOF $ cat >> .hg/hgrc <<EOF
> [remotefilelog] > [remotefilelog]
> data.generations=60 > data.generations=60
> 150 > 150
> fetchpacks=True
> EOF > EOF
Single pack - repack does nothing Single pack - repack does nothing
$ hg prefetch -r 0 $ hg prefetch -r 0
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack [1]
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack [1]
$ hg repack --incremental $ hg repack --incremental
$ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack -r--r--r-- 67 b5a62f3496ccbd2479497cdbc7345f3304735f33.datapack
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack -r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack
3 gen1 packs, 1 gen0 pack - packs 3 gen1 into 1 3 gen1 packs, 1 gen0 pack - packs 3 gen1 into 1
$ hg prefetch -r 1 $ hg prefetch -r 1
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ hg prefetch -r 2 $ hg prefetch -r 2
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ hg prefetch -r 3 $ hg prefetch -r 3
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack -r--r--r-- 67 b5a62f3496ccbd2479497cdbc7345f3304735f33.datapack
-r--r--r-- 65 6c499d21350d79f92fd556b4b7a902569d88e3c9.datapack
-r--r--r-- 61 817d294043bd21a3de01f807721971abe45219ce.datapack
-r--r--r-- 63 ff45add45ab3f59c4f75efc6a087d86c821219d6.datapack
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 254 077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histpack
-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack
-r--r--r-- 172 276d308429d0303762befa376788300f0310f90e.histpack
-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack -r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack
$ hg repack --incremental $ hg repack --incremental
$ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack
-r--r--r-- 225 8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack -r--r--r-- 225 8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack
-r--r--r-- 67 b5a62f3496ccbd2479497cdbc7345f3304735f33.datapack
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack -r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack
-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack
1 gen3 pack, 1 gen0 pack - does nothing 1 gen3 pack, 1 gen0 pack - does nothing
$ hg repack --incremental $ hg repack --incremental
$ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack
-r--r--r-- 225 8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack -r--r--r-- 225 8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack
-r--r--r-- 67 b5a62f3496ccbd2479497cdbc7345f3304735f33.datapack
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack -r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack
-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack
Pull should run background repack Pull should run background repack
$ cat >> .hg/hgrc <<EOF $ cat >> .hg/hgrc <<EOF
> [remotefilelog] > [remotefilelog]
> backgroundrepack=True > backgroundrepack=True
> EOF > EOF
$ clearcache $ clearcache
$ hg prefetch -r 0 $ hg prefetch -r 0
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ hg prefetch -r 1 $ hg prefetch -r 1
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ hg prefetch -r 2 $ hg prefetch -r 2
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ hg prefetch -r 3 $ hg prefetch -r 3
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack
-r--r--r-- 65 6c499d21350d79f92fd556b4b7a902569d88e3c9.datapack
-r--r--r-- 61 817d294043bd21a3de01f807721971abe45219ce.datapack
-r--r--r-- 63 ff45add45ab3f59c4f75efc6a087d86c821219d6.datapack
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 254 077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histpack
-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack
-r--r--r-- 172 276d308429d0303762befa376788300f0310f90e.histpack
-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack
$ hg pull $ hg pull
pulling from ssh://user@dummy/master pulling from ssh://user@dummy/master
searching for changes searching for changes
no changes found no changes found
(running background incremental repack) (running background incremental repack)
$ sleep 0.5 $ sleep 0.5
$ hg debugwaitonrepack >/dev/null 2>&1 $ hg debugwaitonrepack >/dev/null 2>&1
$ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack -r--r--r-- 301 09b8bf49256b3fc2175977ba97d6402e91a9a604.datapack
-r--r--r-- 225 8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack -r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack
Test environment variable resolution Test environment variable resolution
$ CACHEPATH=$TESTTMP/envcache hg prefetch --config 'remotefilelog.cachepath=$CACHEPATH' $ CACHEPATH=$TESTTMP/envcache hg prefetch --config 'remotefilelog.cachepath=$CACHEPATH'
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ find $TESTTMP/envcache | sort $ find $TESTTMP/envcache | sort
$TESTTMP/envcache $TESTTMP/envcache
$TESTTMP/envcache/master $TESTTMP/envcache/master
$TESTTMP/envcache/master/packs $TESTTMP/envcache/master/95
$TESTTMP/envcache/master/packs/54afbfda203716c1aa2636029ccc0df18165129e.dataidx $TESTTMP/envcache/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a
$TESTTMP/envcache/master/packs/54afbfda203716c1aa2636029ccc0df18165129e.datapack $TESTTMP/envcache/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a/577959738234a1eb241ed3ed4b22a575833f56e0
$TESTTMP/envcache/master/packs/dcebd8e8d4d97ee88e40dd8f92d8678c10e1a3ad.histidx $TESTTMP/envcache/repos
$TESTTMP/envcache/master/packs/dcebd8e8d4d97ee88e40dd8f92d8678c10e1a3ad.histpack
Test local remotefilelog blob is correct when based on a pack Test local remotefilelog blob is correct when based on a pack
$ hg prefetch -r . $ hg prefetch -r .
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ echo >> y $ echo >> y
$ hg commit -m y2 $ hg commit -m y2
$ hg debugremotefilelog .hg/store/data/95cb0bfd2977c761298d9624e4b4d4c72a39974a/b70860edba4f8242a1d52f2a94679dd23cb76808 $ hg debugremotefilelog .hg/store/data/95cb0bfd2977c761298d9624e4b4d4c72a39974a/b70860edba4f8242a1d52f2a94679dd23cb76808
size: 9 bytes size: 9 bytes
path: .hg/store/data/95cb0bfd2977c761298d9624e4b4d4c72a39974a/b70860edba4f8242a1d52f2a94679dd23cb76808 path: .hg/store/data/95cb0bfd2977c761298d9624e4b4d4c72a39974a/b70860edba4f8242a1d52f2a94679dd23cb76808
key: b70860edba4f key: b70860edba4f
node => p1 p2 linknode copyfrom node => p1 p2 linknode copyfrom
b70860edba4f => 577959738234 000000000000 08d3fbc98c48 b70860edba4f => 577959738234 000000000000 08d3fbc98c48
577959738234 => 1bb2e6237e03 000000000000 c7faf2fc439a x 577959738234 => 1bb2e6237e03 000000000000 c7faf2fc439a x
1bb2e6237e03 => d4a3ed9310e5 000000000000 0b03bbc9e1e7 1bb2e6237e03 => d4a3ed9310e5 000000000000 0b03bbc9e1e7
d4a3ed9310e5 => aee31534993a 000000000000 421535db10b6 d4a3ed9310e5 => aee31534993a 000000000000 421535db10b6
aee31534993a => 1406e7411862 000000000000 a89d614e2364 aee31534993a => 1406e7411862 000000000000 a89d614e2364
1406e7411862 => 000000000000 000000000000 b292c1e3311f 1406e7411862 => 000000000000 000000000000 b292c1e3311f

tests/test-remotefilelog-repack.t

x x
Incremental repack Incremental repack
$ rm -rf $CACHEDIR/master/packs/* $ rm -rf $CACHEDIR/master/packs/*
$ cat >> .hg/hgrc <<EOF $ cat >> .hg/hgrc <<EOF
> [remotefilelog] > [remotefilelog]
> data.generations=60 > data.generations=60
> 150 > 150
> fetchpacks=True
> EOF > EOF
Single pack - repack does nothing Single pack - repack does nothing
$ hg prefetch -r 0 $ hg prefetch -r 0
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack [1]
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack [1]
$ hg repack --incremental $ hg repack --incremental
$ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack -r--r--r-- 67 b5a62f3496ccbd2479497cdbc7345f3304735f33.datapack
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack -r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack
3 gen1 packs, 1 gen0 pack - packs 3 gen1 into 1 3 gen1 packs, 1 gen0 pack - packs 3 gen1 into 1
$ hg prefetch -r 1 $ hg prefetch -r 1
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ hg prefetch -r 2 $ hg prefetch -r 2
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ hg prefetch -r 3 $ hg prefetch -r 38
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) abort: unknown revision '38'!
[255]
$ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack -r--r--r-- 67 b5a62f3496ccbd2479497cdbc7345f3304735f33.datapack
-r--r--r-- 65 6c499d21350d79f92fd556b4b7a902569d88e3c9.datapack
-r--r--r-- 61 817d294043bd21a3de01f807721971abe45219ce.datapack
-r--r--r-- 63 ff45add45ab3f59c4f75efc6a087d86c821219d6.datapack
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 254 077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histpack
-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack
-r--r--r-- 172 276d308429d0303762befa376788300f0310f90e.histpack
-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack -r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack
For the data packs, setting the limit for the repackmaxpacksize to be 64 such For the data packs, setting the limit for the repackmaxpacksize to be 64 such
that data pack with size 65 is more than the limit. This effectively ensures that data pack with size 65 is more than the limit. This effectively ensures
that no generation has 3 packs and therefore, no packs are chosen for the that no generation has 3 packs and therefore, no packs are chosen for the
incremental repacking. As for the history packs, setting repackmaxpacksize to be incremental repacking. As for the history packs, setting repackmaxpacksize to be
0 which should always result in no repacking. 0 which should always result in no repacking.
$ hg repack --incremental --config remotefilelog.data.repackmaxpacksize=64 \ $ hg repack --incremental --config remotefilelog.data.repackmaxpacksize=64 \
> --config remotefilelog.history.repackmaxpacksize=0 > --config remotefilelog.history.repackmaxpacksize=0
$ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack -r--r--r-- 147 935861cae0be6ce41a0d47a529e4d097e9e68a69.datapack
-r--r--r-- 65 6c499d21350d79f92fd556b4b7a902569d88e3c9.datapack -r--r--r-- 67 b5a62f3496ccbd2479497cdbc7345f3304735f33.datapack
-r--r--r-- 61 817d294043bd21a3de01f807721971abe45219ce.datapack
-r--r--r-- 63 ff45add45ab3f59c4f75efc6a087d86c821219d6.datapack
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 254 077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histpack -r--r--r-- 254 077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histpack
-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack
-r--r--r-- 172 276d308429d0303762befa376788300f0310f90e.histpack
-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack -r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack
Setting limit for the repackmaxpacksize to be the size of the biggest pack file Setting limit for the repackmaxpacksize to be the size of the biggest pack file
which ensures that it is effectively ignored in the incremental repacking. which ensures that it is effectively ignored in the incremental repacking.
$ hg repack --incremental --config remotefilelog.data.repackmaxpacksize=65 \ $ hg repack --incremental --config remotefilelog.data.repackmaxpacksize=65 \
> --config remotefilelog.history.repackmaxpacksize=336 > --config remotefilelog.history.repackmaxpacksize=336
$ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack -r--r--r-- 147 935861cae0be6ce41a0d47a529e4d097e9e68a69.datapack
-r--r--r-- 225 8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack -r--r--r-- 67 b5a62f3496ccbd2479497cdbc7345f3304735f33.datapack
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack -r--r--r-- 254 077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histpack
-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack
1 gen3 pack, 1 gen0 pack - does nothing 1 gen3 pack, 1 gen0 pack - does nothing
$ hg repack --incremental $ hg repack --incremental
$ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack -r--r--r-- 147 935861cae0be6ce41a0d47a529e4d097e9e68a69.datapack
-r--r--r-- 225 8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack -r--r--r-- 67 b5a62f3496ccbd2479497cdbc7345f3304735f33.datapack
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack -r--r--r-- 254 077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histpack
-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack
Pull should run background repack Pull should run background repack
$ cat >> .hg/hgrc <<EOF $ cat >> .hg/hgrc <<EOF
> [remotefilelog] > [remotefilelog]
> backgroundrepack=True > backgroundrepack=True
> EOF > EOF
$ clearcache $ clearcache
$ hg prefetch -r 0 $ hg prefetch -r 0
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ hg prefetch -r 1 $ hg prefetch -r 1
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ hg prefetch -r 2 $ hg prefetch -r 2
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ hg prefetch -r 3 $ hg prefetch -r 3
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack
-r--r--r-- 65 6c499d21350d79f92fd556b4b7a902569d88e3c9.datapack
-r--r--r-- 61 817d294043bd21a3de01f807721971abe45219ce.datapack
-r--r--r-- 63 ff45add45ab3f59c4f75efc6a087d86c821219d6.datapack
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 254 077e7ce5dfe862dc40cc8f3c9742d96a056865f2.histpack
-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack
-r--r--r-- 172 276d308429d0303762befa376788300f0310f90e.histpack
-r--r--r-- 90 c3399b56e035f73c3295276ed098235a08a0ed8c.histpack
$ hg pull $ hg pull
pulling from ssh://user@dummy/master pulling from ssh://user@dummy/master
searching for changes searching for changes
no changes found no changes found
(running background incremental repack) (running background incremental repack)
$ sleep 0.5 $ sleep 0.5
$ hg debugwaitonrepack >/dev/null 2>&1 $ hg debugwaitonrepack >/dev/null 2>&1
$ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack $ ls_l $TESTTMP/hgcache/master/packs/ | grep datapack
-r--r--r-- 59 5b7dec902026f0cddb0ef8acb62f27b5698494d4.datapack -r--r--r-- 301 09b8bf49256b3fc2175977ba97d6402e91a9a604.datapack
-r--r--r-- 225 8fe685c56f6f7edf550bfcec74eeecc5f3c2ba15.datapack
$ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack $ ls_l $TESTTMP/hgcache/master/packs/ | grep histpack
-r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack -r--r--r-- 336 094b530486dad4427a0faf6bcbc031571b99ca24.histpack
Test environment variable resolution Test environment variable resolution
$ CACHEPATH=$TESTTMP/envcache hg prefetch --config 'remotefilelog.cachepath=$CACHEPATH' $ CACHEPATH=$TESTTMP/envcache hg prefetch --config 'remotefilelog.cachepath=$CACHEPATH'
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ find $TESTTMP/envcache | sort $ find $TESTTMP/envcache | sort
$TESTTMP/envcache $TESTTMP/envcache
$TESTTMP/envcache/master $TESTTMP/envcache/master
$TESTTMP/envcache/master/packs $TESTTMP/envcache/master/95
$TESTTMP/envcache/master/packs/54afbfda203716c1aa2636029ccc0df18165129e.dataidx $TESTTMP/envcache/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a
$TESTTMP/envcache/master/packs/54afbfda203716c1aa2636029ccc0df18165129e.datapack $TESTTMP/envcache/master/95/cb0bfd2977c761298d9624e4b4d4c72a39974a/577959738234a1eb241ed3ed4b22a575833f56e0
$TESTTMP/envcache/master/packs/dcebd8e8d4d97ee88e40dd8f92d8678c10e1a3ad.histidx $TESTTMP/envcache/repos
$TESTTMP/envcache/master/packs/dcebd8e8d4d97ee88e40dd8f92d8678c10e1a3ad.histpack
Test local remotefilelog blob is correct when based on a pack Test local remotefilelog blob is correct when based on a pack
$ hg prefetch -r . $ hg prefetch -r .
1 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over * (glob)
$ echo >> y $ echo >> y
$ hg commit -m y2 $ hg commit -m y2
$ hg debugremotefilelog .hg/store/data/95cb0bfd2977c761298d9624e4b4d4c72a39974a/b70860edba4f8242a1d52f2a94679dd23cb76808 $ hg debugremotefilelog .hg/store/data/95cb0bfd2977c761298d9624e4b4d4c72a39974a/b70860edba4f8242a1d52f2a94679dd23cb76808
size: 9 bytes size: 9 bytes
path: .hg/store/data/95cb0bfd2977c761298d9624e4b4d4c72a39974a/b70860edba4f8242a1d52f2a94679dd23cb76808 path: .hg/store/data/95cb0bfd2977c761298d9624e4b4d4c72a39974a/b70860edba4f8242a1d52f2a94679dd23cb76808
key: b70860edba4f key: b70860edba4f
node => p1 p2 linknode copyfrom node => p1 p2 linknode copyfrom
b70860edba4f => 577959738234 000000000000 08d3fbc98c48 b70860edba4f => 577959738234 000000000000 08d3fbc98c48
577959738234 => 1bb2e6237e03 000000000000 c7faf2fc439a x 577959738234 => 1bb2e6237e03 000000000000 c7faf2fc439a x
1bb2e6237e03 => d4a3ed9310e5 000000000000 0b03bbc9e1e7 1bb2e6237e03 => d4a3ed9310e5 000000000000 0b03bbc9e1e7
d4a3ed9310e5 => aee31534993a 000000000000 421535db10b6 d4a3ed9310e5 => aee31534993a 000000000000 421535db10b6
aee31534993a => 1406e7411862 000000000000 a89d614e2364 aee31534993a => 1406e7411862 000000000000 a89d614e2364
1406e7411862 => 000000000000 000000000000 b292c1e3311f 1406e7411862 => 000000000000 000000000000 b292c1e3311f
Test limiting the max delta chain length Test limiting the max delta chain length
$ hg repack --config packs.maxchainlen=1 $ hg repack --config packs.maxchainlen=1
$ hg debugdatapack $TESTTMP/hgcache/master/packs/*.dataidx $ hg debugdatapack $TESTTMP/hgcache/master/packs/*.dataidx
$TESTTMP/hgcache/master/packs/a2731c9a16403457b67337a620931797fce8c821: $TESTTMP/hgcache/master/packs/a8378d888fe4325250720e51311a65d2509be742:
x: x:
Node Delta Base Delta Length Blob Size Node Delta Base Delta Length Blob Size
1bb2e6237e03 000000000000 8 8 1bb2e6237e03 000000000000 8 8
d4a3ed9310e5 1bb2e6237e03 12 6 d4a3ed9310e5 1bb2e6237e03 12 6
aee31534993a 000000000000 4 4 aee31534993a 000000000000 4 4
1406e7411862 aee31534993a 12 2 1406e7411862 aee31534993a 12 2
Total: 36 20 (80.0% bigger) Total: 36 20 (80.0% bigger)
y: y:
Node Delta Base Delta Length Blob Size Node Delta Base Delta Length Blob Size
577959738234 000000000000 8 8 577959738234 000000000000 70 8
Total: 8 8 (0.0% bigger) Total: 70 8 (775.0% bigger)
Test huge pack cleanup using different values of packs.maxpacksize: Test huge pack cleanup using different values of packs.maxpacksize:
$ hg repack --incremental --debug $ hg repack --incremental --debug
$ hg repack --incremental --debug --config packs.maxpacksize=512 $ hg repack --incremental --debug --config packs.maxpacksize=512
removing oversize packfile $TESTTMP/hgcache/master/packs/a2731c9a16403457b67337a620931797fce8c821.datapack (365 bytes) removing oversize packfile $TESTTMP/hgcache/master/packs/a8378d888fe4325250720e51311a65d2509be742.datapack (426 bytes)
removing oversize packfile $TESTTMP/hgcache/master/packs/a2731c9a16403457b67337a620931797fce8c821.dataidx (1.21 KB) removing oversize packfile $TESTTMP/hgcache/master/packs/a8378d888fe4325250720e51311a65d2509be742.dataidx (1.21 KB)
Do a repack where the new pack reuses a delta from the old pack Do a repack where the new pack reuses a delta from the old pack
$ clearcache $ clearcache
$ hg prefetch -r '2::3' $ hg prefetch -r '2::3'
2 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over * (glob)
$ hg repack $ hg repack
$ hg debugdatapack $CACHEDIR/master/packs/*.datapack $ hg debugdatapack $CACHEDIR/master/packs/*.datapack
$TESTTMP/hgcache/master/packs/abf210f6c3aa4dd0ecc7033633ad73591be16c95: $TESTTMP/hgcache/master/packs/abf210f6c3aa4dd0ecc7033633ad73591be16c95:
x: x:
Node Delta Base Delta Length Blob Size Node Delta Base Delta Length Blob Size
1bb2e6237e03 000000000000 8 8 1bb2e6237e03 000000000000 8 8
d4a3ed9310e5 1bb2e6237e03 12 6 d4a3ed9310e5 1bb2e6237e03 12 6
Total: 20 14 (42.9% bigger) Total: 20 14 (42.9% bigger)
$ hg prefetch -r '0::1' $ hg prefetch -r '0::1'
2 files fetched over 1 fetches - (0 misses, 100.00% hit ratio) over * (glob) 2 files fetched over 1 fetches - (2 misses, 0.00% hit ratio) over * (glob)
$ hg repack $ hg repack
$ hg debugdatapack $CACHEDIR/master/packs/*.datapack $ hg debugdatapack $CACHEDIR/master/packs/*.datapack
$TESTTMP/hgcache/master/packs/09b8bf49256b3fc2175977ba97d6402e91a9a604: $TESTTMP/hgcache/master/packs/09b8bf49256b3fc2175977ba97d6402e91a9a604:
x: x:
Node Delta Base Delta Length Blob Size Node Delta Base Delta Length Blob Size
1bb2e6237e03 000000000000 8 8 1bb2e6237e03 000000000000 8 8
d4a3ed9310e5 1bb2e6237e03 12 6 d4a3ed9310e5 1bb2e6237e03 12 6
aee31534993a d4a3ed9310e5 12 4 aee31534993a d4a3ed9310e5 12 4
1406e7411862 aee31534993a 12 2 1406e7411862 aee31534993a 12 2
Total: 44 20 (120.0% bigger) Total: 44 20 (120.0% bigger)