( )⚙ D2885 RFC: use Redis to cache file data

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D2885

RFC: use Redis to cache file data
AbandonedPublic

Authored by indygreg on Mar 16 2018, 7:07 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Summary

This [very hacky] commit implements a new files store that uses
Redis to cache files data. It services requests from Redis (if
available) and falls back to a base store / interface (revlogs in
our case) on misses.

The purpose of this commit is to first demonstrate the value in having
interfaces for storage. If we code to the interface, then another
interface can come along and do useful things - like caching.

The other purpose was to investigate performance. Would a memory-backed
key-value store have a significant impact on performance of our
experimental wire protocol command to serve file data fulltexts for
a specific revisions? The answer is a very resounding yet!

Using the same mozilla-unified revision from the previous commit:

  • no compression: 1478MB; ~94s wall; ~56s CPU w/ hot redis: 1478MB; ~9.6s wall; ~8.6s CPU
  • zstd level 3: 343MB; ~97s wall; ~57s CPU w/ hot redis: 343MB; ~8.5s wall; ~8.3s CPU
  • zstd level 1 w/ hot redis: 377MB; ~6.8s wall; ~6.6s CPU
  • zlib level 6: 367MB; ~116s wall; ~74s CPU w/ hot redis: 367MB; ~36.7s wall; ~36s CPU

For the curious, the ls profiler says that our hotspot without
compression is in socket I/O. With zstd compression, the hotspot is
compression.

I reckon the reason for the socket I/O overhead is because we end up
writing tons more chunks on the wire when uncompressed (compression
will effectively ensure each output chunk is a similar, large'ish
size). All those extra Python function calls and system calls do add
up!

Anyway, I'm definitely happy with the performance improvements. I'd
say this was a useful experiment!

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Mar 16 2018, 7:07 PM
indygreg abandoned this revision.Mar 22 2018, 11:33 AM

This was just an RFC. It doesn't need to be an open review.

Revision Contents
Changeset List

PathPackages
Mmercurial/localrepo.py (6 lines)
Mmercurial/revlogstore.py (48 lines)

Diff 7082

mercurial/localrepo.py

self.cachevfs.createmode = self.store.createmode self.cachevfs.createmode = self.store.createmode
if (self.ui.configbool('devel', 'all-warnings') or if (self.ui.configbool('devel', 'all-warnings') or
self.ui.configbool('devel', 'check-locks')): self.ui.configbool('devel', 'check-locks')):
if util.safehasattr(self.svfs, 'vfs'): # this is filtervfs if util.safehasattr(self.svfs, 'vfs'): # this is filtervfs
self.svfs.vfs.audit = self._getsvfsward(self.svfs.vfs.audit) self.svfs.vfs.audit = self._getsvfsward(self.svfs.vfs.audit)
else: # standard vfs else: # standard vfs
self.svfs.audit = self._getsvfsward(self.svfs.audit) self.svfs.audit = self._getsvfsward(self.svfs.audit)
self._applyopenerreqs() self._applyopenerreqs()
self.filesstore = revlogstore.revlogfilesstore(self.svfs) import redis
basefilesstore = revlogstore.revlogfilesstore(self.svfs)
redisconn = redis.StrictRedis(host='localhost', port=6379, db=0)
self.filesstore = revlogstore.redisacceleratedrevlogfilesstore(
redisconn, basefilesstore)
if create: if create:
self._writerequirements() self._writerequirements()
self._dirstatevalidatewarned = False self._dirstatevalidatewarned = False
self._branchcaches = {} self._branchcaches = {}
self._revbranchcache = None self._revbranchcache = None

mercurial/revlogstore.py

continue continue
if fl.iscensored(rev): if fl.iscensored(rev):
yield 'censored', path, node, None yield 'censored', path, node, None
continue continue
data = fl.read(node) data = fl.read(node)
yield 'ok', path, node, data yield 'ok', path, node, data
def redisfiledatakey(path, node):
return b'filedata:%s:%s' % (path, node)
class redisacceleratedrevlogfilesstore(repository.basefilesstore):
""""A filesstore that can use a redis server to speed up operations."""
def __init__(self, redis, basestore):
self._redis = redis
self._basestore = basestore
def resolvefilesdata(self, entries):
# Our strategy is to batch requests to redis because this is faster
# than a command for every entry.
batch = []
for i, entry in enumerate(entries):
batch.append(entry)
if i and not i % 1000:
for res in self._processfiledatabatch(batch):
yield res
batch = []
if batch:
for res in self._processfiledatabatch(batch):
yield res
def _processfiledatabatch(self, batch):
keys = [redisfiledatakey(path, node) for path, node in batch]
missing = []
for i, redisdata in enumerate(self._redis.mget(keys)):
path, node = batch[i]
if redisdata is None:
missing.append((path, node))
else:
yield 'ok', path, node, redisdata
# Now resolve all the missing data from the base store.
for res, path, node, data in self._basestore.resolvefilesdata(missing):
yield res, path, node, data
# Don't forget to cache it!
if res == 'ok':
self._redis.set(redisfiledatakey(path, node), data)