This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4502

util: allow lrucachedict to track cost of entries
ClosedPublic

Authored by indygreg on Sep 6 2018, 9:17 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHGee087f0d7db5: util: allow lrucachedict to track cost of entries
Summary

Currently, lrucachedict allows tracking of arbitrary items with the
only limit being the total number of items in the cache.

Caches can be a lot more useful when they are bound by the size
of the items in them rather than the number of elements in the
cache.

In preparation for teaching lrucachedict to enforce a max size of
cached items, we teach lrucachedict to optionally associate a numeric
cost value with each node.

We purposefully let the caller define their own cost for nodes.

This does introduce some overhead. Most of it comes from setitem,
since that function now calls into insert(), thus introducing Python
function call overhead.

$ hg perflrucachedict --size 4 --gets 1000000 --sets 1000000 --mixed 1000000
! gets
! wall 0.599552 comb 0.600000 user 0.600000 sys 0.000000 (best of 17)
! wall 0.614643 comb 0.610000 user 0.610000 sys 0.000000 (best of 17)
! inserts
! <not available>
! wall 0.655817 comb 0.650000 user 0.650000 sys 0.000000 (best of 16)
! sets
! wall 0.540448 comb 0.540000 user 0.540000 sys 0.000000 (best of 18)
! wall 0.805644 comb 0.810000 user 0.810000 sys 0.000000 (best of 13)
! mixed
! wall 0.651556 comb 0.660000 user 0.660000 sys 0.000000 (best of 15)
! wall 0.781357 comb 0.780000 user 0.780000 sys 0.000000 (best of 13)

$ hg perflrucachedict --size 1000 --gets 1000000 --sets 1000000 --mixed 1000000
! gets
! wall 0.621014 comb 0.620000 user 0.620000 sys 0.000000 (best of 16)
! wall 0.615146 comb 0.620000 user 0.620000 sys 0.000000 (best of 17)
! inserts
! <not available>
! wall 0.698115 comb 0.700000 user 0.700000 sys 0.000000 (best of 15)
! sets
! wall 0.560247 comb 0.560000 user 0.560000 sys 0.000000 (best of 18)
! wall 0.832495 comb 0.830000 user 0.830000 sys 0.000000 (best of 12)
! mixed
! wall 0.686172 comb 0.680000 user 0.680000 sys 0.000000 (best of 15)
! wall 0.841359 comb 0.840000 user 0.840000 sys 0.000000 (best of 12)

We're still under 1us per insert, which seems like reasonable
performance for a cache.

If we comment out updating of self.totalcost during insert(),
performance of insert() is identical to setitem before. However,
I don't want to make total cost evaluation lazy because it has
significant performance implications for when we need to evaluate the
total cost at mutation time (it requires a cache traversal, which could
be expensive for large caches).

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

indygreg created this revision.Sep 6 2018, 9:17 PM
lothiraldan added a subscriber: lothiraldan.Sep 7 2018, 2:48 PM
lothiraldan added inline comments.
mercurial/util.py
1277

I'm not sure this line is tested, I didnd't see a test where we replace an entry with an associated cost

indygreg planned changes to this revision.Sep 7 2018, 3:12 PM
indygreg added inline comments.
mercurial/util.py
1277

Good catch! I'll send a revised patch.

FWIW, cost accounting on this data structure opens up a lot of potential around caching on revlogs. I have some alpha-quality commits to replace the full revision cache on the revlog with an lrucachedict and to add a decompressed chunk cache to revlogs. Such caches can speed up certain operations drastically. However, we need to be careful about adding always-on caches to revlogs because they can result in memory bloat. I was thinking about adding context manager methods to revlogs to temporarily activate certain aggressive caches in order to facilitate certain operations. e.g. a fulltext or chunk cache when applying delta groups could make it drastically faster to compute deltas during bulk insertion. A chunk cache could make reverse walks significantly faster. Etc. I figured you'd be interested given recent work in this area :)

indygreg updated this revision to Diff 10832.Sep 7 2018, 3:14 PM
lothiraldan added inline comments.Sep 8 2018, 8:36 AM
mercurial/util.py
1277

Having a weighted cache would combine well with our work on intermediate snapshots. If we can keep the right intermediate snapshots in the cache we will get a lot of useful cache hit.

Closed by commit rHGee087f0d7db5: util: allow lrucachedict to track cost of entries (authored by indygreg). · Explain WhySep 12 2018, 10:52 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mcontrib/perf.py (6 lines)
Mmercurial/util.py (27 lines)
Mtests/test-lrucachedict.py (42 lines)

Diff 10832

contrib/perf.py

value = d[key] value = d[key]
value # silence pyflakes warning value # silence pyflakes warning
# Set mode tests insertion speed with cache eviction. # Set mode tests insertion speed with cache eviction.
setseq = [] setseq = []
for i in xrange(sets): for i in xrange(sets):
setseq.append(random.randint(0, sys.maxint)) setseq.append(random.randint(0, sys.maxint))
def doinserts():
d = util.lrucachedict(size)
for v in setseq:
d.insert(v, v)
def dosets(): def dosets():
d = util.lrucachedict(size) d = util.lrucachedict(size)
for v in setseq: for v in setseq:
d[v] = v d[v] = v
# Mixed mode randomly performs gets and sets with eviction. # Mixed mode randomly performs gets and sets with eviction.
mixedops = [] mixedops = []
for i in xrange(mixed): for i in xrange(mixed):
except KeyError: except KeyError:
pass pass
else: else:
d[v] = v d[v] = v
benches = [ benches = [
(doinit, b'init'), (doinit, b'init'),
(dogets, b'gets'), (dogets, b'gets'),
(doinserts, b'inserts'),
(dosets, b'sets'), (dosets, b'sets'),
(domixed, b'mixed') (domixed, b'mixed')
] ]
for fn, title in benches: for fn, title in benches:
timer, fm = gettimer(ui, opts) timer, fm = gettimer(ui, opts)
timer(fn, title=title) timer(fn, title=title)
fm.end() fm.end()

mercurial/util.py

yield yield
class _lrucachenode(object): class _lrucachenode(object):
"""A node in a doubly linked list. """A node in a doubly linked list.
Holds a reference to nodes on either side as well as a key-value Holds a reference to nodes on either side as well as a key-value
pair for the dictionary entry. pair for the dictionary entry.
""" """
__slots__ = (u'next', u'prev', u'key', u'value') __slots__ = (u'next', u'prev', u'key', u'value', u'cost')
def __init__(self): def __init__(self):
self.next = None self.next = None
self.prev = None self.prev = None
self.key = _notset self.key = _notset
self.value = None self.value = None
self.cost = 0
def markempty(self): def markempty(self):
"""Mark the node as emptied.""" """Mark the node as emptied."""
self.key = _notset self.key = _notset
self.value = None
self.cost = 0
class lrucachedict(object): class lrucachedict(object):
"""Dict that caches most recent accesses and sets. """Dict that caches most recent accesses and sets.
The dict consists of an actual backing dict - indexed by original The dict consists of an actual backing dict - indexed by original
key - and a doubly linked circular list defining the order of entries in key - and a doubly linked circular list defining the order of entries in
the cache. the cache.
The head node is the newest entry in the cache. If the cache is full, The head node is the newest entry in the cache. If the cache is full,
we recycle head.prev and make it the new head. Cache accesses result in we recycle head.prev and make it the new head. Cache accesses result in
the node being moved to before the existing head and being marked as the the node being moved to before the existing head and being marked as the
new head node. new head node.
Items in the cache can be inserted with an optional "cost" value. This is
simply an integer that is specified by the caller. The cache can be queried
for the total cost of all items presently in the cache.
""" """
def __init__(self, max): def __init__(self, max):
self._cache = {} self._cache = {}
self._head = head = _lrucachenode() self._head = head = _lrucachenode()
head.prev = head head.prev = head
head.next = head head.next = head
self._size = 1 self._size = 1
self.capacity = max self.capacity = max
self.totalcost = 0
def __len__(self): def __len__(self):
return len(self._cache) return len(self._cache)
def __contains__(self, k): def __contains__(self, k):
return k in self._cache return k in self._cache
def __iter__(self): def __iter__(self):
# We don't have to iterate in cache order, but why not. # We don't have to iterate in cache order, but why not.
n = self._head n = self._head
for i in range(len(self._cache)): for i in range(len(self._cache)):
yield n.key yield n.key
n = n.next n = n.next
def __getitem__(self, k): def __getitem__(self, k):
node = self._cache[k] node = self._cache[k]
self._movetohead(node) self._movetohead(node)
return node.value return node.value
def __setitem__(self, k, v): def insert(self, k, v, cost=0):
"""Insert a new item in the cache with optional cost value."""
node = self._cache.get(k) node = self._cache.get(k)
# Replace existing value and mark as newest. # Replace existing value and mark as newest.
if node is not None: if node is not None:
self.totalcost -= node.cost
lothiraldanUnsubmitted
Not Done

I'm not sure this line is tested, I didnd't see a test where we replace an entry with an associated cost

lothiraldan: I'm not sure this line is tested, I didnd't see a test where we replace an entry with an…
indygregAuthorUnsubmitted
Not Done

Good catch! I'll send a revised patch.

FWIW, cost accounting on this data structure opens up a lot of potential around caching on revlogs. I have some alpha-quality commits to replace the full revision cache on the revlog with an lrucachedict and to add a decompressed chunk cache to revlogs. Such caches can speed up certain operations drastically. However, we need to be careful about adding always-on caches to revlogs because they can result in memory bloat. I was thinking about adding context manager methods to revlogs to temporarily activate certain aggressive caches in order to facilitate certain operations. e.g. a fulltext or chunk cache when applying delta groups could make it drastically faster to compute deltas during bulk insertion. A chunk cache could make reverse walks significantly faster. Etc. I figured you'd be interested given recent work in this area :)

indygreg: Good catch! I'll send a revised patch. FWIW, cost accounting on this data structure opens up a…
lothiraldanUnsubmitted
Not Done

Having a weighted cache would combine well with our work on intermediate snapshots. If we can keep the right intermediate snapshots in the cache we will get a lot of useful cache hit.

lothiraldan: Having a weighted cache would combine well with our work on intermediate snapshots. If we can…
node.value = v node.value = v
node.cost = cost
self.totalcost += cost
self._movetohead(node) self._movetohead(node)
return return
if self._size < self.capacity: if self._size < self.capacity:
node = self._addcapacity() node = self._addcapacity()
else: else:
# Grab the last/oldest item. # Grab the last/oldest item.
node = self._head.prev node = self._head.prev
# At capacity. Kill the old entry. # At capacity. Kill the old entry.
if node.key is not _notset: if node.key is not _notset:
self.totalcost -= node.cost
del self._cache[node.key] del self._cache[node.key]
node.key = k node.key = k
node.value = v node.value = v
node.cost = cost
self.totalcost += cost
self._cache[k] = node self._cache[k] = node
# And mark it as newest entry. No need to adjust order since it # And mark it as newest entry. No need to adjust order since it
# is already self._head.prev. # is already self._head.prev.
self._head = node self._head = node
def __setitem__(self, k, v):
self.insert(k, v)
def __delitem__(self, k): def __delitem__(self, k):
node = self._cache.pop(k) node = self._cache.pop(k)
self.totalcost -= node.cost
node.markempty() node.markempty()
# Temporarily mark as newest item before re-adjusting head to make # Temporarily mark as newest item before re-adjusting head to make
# this node the oldest item. # this node the oldest item.
self._movetohead(node) self._movetohead(node)
self._head = node.next self._head = node.next
# Additional dict methods. # Additional dict methods.
def get(self, k, default=None): def get(self, k, default=None):
try: try:
return self._cache[k].value return self._cache[k].value
except KeyError: except KeyError:
return default return default
def clear(self): def clear(self):
n = self._head n = self._head
while n.key is not _notset: while n.key is not _notset:
self.totalcost -= n.cost
n.markempty() n.markempty()
n = n.next n = n.next
self._cache.clear() self._cache.clear()
def copy(self, capacity=None): def copy(self, capacity=None):
"""Create a new cache as a copy of the current one. """Create a new cache as a copy of the current one.
# Find the first non-empty entry. # Find the first non-empty entry.
n = self._head.prev n = self._head.prev
while n.key is _notset and n is not self._head: while n.key is _notset and n is not self._head:
n = n.prev n = n.prev
# We could potentially skip the first N items when decreasing capacity. # We could potentially skip the first N items when decreasing capacity.
# But let's keep it simple unless it is a performance problem. # But let's keep it simple unless it is a performance problem.
for i in range(len(self._cache)): for i in range(len(self._cache)):
result[n.key] = n.value result.insert(n.key, n.value, cost=n.cost)
n = n.prev n = n.prev
return result return result
def popoldest(self): def popoldest(self):
"""Remove the oldest item from the cache. """Remove the oldest item from the cache.
Returns the (key, value) describing the removed cache entry. Returns the (key, value) describing the removed cache entry.
""" """
if not self._cache: if not self._cache:
return return
# Walk the linked list backwards starting at tail node until we hit # Walk the linked list backwards starting at tail node until we hit
# a non-empty node. # a non-empty node.
n = self._head.prev n = self._head.prev
while n.key is _notset: while n.key is _notset:
n = n.prev n = n.prev
key, value = n.key, n.value key, value = n.key, n.value
# And remove it from the cache and mark it as empty. # And remove it from the cache and mark it as empty.
del self._cache[n.key] del self._cache[n.key]
self.totalcost -= n.cost
n.markempty() n.markempty()
return key, value return key, value
def _movetohead(self, node): def _movetohead(self, node):
"""Mark a node as the newest, making it the new head. """Mark a node as the newest, making it the new head.
When a node is accessed, it becomes the freshest entry in the LRU When a node is accessed, it becomes the freshest entry in the LRU

tests/test-lrucachedict.py

from __future__ import absolute_import, print_function from __future__ import absolute_import, print_function
import unittest import unittest
import silenttestrunner import silenttestrunner
from mercurial import ( from mercurial import (
util, util,
) )
class testlrucachedict(unittest.TestCase): class testlrucachedict(unittest.TestCase):
def testsimple(self): def testsimple(self):
d = util.lrucachedict(4) d = util.lrucachedict(4)
self.assertEqual(d.capacity, 4) self.assertEqual(d.capacity, 4)
d['a'] = 'va' d.insert('a', 'va', cost=2)
d['b'] = 'vb' d['b'] = 'vb'
d['c'] = 'vc' d['c'] = 'vc'
d['d'] = 'vd' d.insert('d', 'vd', cost=42)
self.assertEqual(d['a'], 'va') self.assertEqual(d['a'], 'va')
self.assertEqual(d['b'], 'vb') self.assertEqual(d['b'], 'vb')
self.assertEqual(d['c'], 'vc') self.assertEqual(d['c'], 'vc')
self.assertEqual(d['d'], 'vd') self.assertEqual(d['d'], 'vd')
self.assertEqual(d.totalcost, 44)
# 'a' should be dropped because it was least recently used. # 'a' should be dropped because it was least recently used.
d['e'] = 've' d['e'] = 've'
self.assertNotIn('a', d) self.assertNotIn('a', d)
self.assertIsNone(d.get('a')) self.assertIsNone(d.get('a'))
self.assertEqual(d.totalcost, 42)
self.assertEqual(d['b'], 'vb') self.assertEqual(d['b'], 'vb')
self.assertEqual(d['c'], 'vc') self.assertEqual(d['c'], 'vc')
self.assertEqual(d['d'], 'vd') self.assertEqual(d['d'], 'vd')
self.assertEqual(d['e'], 've') self.assertEqual(d['e'], 've')
# Replacing item with different cost adjusts totalcost.
d.insert('e', 've', cost=4)
self.assertEqual(d.totalcost, 46)
# Touch entries in some order (both get and set). # Touch entries in some order (both get and set).
d['e'] d['e']
d['c'] = 'vc2' d['c'] = 'vc2'
d['d'] d['d']
d['b'] = 'vb2' d['b'] = 'vb2'
# 'e' should be dropped now # 'e' should be dropped now
d['f'] = 'vf' d['f'] = 'vf'
d['a'] d['a']
d['b'] d['b']
for key in ('a', 'b'): for key in ('a', 'b'):
self.assertIn(key, d) self.assertIn(key, d)
def testcopypartial(self): def testcopypartial(self):
d = util.lrucachedict(4) d = util.lrucachedict(4)
d['a'] = 'va' d.insert('a', 'va', cost=4)
d['b'] = 'vb' d.insert('b', 'vb', cost=2)
dc = d.copy() dc = d.copy()
self.assertEqual(len(dc), 2) self.assertEqual(len(dc), 2)
self.assertEqual(dc.totalcost, 6)
for key in ('a', 'b'): for key in ('a', 'b'):
self.assertIn(key, dc) self.assertIn(key, dc)
self.assertEqual(dc[key], 'v%s' % key) self.assertEqual(dc[key], 'v%s' % key)
self.assertEqual(len(d), 2) self.assertEqual(len(d), 2)
for key in ('a', 'b'): for key in ('a', 'b'):
self.assertIn(key, d) self.assertIn(key, d)
self.assertEqual(d[key], 'v%s' % key) self.assertEqual(d[key], 'v%s' % key)
d['c'] = 'vc' d['c'] = 'vc'
del d['b'] del d['b']
self.assertEqual(d.totalcost, 4)
dc = d.copy() dc = d.copy()
self.assertEqual(len(dc), 2) self.assertEqual(len(dc), 2)
self.assertEqual(dc.totalcost, 4)
for key in ('a', 'c'): for key in ('a', 'c'):
self.assertIn(key, dc) self.assertIn(key, dc)
self.assertEqual(dc[key], 'v%s' % key) self.assertEqual(dc[key], 'v%s' % key)
def testcopyempty(self): def testcopyempty(self):
d = util.lrucachedict(4) d = util.lrucachedict(4)
dc = d.copy() dc = d.copy()
self.assertEqual(len(dc), 0) self.assertEqual(len(dc), 0)
def testcopyfull(self): def testcopyfull(self):
d = util.lrucachedict(4) d = util.lrucachedict(4)
d['a'] = 'va' d.insert('a', 'va', cost=42)
d['b'] = 'vb' d['b'] = 'vb'
d['c'] = 'vc' d['c'] = 'vc'
d['d'] = 'vd' d['d'] = 'vd'
dc = d.copy() dc = d.copy()
for key in ('a', 'b', 'c', 'd'): for key in ('a', 'b', 'c', 'd'):
self.assertIn(key, dc) self.assertIn(key, dc)
self.assertEqual(dc[key], 'v%s' % key) self.assertEqual(dc[key], 'v%s' % key)
self.assertEqual(d.totalcost, 42)
self.assertEqual(dc.totalcost, 42)
# 'a' should be dropped because it was least recently used. # 'a' should be dropped because it was least recently used.
dc['e'] = 've' dc['e'] = 've'
self.assertNotIn('a', dc) self.assertNotIn('a', dc)
for key in ('b', 'c', 'd', 'e'): for key in ('b', 'c', 'd', 'e'):
self.assertIn(key, dc) self.assertIn(key, dc)
self.assertEqual(dc[key], 'v%s' % key) self.assertEqual(dc[key], 'v%s' % key)
self.assertEqual(d.totalcost, 42)
self.assertEqual(dc.totalcost, 0)
# Contents and order of original dict should remain unchanged. # Contents and order of original dict should remain unchanged.
dc['b'] = 'vb_new' dc['b'] = 'vb_new'
self.assertEqual(list(iter(d)), ['d', 'c', 'b', 'a']) self.assertEqual(list(iter(d)), ['d', 'c', 'b', 'a'])
for key in ('a', 'b', 'c', 'd'): for key in ('a', 'b', 'c', 'd'):
self.assertEqual(d[key], 'v%s' % key) self.assertEqual(d[key], 'v%s' % key)
def testcopydecreasecapacity(self): def testcopydecreasecapacity(self):
d = util.lrucachedict(5) d = util.lrucachedict(5)
d['a'] = 'va' d.insert('a', 'va', cost=4)
d['b'] = 'vb' d.insert('b', 'vb', cost=2)
d['c'] = 'vc' d['c'] = 'vc'
d['d'] = 'vd' d['d'] = 'vd'
dc = d.copy(2) dc = d.copy(2)
self.assertEqual(dc.totalcost, 0)
for key in ('a', 'b'): for key in ('a', 'b'):
self.assertNotIn(key, dc) self.assertNotIn(key, dc)
for key in ('c', 'd'): for key in ('c', 'd'):
self.assertIn(key, dc) self.assertIn(key, dc)
self.assertEqual(dc[key], 'v%s' % key) self.assertEqual(dc[key], 'v%s' % key)
dc['e'] = 've' dc.insert('e', 've', cost=7)
self.assertEqual(dc.totalcost, 7)
self.assertNotIn('c', dc) self.assertNotIn('c', dc)
for key in ('d', 'e'): for key in ('d', 'e'):
self.assertIn(key, dc) self.assertIn(key, dc)
self.assertEqual(dc[key], 'v%s' % key) self.assertEqual(dc[key], 'v%s' % key)
# Original should remain unchanged. # Original should remain unchanged.
self.assertEqual(d.totalcost, 6)
for key in ('a', 'b', 'c', 'd'): for key in ('a', 'b', 'c', 'd'):
self.assertIn(key, d) self.assertIn(key, d)
self.assertEqual(d[key], 'v%s' % key) self.assertEqual(d[key], 'v%s' % key)
def testcopyincreasecapacity(self): def testcopyincreasecapacity(self):
d = util.lrucachedict(5) d = util.lrucachedict(5)
d['a'] = 'va' d['a'] = 'va'
d['b'] = 'vb' d['b'] = 'vb'
# Original should remain unchanged. # Original should remain unchanged.
for key in ('a', 'b', 'c', 'd'): for key in ('a', 'b', 'c', 'd'):
self.assertIn(key, d) self.assertIn(key, d)
self.assertEqual(d[key], 'v%s' % key) self.assertEqual(d[key], 'v%s' % key)
def testpopoldest(self): def testpopoldest(self):
d = util.lrucachedict(4) d = util.lrucachedict(4)
d['a'] = 'va' d.insert('a', 'va', cost=10)
d['b'] = 'vb' d.insert('b', 'vb', cost=5)
self.assertEqual(len(d), 2) self.assertEqual(len(d), 2)
self.assertEqual(d.popoldest(), ('a', 'va')) self.assertEqual(d.popoldest(), ('a', 'va'))
self.assertEqual(len(d), 1) self.assertEqual(len(d), 1)
self.assertEqual(d.totalcost, 5)
self.assertEqual(d.popoldest(), ('b', 'vb')) self.assertEqual(d.popoldest(), ('b', 'vb'))
self.assertEqual(len(d), 0) self.assertEqual(len(d), 0)
self.assertEqual(d.totalcost, 0)
self.assertIsNone(d.popoldest()) self.assertIsNone(d.popoldest())
d['a'] = 'va' d['a'] = 'va'
d['b'] = 'vb' d['b'] = 'vb'
d['c'] = 'vc' d['c'] = 'vc'
d['d'] = 'vd' d['d'] = 'vd'
self.assertEqual(d.popoldest(), ('a', 'va')) self.assertEqual(d.popoldest(), ('a', 'va'))