This is an archive of the discontinued Mercurial Phabricator instance.

Differential D519

remotefilelog: keepset tuple bug fix
ClosedPublic

Authored by ms2316 on Aug 25 2017, 11:02 AM.
Tags
None
Subscribers
ryanmce

Details

Reviewers
simonfar
durham
ryanmce
Group Reviewers
Restricted Project
Commits
rFBHGX72fc021bb146: remotefilelog: keepset tuple bug fix
Summary

Keepset computation algorithm added tuples to the keepset that are the results
of diffs between two manifests. Now it has been fixed and only new node from
the tuple is added to the keepset.

Test Plan
  • Run hg gc on www, fbsource
  • Ensure the unit tests pass

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ms2316 created this revision.Aug 25 2017, 11:02 AM
Herald added a reviewer: Restricted Project. · View Herald TranscriptAug 25 2017, 11:02 AM
ryanmce requested changes to this revision.Aug 25 2017, 11:05 AM
ryanmce added a subscriber: ryanmce.

This needs a test to prevent regressions.

This revision now requires changes to proceed.Aug 25 2017, 11:05 AM
ms2316 edited edge metadata.Aug 25 2017, 12:45 PM
ms2316 updated this revision to Diff 1295.

Added a test case that ensures that when keepkeys are populated from a diff between two manifests, nothing breaks. Also ensures that when keepkeys are populated
from manifest deltas nothing breaks as well.

ryanmce requested changes to this revision.Aug 25 2017, 12:47 PM

Sweet, thanks for the test case. I don't think we should be setting PYTHONPATH though?

tests/test-remotefilelog-keepset.t
2

This looks sketchy... why are you doing it?

This revision now requires changes to proceed.Aug 25 2017, 12:47 PM
ryanmce accepted this revision.Aug 25 2017, 12:54 PM

Ah, it's used in other tests. Maybe @durham knows why? Go ahead and land it.

This revision is now accepted and ready to land.Aug 25 2017, 12:54 PM
Closed by commit rFBHGX72fc021bb146: remotefilelog: keepset tuple bug fix (authored by ms2316). · Explain WhyAug 29 2017, 12:00 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 1399

remotefilelog/__init__.py

], _('hg debughistorypack <path>'), norepo=True) ], _('hg debughistorypack <path>'), norepo=True)
def debughistorypack(ui, path, **opts): def debughistorypack(ui, path, **opts):
return debugcommands.debughistorypack(ui, path) return debugcommands.debughistorypack(ui, path)
@command('debugkeepset', [ @command('debugkeepset', [
], _('hg debugkeepset')) ], _('hg debugkeepset'))
def debugkeepset(ui, repo, **opts): def debugkeepset(ui, repo, **opts):
# The command is used to measure keepset computation time # The command is used to measure keepset computation time
repackmod.keepset(repo) def keyfn(fname, fnode):
return fileserverclient.getcachekey(repo.name, fname, hex(fnode))
repackmod.keepset(repo, keyfn)
return return
@command('debugwaitonrepack', [ @command('debugwaitonrepack', [
], _('hg debugwaitonrepack')) ], _('hg debugwaitonrepack'))
def debugwaitonrepack(ui, repo, **opts): def debugwaitonrepack(ui, repo, **opts):
return debugcommands.debugwaitonrepack(repo) return debugcommands.debugwaitonrepack(repo)
@command('debugwaitonprefetch', [ @command('debugwaitonprefetch', [

remotefilelog/repack.py

if lastmanifest: if lastmanifest:
m = repo[r].manifest().diff(lastmanifest) m = repo[r].manifest().diff(lastmanifest)
else: else:
m = repo[r].manifest() m = repo[r].manifest()
lastmanifest = repo[r].manifest() lastmanifest = repo[r].manifest()
processed.add(r) processed.add(r)
# populate keepkeys with keys from the current manifest # populate keepkeys with keys from the current manifest
if type(m) is dict:
# m is a result of diff of two manifests and is a dictionary that
# maps filename to ((newnode, newflag), (oldnode, oldflag)) tuple
for filename, diff in m.iteritems():
if diff[0][0] is not None:
keepkeys.add(keyfn(filename, diff[0][0]))
else:
# m is a manifest object
for filename, filenode in m.iteritems(): for filename, filenode in m.iteritems():
keepkeys.add(keyfn(filename, filenode)) keepkeys.add(keyfn(filename, filenode))
return keepkeys return keepkeys
class repacker(object): class repacker(object):
"""Class for orchestrating the repack of data and history information into a """Class for orchestrating the repack of data and history information into a
new format. new format.
""" """
def __init__(self, repo, data, history, category, gc=False, isold=None): def __init__(self, repo, data, history, category, gc=False, isold=None):

tests/test-remotefilelog-keepset.t

  • This file was added.
$ PYTHONPATH=$TESTDIR/..:$PYTHONPATH
$ export PYTHONPATH
ryanmceUnsubmitted
Not Done

This looks sketchy... why are you doing it?

ryanmce: This looks sketchy... why are you doing it?
$ . "$TESTDIR/library.sh"
$ hginit master
$ cd master
$ cat >> .hg/hgrc <<EOF
> [remotefilelog]
> server=True
> serverexpiration=-1
> EOF
$ echo x > x
$ hg commit -qAm x
$ echo y > y
$ hg commit -qAm y
$ echo z > z
$ hg commit -qAm z
$ cd ..
$ hgcloneshallow ssh://user@dummy/master shallow -q
3 files fetched over 1 fetches - (3 misses, 0.00% hit ratio) over *s (glob)
# Compute keepset for 0th and 2nd commit, which implies that we do not process
# the 1st commit, therefore we diff 2nd manifest with the 0th manifest and
# populate the keepkeys from the diff
$ cd shallow
$ cat >> .hg/hgrc <<EOF
> [remotefilelog]
> pullprefetch=0+2
> EOF
$ hg debugkeepset
# Compute keepset for all commits, which implies that we only process deltas of
# manifests of commits 1 and 2 and therefore populate the keepkeys from deltas
$ cat >> .hg/hgrc <<EOF
> [remotefilelog]
> pullprefetch=all()
> EOF
$ hg debugkeepset