Page MenuHomePhabricator

repack: delete excessively big packfiles before incremental repack
ClosedPublic

Authored by phillco on Nov 27 2017, 4:37 PM.

Details

Summary

Abnormally huge packfiles (20 GB+) are unwanted and can cause more problems
than they solve. Let's add a config to simply delete them at the beginning of
the repack proces.

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

phillco created this revision.Nov 27 2017, 4:37 PM
Herald added a reviewer: Restricted Project. · View Herald TranscriptNov 27 2017, 4:37 PM
phillco retitled this revision from repck: delete excessively big packfiles before incremental repack to repack: delete excessively big packfiles before incremental repack.Nov 27 2017, 4:38 PM
phillco updated this revision to Diff 3895.
durham accepted this revision.Nov 27 2017, 11:27 PM
durham added a subscriber: durham.
durham added inline comments.
remotefilelog/repack.py
156

I'd use ui.configbytes(...), that way users can specify values like '512M' and '20G'

184

I'd probably make this two lines for clarity.

files = osutil.listdir(packpath, stat=True)
files = _deletebigpacks(repo, packpath, files)
This revision is now accepted and ready to land.Nov 27 2017, 11:27 PM

Based on yesterday's conversation, do we need to limit this to remotefilelog for now? @simpkins

Probably safe to limit it to remotefilelog. We can expand it as needed.

singhsrb added inline comments.
remotefilelog/repack.py
289

You can even remove the big files at this point. Something like

if size > maxsize:
    # Delete the file
elif size > limit:
    # Use the current code.

Would reuse a lot of work.

phillco marked 2 inline comments as done.Nov 30 2017, 3:33 PM
phillco updated this revision to Diff 4016.
phillco added inline comments.Nov 30 2017, 3:34 PM
remotefilelog/repack.py
289

I think this is the size of the index file, not the data file.

phillco closed this revision.Nov 30 2017, 3:47 PM

Landed, but I could submit a rewrite to reuse the existing iteration points.