This is an archive of the discontinued Mercurial Phabricator instance.

Differential D141

infinitepush: remove some unused options
ClosedPublic

Authored by stash on Jul 19 2017, 6:11 AM.
Tags
None
Subscribers
ryanmce

Details

Reviewers
ryanmce
Group Reviewers
Restricted Project
Commits
rFBHGXd87fcdcd2cff: infinitepush: remove some unused options
Summary

Infinitepush has too many options and many of them are unused or has never
been changed from the default value. Let's remove some of them.

Test Plan

Run infinitepush tests

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Branch
default
Lint
Lint OK
Unit
Unit Test Errors

Event Timeline

stash created this revision.Jul 19 2017, 6:11 AM
ryanmce accepted this revision.Jul 19 2017, 6:47 AM
ryanmce added a subscriber: ryanmce.

Seems useful

This revision is now accepted and ready to land.Jul 19 2017, 6:47 AM
Closed by commit rFBHGXd87fcdcd2cff: infinitepush: remove some unused options (authored by stash). · Explain WhyJul 20 2017, 6:44 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Diff 296

infinitepush/__init__.py

# Server-side option. Used only if indextype=sql. # Server-side option. Used only if indextype=sql.
# Sets mysql innodb_lock_wait_timeout option. # Sets mysql innodb_lock_wait_timeout option.
locktimeout = 120 locktimeout = 120
# Server-side option. Used only if indextype=sql. # Server-side option. Used only if indextype=sql.
# Name of the repository # Name of the repository
reponame = '' reponame = ''
# Client-side option
pushbackuplog = FILE
# Client-side option. Used by --list-remote option. List of remote scratch # Client-side option. Used by --list-remote option. List of remote scratch
# patterns to list if no patterns are specified. # patterns to list if no patterns are specified.
defaultremotepatterns = ['*'] defaultremotepatterns = ['*']
# Server-side option. If bookmark that was pushed matches # Server-side option. If bookmark that was pushed matches
# `fillmetadatabranchpattern` then background # `fillmetadatabranchpattern` then background
# `hg debugfillinfinitepushmetadata` process will save metadata # `hg debugfillinfinitepushmetadata` process will save metadata
# in infinitepush index for nodes that are ancestor of the bookmark. # in infinitepush index for nodes that are ancestor of the bookmark.

infinitepush/backupcommands.py

# Copyright 2017 Facebook, Inc. # Copyright 2017 Facebook, Inc.
# #
# This software may be used and distributed according to the terms of the # This software may be used and distributed according to the terms of the
# GNU General Public License version 2 or any later version. # GNU General Public License version 2 or any later version.
""" """
[infinitepushbackup] [infinitepushbackup]
# path to the directory where pushback logs should be stored # path to the directory where pushback logs should be stored
logdir = path/to/dir logdir = path/to/dir
# max number of logs for one repo for one user
maxlognumber = 5
# There can be at most one backup process per repo. This config options
# determines how much time to wait on the lock. If timeout happens then
# backups process aborts.
waittimeout = 30
# Backup at most maxheadstobackup heads, other heads are ignored. # Backup at most maxheadstobackup heads, other heads are ignored.
# Negative number means backup everything. # Negative number means backup everything.
maxheadstobackup = -1 maxheadstobackup = -1
# With probability 1/N confirm if current backup is consistent. If this is
# zero or negative then no backup checks will be performed
backupcheckfreq = 50
# Previously there was a bug in infinitepush backup and it made separate # Previously there was a bug in infinitepush backup and it made separate
# backup for the same repo if pushbackup was called from different working # backup for the same repo if pushbackup was called from different working
# copies. This option cleans up the mess # copies. This option cleans up the mess
tempcleanworkingcopiesbackups = False tempcleanworkingcopiesbackups = False
# Nodes that should not be backed up. Ancestors of these nodes won't be # Nodes that should not be backed up. Ancestors of these nodes won't be
# backed up either # backed up either
dontbackupnodes = [] dontbackupnodes = []
# Special option that may be used to trigger re-backuping. For example, # Special option that may be used to trigger re-backuping. For example,
# if there was a bug in infinitepush backups, then changing the value of # if there was a bug in infinitepush backups, then changing the value of
# this option will force all clients to make a "clean" backup # this option will force all clients to make a "clean" backup
backupgeneration = 0 backupgeneration = 0
# Hostname value to use. If not specified then socket.gethostname() will # Hostname value to use. If not specified then socket.gethostname() will
# be used # be used
hostname = '' hostname = ''
""" """
from __future__ import absolute_import from __future__ import absolute_import
import errno import errno
import json import json
import os import os
import random
import re import re
import socket import socket
import stat import stat
import subprocess import subprocess
import time import time
from .bundleparts import ( from .bundleparts import (
getscratchbookmarkspart, getscratchbookmarkspart,
userlogdir = os.path.join(logdir, username) userlogdir = os.path.join(logdir, username)
util.makedirs(userlogdir) util.makedirs(userlogdir)
if not _checkuserlogdir(userlogdir): if not _checkuserlogdir(userlogdir):
raise WrongPermissionsException(userlogdir) raise WrongPermissionsException(userlogdir)
reporoot = repo.origroot reporoot = repo.origroot
reponame = os.path.basename(reporoot) reponame = os.path.basename(reporoot)
_removeoldlogfiles(userlogdir, reponame)
maxlogfilenumber = ui.configint('infinitepushbackup',
'maxlognumber', 5)
_removeoldlogfiles(userlogdir, reponame, maxlogfilenumber)
logfile = _getlogfilename(logdir, username, reponame) logfile = _getlogfilename(logdir, username, reponame)
except (OSError, IOError) as e: except (OSError, IOError) as e:
ui.debug('infinitepush backup log is disabled: %s\n' % e) ui.debug('infinitepush backup log is disabled: %s\n' % e)
except WrongPermissionsException as e: except WrongPermissionsException as e:
ui.debug(('%s directory has incorrect permission, ' + ui.debug(('%s directory has incorrect permission, ' +
'infinitepush backup logging will be disabled\n') % 'infinitepush backup logging will be disabled\n') %
e.logdir) e.logdir)
finally: finally:
os.umask(oldumask) os.umask(oldumask)
if not logfile: if not logfile:
logfile = os.devnull logfile = os.devnull
with open(logfile, 'a') as f: with open(logfile, 'a') as f:
subprocess.Popen(background_cmd, shell=False, stdout=f, subprocess.Popen(background_cmd, shell=False, stdout=f,
stderr=subprocess.STDOUT) stderr=subprocess.STDOUT)
return 0 return 0
try: try:
timeout = ui.configint('infinitepushbackup', 'waittimeout', 30) # Wait at most 30 seconds, because that's the average backup time
timeout = 30
srcrepo = _getsrcrepo(repo) srcrepo = _getsrcrepo(repo)
with lockmod.lock(srcrepo.vfs, _backuplockname, timeout=timeout): with lockmod.lock(srcrepo.vfs, _backuplockname, timeout=timeout):
backupcheckfreq = ui.configint('infinitepushbackup',
'backupcheckfreq', 50)
if backupcheckfreq > 0 and random.randrange(backupcheckfreq) == 0:
bkpstate = _readlocalbackupstate(ui, srcrepo)
if not _dobackupcheck(bkpstate, ui, srcrepo, dest, **opts):
ui.log('infinitepushbackup',
'failed infinitepush backup check',
infinitepushbackupcheck='failure')
return _dobackup(ui, repo, dest, **opts) return _dobackup(ui, repo, dest, **opts)
except error.LockHeld as e: except error.LockHeld as e:
if e.errno == errno.ETIMEDOUT: if e.errno == errno.ETIMEDOUT:
ui.warn(_('timeout waiting on backup lock\n')) ui.warn(_('timeout waiting on backup lock\n'))
ui.log('infinitepushbackup', 'timeout waiting on backup lock',
infinitepushbackuplock='timeout')
return 0 return 0
else: else:
raise raise
@command('pullbackup', restoreoptions) @command('pullbackup', restoreoptions)
def restore(ui, repo, dest=None, **opts): def restore(ui, repo, dest=None, **opts):
""" """
Pulls commits from infinitepush that were previously saved with Pulls commits from infinitepush that were previously saved with
consists of reponame (basename of repo path) and current day consists of reponame (basename of repo path) and current day
(see _timeformat). That means that two different repos with the same name (see _timeformat). That means that two different repos with the same name
can share the same log file. This is not a big problem so we ignore it. can share the same log file. This is not a big problem so we ignore it.
''' '''
currentday = time.strftime(_timeformat) currentday = time.strftime(_timeformat)
return os.path.join(logdir, username, reponame + currentday) return os.path.join(logdir, username, reponame + currentday)
def _removeoldlogfiles(userlogdir, reponame, maxlogfilenumber): def _removeoldlogfiles(userlogdir, reponame):
existinglogfiles = [] existinglogfiles = []
for entry in osutil.listdir(userlogdir): for entry in osutil.listdir(userlogdir):
filename = entry[0] filename = entry[0]
fullpath = os.path.join(userlogdir, filename) fullpath = os.path.join(userlogdir, filename)
if filename.startswith(reponame) and os.path.isfile(fullpath): if filename.startswith(reponame) and os.path.isfile(fullpath):
try: try:
time.strptime(filename[len(reponame):], _timeformat) time.strptime(filename[len(reponame):], _timeformat)
except ValueError: except ValueError:
continue continue
existinglogfiles.append(filename) existinglogfiles.append(filename)
# _timeformat gives us a property that if we sort log file names in # _timeformat gives us a property that if we sort log file names in
# descending order then newer files are going to be in the beginning # descending order then newer files are going to be in the beginning
existinglogfiles = sorted(existinglogfiles, reverse=True) existinglogfiles = sorted(existinglogfiles, reverse=True)
# Delete logs that are older than 5 days
maxlogfilenumber = 5
if len(existinglogfiles) > maxlogfilenumber: if len(existinglogfiles) > maxlogfilenumber:
for filename in existinglogfiles[maxlogfilenumber:]: for filename in existinglogfiles[maxlogfilenumber:]:
os.unlink(os.path.join(userlogdir, filename)) os.unlink(os.path.join(userlogdir, filename))
def _checkcommonlogdir(logdir): def _checkcommonlogdir(logdir):
'''Checks permissions of the log directory '''Checks permissions of the log directory
We want log directory to actually be a directory, have restricting We want log directory to actually be a directory, have restricting

tests/test-infinitepush-pullbackup.t

$ hg clone ssh://user@dummy/repo restored -q $ hg clone ssh://user@dummy/repo restored -q
Backup Backup
$ cd backupsource $ cd backupsource
$ mkcommit firstcommit $ mkcommit firstcommit
$ hg book abook $ hg book abook
Actually do a backup, make sure that backup check doesn't fail for empty backup state Actually do a backup, make sure that backup check doesn't fail for empty backup state
$ hg pushbackup --config infinitepushbackup.backupcheckfreq=1 $ hg pushbackup
starting backup .* (re) starting backup .* (re)
searching for changes searching for changes
remote: pushing 1 commit: remote: pushing 1 commit:
remote: 89ecc969c0ac firstcommit remote: 89ecc969c0ac firstcommit
finished in \d+\.(\d+)? seconds (re) finished in \d+\.(\d+)? seconds (re)
$ cd .. $ cd ..
Restore Restore
remote: c03baa769a20 newcommit remote: c03baa769a20 newcommit
finished in \d+\.(\d+)? seconds (re) finished in \d+\.(\d+)? seconds (re)
$ cd ../backupsource $ cd ../backupsource
$ hg debugcheckbackup --user anotheruser --all | sort $ hg debugcheckbackup --user anotheruser --all | sort
unknown revision 'e0230a60975b38a9014f098fb973199efd25c46f' unknown revision 'e0230a60975b38a9014f098fb973199efd25c46f'
checking \$TESTTMP/backupsource on .* (re) checking \$TESTTMP/backupsource on .* (re)
checking \$TESTTMP/backupsource2 on .* (re) checking \$TESTTMP/backupsource2 on .* (re)
Check infinitepushbackup.backupcheckfreq config option
$ HGUSER=anotheruser hg pushbackup --config infinitepushbackup.backupcheckfreq=1
unknown revision 'e0230a60975b38a9014f098fb973199efd25c46f'
starting backup .* (re)
nothing to backup
finished in \d+\.(\d+)? seconds (re)
$ HGUSER=anotheruser hg pushbackup --config infinitepushbackup.backupcheckfreq=0
starting backup .* (re)
nothing to backup
finished in \d+\.(\d+)? seconds (re)
$ HGUSER=anotheruser hg pushbackup --config infinitepushbackup.backupcheckfreq=-100
starting backup .* (re)
nothing to backup
finished in \d+\.(\d+)? seconds (re)
Test getavailablebackups command Test getavailablebackups command
$ hg getavailablebackups $ hg getavailablebackups
user test has 4 available backups: user test has 4 available backups:
\$TESTTMP on .* (re) \$TESTTMP on .* (re)
\$TESTTMP/backupsource on .* (re) \$TESTTMP/backupsource on .* (re)
\$TESTTMP/backupsource2 on .* (re) \$TESTTMP/backupsource2 on .* (re)
\$TESTTMP/bookmarks/backupsource3 on .* (re) \$TESTTMP/bookmarks/backupsource3 on .* (re)
$ hg getavailablebackups --user anotheruser $ hg getavailablebackups --user anotheruser
user anotheruser has 2 available backups: user anotheruser has 2 available backups:
\$TESTTMP/backupsource on .* (re) \$TESTTMP/backupsource on .* (re)
\$TESTTMP/backupsource2 on .* (re) \$TESTTMP/backupsource2 on .* (re)
$ hg getavailablebackups --json $ hg getavailablebackups --json
{".*": \["\$TESTTMP", "\$TESTTMP/backupsource", "\$TESTTMP/backupsource2", "\$TESTTMP/bookmarks/backupsource3"\]} (re) {".*": \["\$TESTTMP", "\$TESTTMP/backupsource", "\$TESTTMP/backupsource2", "\$TESTTMP/bookmarks/backupsource3"\]} (re)