This is an archive of the discontinued Mercurial Phabricator instance.

Differential D1264

rage: add info useful for infinitepushbackup debug
ClosedPublic

Authored by mbthomas on Oct 30 2017, 11:16 AM.
Tags
None
Subscribers
None

Details

Reviewers
stash
Group Reviewers
Restricted Project
Commits
rFBHGX65ac896fa2a8: rage: add info useful for infinitepushbackup debug
Summary

Adds information that is useful for debugging infinitepush backup to the rage
output. In particular:

  • Full node hashes in the unfiltered smartlog, so that they can be looked up in infinitepush.
  • The contents of infinitepushbackupstate.
  • The last 100 lines of the infinitepush backup logs for this repo.
Test Plan

Run hg rage and confirm output contains infinitepush information.

Excerpts from sample rage outputs:

hg sl (filtered):
---------------------------
...
| @  cef39429a243bc6433ab1a5d9db91b68d3245c8f
|/   4155:cef394  mbthomas  D1264
|    rage: add info useful for infinitepushbackup debug
|
| x  21593d9631c2f3f71769778f1b08a5020f33710b
|/   4154:21593d (hidden)  mbthomas
|    rage: add info useful for infinitepushbackup debug
|
| x  225dae77dbbab3e94f332cbf991538df0b90b8d3
|/   4153:225dae (hidden)  mbthomas
|    rage: add info useful for infinitepushbackup debug
|
...
infinitepush backup state:
---------------------------
{"bookmarks": {"tmp": "2ed25c96dc27000c0d00e1a86ad354205715f260", "arcpatch-D698-1": "0ad49fb97fa44dda2e6d0725615342a195489c79", "arcpatch-D698": "ee2d7b5d8dc193620a294faf0505ec25fb1a8f78", "pt-revlog": "ccb2668182bdd628bcc5efd8fb8a3ea186f85a08"}, "heads": ["34fa638f6387e81b5039d29ce50a755131479870", "3a5be2d8d5705302f7a754a69cabac884e17d34d", "ee2d7b5d8dc193620a294faf0505ec25fb1a8f78", "a904118ae039bd0ee7639753b6f194f21bfa7c64", "ccb2668182bdd628bcc5efd8fb8a3ea186f85a08", "0ad49fb97fa44dda2e6d0725615342a195489c79", "d1f7bc030c734c5ca2b9f3db2f52759a113db3c9"]}
...
infinitepush backup logs:
---------------------------
infinitepushbackup.logdir not set
...
infinitepush backup logs:
---------------------------
testrepo20170816 (first 12 lines omitted):
  remote:     3dc5cc214efc  [...]
  remote:     03095008ba11  [...]
  finished in 17.108702 seconds

testrepo20170818:
  starting backup 06:10:40 18 Aug 2017 PDT
  nothing to backup
  finished in 2.110974 seconds
  starting backup 06:11:30 18 Aug 2017 PDT
  nothing to backup
  finished in 1.479330 seconds

Diff Detail

Repository
rFBHGX Facebook Mercurial Extensions
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

mbthomas created this revision.Oct 30 2017, 11:16 AM
Herald added a reviewer: Restricted Project. · View Herald TranscriptOct 30 2017, 11:16 AM
stash requested changes to this revision.Oct 30 2017, 4:48 PM

Can you also write the test plan?

hgext3rd/rage.py
182–187

Why do you join with ' '? Why not '/n'?

I actually don't think that it makes sense to return lines for all logfiles. Just the last one should be enough

This revision now requires changes to proceed.Oct 30 2017, 4:48 PM
mbthomas added inline comments.Nov 1 2017, 6:51 AM
hgext3rd/rage.py
182–187

The log lines already have '\n' at the end from readlines(). The ' ' in the join is just to indent the lines to give a visual output like:

filename:
  log lines

I've included more than the most recent file, as it might be the case that the day has just changed and the problem was from the previous day (I think we've seen that at least once).

mbthomas edited the test plan for this revision. (Show Details)Nov 1 2017, 7:04 AM
stash accepted this revision.Nov 2 2017, 4:22 PM
This revision is now accepted and ready to land.Nov 2 2017, 4:22 PM
Closed by commit rFBHGX65ac896fa2a8: rage: add info useful for infinitepushbackup debug (authored by mbthomas). · Explain WhyNov 8 2017, 6:27 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mhgext3rd/rage.py (53 lines)

Diff 3350

hgext3rd/rage.py

rpmbin = ui.config('rage', 'rpmbin', 'rpm') rpmbin = ui.config('rage', 'rpmbin', 'rpm')
for name in ['hg', 'hg.real']: for name in ['hg', 'hg.real']:
path = which(name) path = which(name)
if not path: if not path:
continue continue
result.add(shcmd('%s -qf %s' % (rpmbin, path), check=False)) result.add(shcmd('%s -qf %s' % (rpmbin, path), check=False))
return ''.join(result) return ''.join(result)
def infinitepushbackuplogs(ui, repo):
"""Contents of recent infinitepush log files."""
logdir = ui.config('infinitepushbackup', 'logdir')
if not logdir:
return "infinitepushbackup.logdir not set"
try:
username = util.shortuser(ui.username())
except Exception:
username = 'unknown'
userlogdir = os.path.join(logdir, username)
if not os.path.exists(userlogdir):
return "log directory does not exist: %s" % userlogdir
# Log filenames are the reponame with the date (YYYYMMDD) appended.
reponame = os.path.basename(repo.origroot)
logfiles = sorted([f for f in os.listdir(userlogdir)
if f[:-8] == reponame])
if not logfiles:
return "no log files found for %s in %s" % (reponame, userlogdir)
# Display the last 100 lines from the most recent log files.
logs = []
linelimit = 100
for logfile in reversed(logfiles):
loglines = open(os.path.join(userlogdir, logfile)).readlines()
linecount = len(loglines)
if linecount > linelimit:
logcontent = ' '.join(loglines[-linelimit:])
logs.append("%s (first %s lines omitted):\n %s\n"
% (logfile, linecount - linelimit, logcontent))
break
else:
logcontent = ' '.join(loglines)
stashUnsubmitted
Not Done

Why do you join with ' '? Why not '/n'?

I actually don't think that it makes sense to return lines for all logfiles. Just the last one should be enough

stash: Why do you join with ' '? Why not '/n'? I actually don't think that it makes sense to return…
mbthomasAuthorUnsubmitted
Not Done

The log lines already have '\n' at the end from readlines(). The ' ' in the join is just to indent the lines to give a visual output like:

filename:
  log lines

I've included more than the most recent file, as it might be the case that the day has just changed and the problem was from the previous day (I think we've seen that at least once).

mbthomas: The log lines already have `'\n'` at the end from `readlines()`. The `' '` in the join is…
logs.append("%s:\n %s\n" % (logfile, logcontent))
linelimit -= linecount
return ''.join(reversed(logs))
@command('^rage', rageopts , _('hg rage')) @command('^rage', rageopts , _('hg rage'))
def rage(ui, repo, *pats, **opts): def rage(ui, repo, *pats, **opts):
"""collect useful diagnostics for asking help from the source control team """collect useful diagnostics for asking help from the source control team
The rage command collects useful diagnostic information. The rage command collects useful diagnostic information.
By default, the information will be uploaded to Phabricator and By default, the information will be uploaded to Phabricator and
instructions about how to ask for help will be printed. instructions about how to ask for help will be printed.
_repo = opts['_repo'] _repo = opts['_repo']
del opts['_repo'] del opts['_repo']
ui.pushbuffer(error=True) ui.pushbuffer(error=True)
try: try:
func(ui, _repo, *args, **opts) func(ui, _repo, *args, **opts)
finally: finally:
return ui.popbuffer() return ui.popbuffer()
def hgfile(filename):
if repo.vfs.exists(filename):
return repo.vfs(filename).read()
else:
return "File not found: %s" % repo.vfs.join(filename)
if opts.get('oncall') and opts.get('preview'): if opts.get('oncall') and opts.get('preview'):
raise error.Abort('--preview and --oncall cannot be used together') raise error.Abort('--preview and --oncall cannot be used together')
basic = [ basic = [
('date', time.ctime()), ('date', time.ctime()),
('unixname', encoding.environ.get('LOGNAME')), ('unixname', encoding.environ.get('LOGNAME')),
('hostname', socket.gethostname()), ('hostname', socket.gethostname()),
('repo location', _failsafe(lambda: repo.root)), ('repo location', _failsafe(lambda: repo.root)),
('active bookmark', ('active bookmark',
_failsafe(lambda: bookmarks._readactive(repo, repo._bookmarks))), _failsafe(lambda: bookmarks._readactive(repo, repo._bookmarks))),
('hg version', _failsafe( ('hg version', _failsafe(
lambda: __import__('mercurial.__version__').__version__.version)), lambda: __import__('mercurial.__version__').__version__.version)),
('obsstore size', _failsafe( ('obsstore size', _failsafe(
lambda: str(repo.vfs.stat('store/obsstore').st_size))), lambda: str(repo.vfs.stat('store/obsstore').st_size))),
] ]
ui._colormode = None ui._colormode = None
detailed = [ detailed = [
('df -h', _failsafe(lambda: shcmd('df -h', check=False))), ('df -h', _failsafe(lambda: shcmd('df -h', check=False))),
# smartlog as the user sees it # smartlog as the user sees it
('hg sl (filtered)', _failsafe(lambda: hgcmd( ('hg sl (filtered)', _failsafe(lambda: hgcmd(
smartlog.smartlog, template='{hsl}'))), smartlog.smartlog, template='{hsl}'))),
# unfiltered smartlog for recent hidden changesets # unfiltered smartlog for recent hidden changesets, including full
# node identity
('hg sl (unfiltered)', _failsafe(lambda: hgcmd( ('hg sl (unfiltered)', _failsafe(lambda: hgcmd(
smartlog.smartlog, _repo=repo.unfiltered(), template='{hsl}'))), smartlog.smartlog, _repo=repo.unfiltered(),
template='{node}\n{hsl}'))),
('first 20 lines of "hg status"', ('first 20 lines of "hg status"',
_failsafe(lambda: _failsafe(lambda:
'\n'.join(hgcmd(commands.status).splitlines()[:20]))), '\n'.join(hgcmd(commands.status).splitlines()[:20]))),
('hg blackbox -l60', ('hg blackbox -l60',
_failsafe(lambda: hgcmd(blackbox.blackbox, limit=60))), _failsafe(lambda: hgcmd(blackbox.blackbox, limit=60))),
('hg summary', _failsafe(lambda: hgcmd(commands.summary))), ('hg summary', _failsafe(lambda: hgcmd(commands.summary))),
('hg config (local)', _failsafe(lambda: '\n'.join(localconfig(ui)))), ('hg config (local)', _failsafe(lambda: '\n'.join(localconfig(ui)))),
('hg sparse', ('hg sparse',
_failsafe( _failsafe(
lambda: hgcmd( lambda: hgcmd(
sparse.sparse, include=False, exclude=False, delete=False, sparse.sparse, include=False, exclude=False, delete=False,
force=False, enable_profile=False, disable_profile=False, force=False, enable_profile=False, disable_profile=False,
refresh=False, reset=False, import_rules=False, refresh=False, reset=False, import_rules=False,
clear_rules=False))), clear_rules=False))),
('usechg', _failsafe(usechginfo)), ('usechg', _failsafe(usechginfo)),
('rpm info', _failsafe(partial(rpminfo, ui))), ('rpm info', _failsafe(partial(rpminfo, ui))),
('klist', _failsafe(lambda: shcmd('klist', check=False))), ('klist', _failsafe(lambda: shcmd('klist', check=False))),
('ifconfig', _failsafe(lambda: shcmd('ifconfig'))), ('ifconfig', _failsafe(lambda: shcmd('ifconfig'))),
('airport', _failsafe( ('airport', _failsafe(
lambda: shcmd('/System/Library/PrivateFrameworks/Apple80211.' + lambda: shcmd('/System/Library/PrivateFrameworks/Apple80211.' +
'framework/Versions/Current/Resources/airport ' + 'framework/Versions/Current/Resources/airport ' +
'--getinfo', check=False))), '--getinfo', check=False))),
('hg debugobsolete <smartlog>', ('hg debugobsolete <smartlog>',
_failsafe(lambda: obsoleteinfo(repo, hgcmd))), _failsafe(lambda: obsoleteinfo(repo, hgcmd))),
('infinitepush backup state',
_failsafe(lambda: hgfile('infinitepushbackupstate'))),
('infinitepush backup logs',
_failsafe(lambda: infinitepushbackuplogs(ui, repo))),
('hg config (all)', _failsafe(lambda: hgcmd(commands.config))), ('hg config (all)', _failsafe(lambda: hgcmd(commands.config))),
] ]
if util.safehasattr(repo, 'name'): if util.safehasattr(repo, 'name'):
# Add the contents of both local and shared pack directories. # Add the contents of both local and shared pack directories.
packlocs = { packlocs = {
'local': lambda category: shallowutil.getlocalpackpath( 'local': lambda category: shallowutil.getlocalpackpath(
repo.svfs.vfs.base, category), repo.svfs.vfs.base, category),