( )⚙ D6452 tests: make run-tests exit non-zero if there are "errors"

This is an archive of the discontinued Mercurial Phabricator instance.

Differential D6452

tests: make run-tests exit non-zero if there are "errors"
ClosedPublic

Authored by spectral on May 29 2019, 2:26 AM.

Details

Reviewers
pulkit
Group Reviewers
hg-reviewers
Commits
rHG15d5a2de44aa: tests: make run-tests exit non-zero if there are "errors"
Summary

Previously, if there was an error such as a broken .t file that caused
run-tests.py to encounter an exception during parsing, the test would be
considered in an "errored" state, which is separate from "failed".

The check for whether to exit non-zero or not was based entirely on whether
there were any tests in a "failed" state, so if there was only an error,
run-tests would exit with 0. Our test infrastructure would then consider the
test as passing, causing us to have some tests with false negatives that have
gone undetected for a few weeks now.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

spectral created this revision.May 29 2019, 2:26 AM
pulkit accepted this revision.May 29 2019, 8:40 AM
Closed by commit rHG15d5a2de44aa: tests: make run-tests exit non-zero if there are "errors" (authored by spectral). · Explain WhyMay 29 2019, 8:54 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mtests/run-tests.py (2 lines)

Diff 15280

tests/run-tests.py

assert self._installdir assert self._installdir
self._installchg() self._installchg()
log('running %d tests using %d parallel processes' % ( log('running %d tests using %d parallel processes' % (
num_tests, jobs)) num_tests, jobs))
result = runner.run(suite) result = runner.run(suite)
if result.failures: if result.failures or result.errors:
failed = True failed = True
result.onEnd() result.onEnd()
if self.options.anycoverage: if self.options.anycoverage:
self._outputcoverage() self._outputcoverage()
except KeyboardInterrupt: except KeyboardInterrupt:
failed = True failed = True