This is an archive of the discontinued Mercurial Phabricator instance.

Differential D3700

run-tests: add support for external test result
ClosedPublic

Authored by lothiraldan on Jun 7 2018, 3:19 PM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHGc44ae5997869: run-tests: add support for external test result
Summary

The goal is to begin experiment with custom test result. I'm not sure we
should offers any backward-compatibility guarantee on that plugin API as it
doesn't change often and shouldn't have too much clients.

Diff Detail

Repository
rHG Mercurial
Lint
Lint Skipped
Unit
Unit Tests Skipped

Event Timeline

lothiraldan created this revision.Jun 7 2018, 3:19 PM
durin42 added a subscriber: durin42.Jun 12 2018, 12:58 PM

I see some what, but not any why. Why is this useful?

lothiraldan updated this revision to Diff 9031.Jun 12 2018, 5:05 PM
lothiraldan added a comment.Jun 12 2018, 5:11 PM
In D3700#58376, @durin42 wrote:

I see some what, but not any why. Why is this useful?

I need this changeset to integrate the mercurial test runner with some external tools.

durin42 added a comment.Jun 12 2018, 5:12 PM
In D3700#58403, @lothiraldan wrote:
In D3700#58376, @durin42 wrote:

I see some what, but not any why. Why is this useful?

I need this changeset to integrate the mercurial test runner with some external tools.

I'd still like more information. Why is the json report inadquate? What's your goal?

(Remember my perspective: every feature here is a liability, so anything we're not using for development on Mercurial is something I'm hesitant to take on in run-tests)

lothiraldan added a comment.EditedJul 2 2018, 8:16 AM
In D3700#58404, @durin42 wrote:
In D3700#58403, @lothiraldan wrote:
In D3700#58376, @durin42 wrote:

I see some what, but not any why. Why is this useful?

I need this changeset to integrate the mercurial test runner with some external tools.

I'd still like more information. Why is the json report inadquate? What's your goal?
(Remember my perspective: every feature here is a liability, so anything we're not using for development on Mercurial is something I'm hesitant to take on in run-tests)

I'm trying to integrate Mercurial with a new test format I'm developing called LITF (https://github.com/Lothiraldan/litf). The LITF test format is stream-based while the json report, while having the needed information, only dumps a file at the end. Moreover, the format is not final yet, I would like to avoid generating unnecessary noise on the phabricator / mailing-list.

The end-goal is to have better tooling to launch only a subset of tests, see the results of tests from a remote machine and speed up the development process by allowing to relaunch only failed tests in a subset for example.

durin42 accepted this revision as: durin42.Jul 3 2018, 2:00 PM

I guess I can live with that.

tests/basic_test_result.py
37

shouldn't this say "SKIP!"?

lothiraldan updated this revision to Diff 9476.Jul 9 2018, 10:41 AM
lothiraldan marked an inline comment as done.Jul 9 2018, 10:42 AM
durin42 accepted this revision.Jul 9 2018, 10:44 AM
This revision is now accepted and ready to land.Jul 9 2018, 10:44 AM
Closed by commit rHGc44ae5997869: run-tests: add support for external test result (authored by lothiraldan). · Explain WhyJul 9 2018, 10:45 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
AMtests/basic_test_result.py (46 lines)
Mtests/run-tests.py (19 lines)
Mtests/test-run-tests.t (9 lines)

Diff 9031

tests/basic_test_result.py

  • This file was added.
from __future__ import print_function
import unittest
class TestResult(unittest._TextTestResult):
def __init__(self, options, *args, **kwargs):
super(TestResult, self).__init__(*args, **kwargs)
self._options = options
# unittest.TestResult didn't have skipped until 2.7. We need to
# polyfill it.
self.skipped = []
# We have a custom "ignored" result that isn't present in any Python
# unittest implementation. It is very similar to skipped. It may make
# sense to map it into skip some day.
self.ignored = []
self.times = []
self._firststarttime = None
# Data stored for the benefit of generating xunit reports.
self.successes = []
self.faildata = {}
def addFailure(self, test, reason):
print("FAILURE!", test, reason)
def addSuccess(self, test):
print("SUCCESS!", test)
def addError(self, test, err):
print("ERR!", test, err)
# Polyfill.
def addSkip(self, test, reason):
print("ERR!", test, reason)
durin42Unsubmitted
Done

shouldn't this say "SKIP!"?

durin42: shouldn't this say "SKIP!"?
def addIgnore(self, test, reason):
print("IGNORE!", test, reason)
def addOutputMismatch(self, test, ret, got, expected):
return False
def stopTest(self, test, interrupted=False):
super(TestResult, self).stopTest(test)

tests/run-tests.py

endtime[4] - origin, # end date in run context endtime[4] - origin, # end date in run context
)) ))
if interrupted: if interrupted:
with iolock: with iolock:
self.stream.writeln('INTERRUPTED: %s (after %d seconds)' % ( self.stream.writeln('INTERRUPTED: %s (after %d seconds)' % (
test.name, self.times[-1][3])) test.name, self.times[-1][3]))
def getTestResult():
"""
Returns the relevant test result
"""
if "CUSTOM_TEST_RESULT" in os.environ:
testresultmodule = __import__(os.environ["CUSTOM_TEST_RESULT"])
return testresultmodule.TestResult
else:
return TestResult
class TestSuite(unittest.TestSuite): class TestSuite(unittest.TestSuite):
"""Custom unittest TestSuite that knows how to execute Mercurial tests.""" """Custom unittest TestSuite that knows how to execute Mercurial tests."""
def __init__(self, testdir, jobs=1, whitelist=None, blacklist=None, def __init__(self, testdir, jobs=1, whitelist=None, blacklist=None,
retest=False, keywords=None, loop=False, runs_per_test=1, retest=False, keywords=None, loop=False, runs_per_test=1,
loadtest=None, showchannels=False, loadtest=None, showchannels=False,
*args, **kwargs): *args, **kwargs):
"""Create a new instance that can run tests with a configuration. """Create a new instance that can run tests with a configuration.
"""Custom unittest test runner that uses appropriate settings.""" """Custom unittest test runner that uses appropriate settings."""
def __init__(self, runner, *args, **kwargs): def __init__(self, runner, *args, **kwargs):
super(TextTestRunner, self).__init__(*args, **kwargs) super(TextTestRunner, self).__init__(*args, **kwargs)
self._runner = runner self._runner = runner
def listtests(self, test): def listtests(self, test):
result = TestResult(self._runner.options, self.stream, result = getTestResult()(self._runner.options, self.stream,
self.descriptions, 0) self.descriptions, 0)
test = sorted(test, key=lambda t: t.name) test = sorted(test, key=lambda t: t.name)
for t in test: for t in test:
print(t.name) print(t.name)
result.addSuccess(t) result.addSuccess(t)
if self._runner.options.xunit: if self._runner.options.xunit:
with open(self._runner.options.xunit, "wb") as xuf: with open(self._runner.options.xunit, "wb") as xuf:
self._writexunit(result, xuf) self._writexunit(result, xuf)
if self._runner.options.json: if self._runner.options.json:
jsonpath = os.path.join(self._runner._outputdir, b'report.json') jsonpath = os.path.join(self._runner._outputdir, b'report.json')
with open(jsonpath, 'w') as fp: with open(jsonpath, 'w') as fp:
self._writejson(result, fp) self._writejson(result, fp)
return result return result
def run(self, test): def run(self, test):
result = TestResult(self._runner.options, self.stream, result = getTestResult()(self._runner.options, self.stream,
self.descriptions, self.verbosity) self.descriptions, self.verbosity)
test(result) test(result)
failed = len(result.failures) failed = len(result.failures)
skipped = len(result.skipped) skipped = len(result.skipped)
ignored = len(result.ignored) ignored = len(result.ignored)
with iolock: with iolock:
self.stream.writeln('') self.stream.writeln('')

tests/test-run-tests.t

$ rm -f test-glob-backslash.t $ rm -f test-glob-backslash.t
Test globbing of local IP addresses Test globbing of local IP addresses
$ echo 172.16.18.1 $ echo 172.16.18.1
$LOCALIP (glob) $LOCALIP (glob)
$ echo dead:beef::1 $ echo dead:beef::1
$LOCALIP (glob) $LOCALIP (glob)
Add support for external test formatter
=======================================
$ CUSTOM_TEST_RESULT=basic_test_result $PYTHON $TESTDIR/run-tests.py --with-hg=`which hg` "$@" test-success.t test-failure.t
# Ran 2 tests, 0 skipped, 0 failed.
FAILURE! test-failure.t output changed
SUCCESS! test-success.t
Test reusability for third party tools Test reusability for third party tools
====================================== ======================================
$ mkdir "$TESTTMP"/anothertests $ mkdir "$TESTTMP"/anothertests
$ cd "$TESTTMP"/anothertests $ cd "$TESTTMP"/anothertests
test that `run-tests.py` can execute hghave, even if it runs not in test that `run-tests.py` can execute hghave, even if it runs not in
Mercurial source tree. Mercurial source tree.