This is an archive of the discontinued Mercurial Phabricator instance.

Differential D3700

run-tests: add support for external test result
ClosedPublic

Authored by lothiraldan on Jun 7 2018, 3:19 PM.

Details

Reviewers
durin42
Group Reviewers
hg-reviewers
Commits
rHGc44ae5997869: run-tests: add support for external test result
Summary

The goal is to begin experiment with custom test result. I'm not sure we
should offers any backward-compatibility guarantee on that plugin API as it
doesn't change often and shouldn't have too much clients.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

lothiraldan created this revision.Jun 7 2018, 3:19 PM
durin42 added a subscriber: durin42.Jun 12 2018, 12:58 PM

I see some what, but not any why. Why is this useful?

lothiraldan updated this revision to Diff 9031.Jun 12 2018, 5:05 PM
lothiraldan added a comment.Jun 12 2018, 5:11 PM
In D3700#58376, @durin42 wrote:

I see some what, but not any why. Why is this useful?

I need this changeset to integrate the mercurial test runner with some external tools.

durin42 added a comment.Jun 12 2018, 5:12 PM
In D3700#58403, @lothiraldan wrote:
In D3700#58376, @durin42 wrote:

I see some what, but not any why. Why is this useful?

I need this changeset to integrate the mercurial test runner with some external tools.

I'd still like more information. Why is the json report inadquate? What's your goal?

(Remember my perspective: every feature here is a liability, so anything we're not using for development on Mercurial is something I'm hesitant to take on in run-tests)

lothiraldan added a comment.EditedJul 2 2018, 8:16 AM
In D3700#58404, @durin42 wrote:
In D3700#58403, @lothiraldan wrote:
In D3700#58376, @durin42 wrote:

I see some what, but not any why. Why is this useful?

I need this changeset to integrate the mercurial test runner with some external tools.

I'd still like more information. Why is the json report inadquate? What's your goal?
(Remember my perspective: every feature here is a liability, so anything we're not using for development on Mercurial is something I'm hesitant to take on in run-tests)

I'm trying to integrate Mercurial with a new test format I'm developing called LITF (https://github.com/Lothiraldan/litf). The LITF test format is stream-based while the json report, while having the needed information, only dumps a file at the end. Moreover, the format is not final yet, I would like to avoid generating unnecessary noise on the phabricator / mailing-list.

The end-goal is to have better tooling to launch only a subset of tests, see the results of tests from a remote machine and speed up the development process by allowing to relaunch only failed tests in a subset for example.

durin42 accepted this revision as: durin42.Jul 3 2018, 2:00 PM

I guess I can live with that.

tests/basic_test_result.py
38

shouldn't this say "SKIP!"?

lothiraldan updated this revision to Diff 9476.Jul 9 2018, 10:41 AM
lothiraldan marked an inline comment as done.Jul 9 2018, 10:42 AM
durin42 accepted this revision.Jul 9 2018, 10:44 AM
This revision is now accepted and ready to land.Jul 9 2018, 10:44 AM
Closed by commit rHGc44ae5997869: run-tests: add support for external test result (authored by lothiraldan). · Explain WhyJul 9 2018, 10:45 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
AMtests/basic_test_result.py (46 lines)
Mtests/run-tests.py (19 lines)
Mtests/test-run-tests.t (9 lines)

Diff 9478

tests/basic_test_result.py

  • This file was added.
from __future__ import print_function
import unittest
class TestResult(unittest._TextTestResult):
def __init__(self, options, *args, **kwargs):
super(TestResult, self).__init__(*args, **kwargs)
self._options = options
# unittest.TestResult didn't have skipped until 2.7. We need to
# polyfill it.
self.skipped = []
# We have a custom "ignored" result that isn't present in any Python
# unittest implementation. It is very similar to skipped. It may make
# sense to map it into skip some day.
self.ignored = []
self.times = []
self._firststarttime = None
# Data stored for the benefit of generating xunit reports.
self.successes = []
self.faildata = {}
def addFailure(self, test, reason):
print("FAILURE!", test, reason)
def addSuccess(self, test):
print("SUCCESS!", test)
def addError(self, test, err):
print("ERR!", test, err)
# Polyfill.
def addSkip(self, test, reason):
print("SKIP!", test, reason)
durin42Unsubmitted
Done

shouldn't this say "SKIP!"?

durin42: shouldn't this say "SKIP!"?
def addIgnore(self, test, reason):
print("IGNORE!", test, reason)
def addOutputMismatch(self, test, ret, got, expected):
return False
def stopTest(self, test, interrupted=False):
super(TestResult, self).stopTest(test)

tests/run-tests.py

endtime[4] - origin, # end date in run context endtime[4] - origin, # end date in run context
)) ))
if interrupted: if interrupted:
with iolock: with iolock:
self.stream.writeln('INTERRUPTED: %s (after %d seconds)' % ( self.stream.writeln('INTERRUPTED: %s (after %d seconds)' % (
test.name, self.times[-1][3])) test.name, self.times[-1][3]))
def getTestResult():
"""
Returns the relevant test result
"""
if "CUSTOM_TEST_RESULT" in os.environ:
testresultmodule = __import__(os.environ["CUSTOM_TEST_RESULT"])
return testresultmodule.TestResult
else:
return TestResult
class TestSuite(unittest.TestSuite): class TestSuite(unittest.TestSuite):
"""Custom unittest TestSuite that knows how to execute Mercurial tests.""" """Custom unittest TestSuite that knows how to execute Mercurial tests."""
def __init__(self, testdir, jobs=1, whitelist=None, blacklist=None, def __init__(self, testdir, jobs=1, whitelist=None, blacklist=None,
retest=False, keywords=None, loop=False, runs_per_test=1, retest=False, keywords=None, loop=False, runs_per_test=1,
loadtest=None, showchannels=False, loadtest=None, showchannels=False,
*args, **kwargs): *args, **kwargs):
"""Create a new instance that can run tests with a configuration. """Create a new instance that can run tests with a configuration.
"""Custom unittest test runner that uses appropriate settings.""" """Custom unittest test runner that uses appropriate settings."""
def __init__(self, runner, *args, **kwargs): def __init__(self, runner, *args, **kwargs):
super(TextTestRunner, self).__init__(*args, **kwargs) super(TextTestRunner, self).__init__(*args, **kwargs)
self._runner = runner self._runner = runner
def listtests(self, test): def listtests(self, test):
result = TestResult(self._runner.options, self.stream, result = getTestResult()(self._runner.options, self.stream,
self.descriptions, 0) self.descriptions, 0)
test = sorted(test, key=lambda t: t.name) test = sorted(test, key=lambda t: t.name)
for t in test: for t in test:
print(t.name) print(t.name)
result.addSuccess(t) result.addSuccess(t)
if self._runner.options.xunit: if self._runner.options.xunit:
with open(self._runner.options.xunit, "wb") as xuf: with open(self._runner.options.xunit, "wb") as xuf:
self._writexunit(result, xuf) self._writexunit(result, xuf)
if self._runner.options.json: if self._runner.options.json:
jsonpath = os.path.join(self._runner._outputdir, b'report.json') jsonpath = os.path.join(self._runner._outputdir, b'report.json')
with open(jsonpath, 'w') as fp: with open(jsonpath, 'w') as fp:
self._writejson(result, fp) self._writejson(result, fp)
return result return result
def run(self, test): def run(self, test):
result = TestResult(self._runner.options, self.stream, result = getTestResult()(self._runner.options, self.stream,
self.descriptions, self.verbosity) self.descriptions, self.verbosity)
test(result) test(result)
failed = len(result.failures) failed = len(result.failures)
skipped = len(result.skipped) skipped = len(result.skipped)
ignored = len(result.ignored) ignored = len(result.ignored)
with iolock: with iolock:
self.stream.writeln('') self.stream.writeln('')

tests/test-run-tests.t

$ rm -f test-glob-backslash.t $ rm -f test-glob-backslash.t
Test globbing of local IP addresses Test globbing of local IP addresses
$ echo 172.16.18.1 $ echo 172.16.18.1
$LOCALIP (glob) $LOCALIP (glob)
$ echo dead:beef::1 $ echo dead:beef::1
$LOCALIP (glob) $LOCALIP (glob)
Add support for external test formatter
=======================================
$ CUSTOM_TEST_RESULT=basic_test_result $PYTHON $TESTDIR/run-tests.py --with-hg=`which hg` "$@" test-success.t test-failure.t
# Ran 2 tests, 0 skipped, 0 failed.
FAILURE! test-failure.t output changed
SUCCESS! test-success.t
Test reusability for third party tools Test reusability for third party tools
====================================== ======================================
$ mkdir "$TESTTMP"/anothertests $ mkdir "$TESTTMP"/anothertests
$ cd "$TESTTMP"/anothertests $ cd "$TESTTMP"/anothertests
test that `run-tests.py` can execute hghave, even if it runs not in test that `run-tests.py` can execute hghave, even if it runs not in
Mercurial source tree. Mercurial source tree.