This is an archive of the discontinued Mercurial Phabricator instance.

Differential D3880

tests: add test showing puzzling test output with (glob) lines
ClosedPublic

Authored by martinvonz on Jul 3 2018, 7:42 PM.

Details

Reviewers
None
Group Reviewers
hg-reviewers
Commits
rHG3fb46734508d: tests: add test showing puzzling test output with (glob) lines
Summary

When using multiple (glob) lines, the test runner often moves the
lines around in the output on failure. I have run into this problem
many times. The added test shows one example of this. Note that it
doesn't show the mismatching lines and instead shows diff hunks that
appear to replace a literal by a glob, and other hunks that should
clearly match.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Revision Contents
Changeset List

PathPackages
Mtests/test-run-tests.t (47 lines)

Diff 9451

tests/test-run-tests.t

ERROR: test-failure.t output changed ERROR: test-failure.t output changed
! !
Failed test-failure.t: output changed Failed test-failure.t: output changed
# Ran 1 tests, 0 skipped, 1 failed. # Ran 1 tests, 0 skipped, 1 failed.
python hash seed: * (glob) python hash seed: * (glob)
[1] [1]
test how multiple globs gets matched with lines in output
$ cat > test-failure-globs.t <<EOF
> $ echo "context"; echo "context"; \
> echo "key: 1"; echo "value: not a"; \
> echo "key: 2"; echo "value: not b"; \
> echo "key: 3"; echo "value: c"; \
> echo "key: 4"; echo "value: d"
> context
> context
> key: 1
> value: a
> key: 2
> value: b
> key: 3
> value: * (glob)
> key: 4
> value: * (glob)
> EOF
$ rt test-failure-globs.t
--- $TESTTMP/test-failure-globs.t
+++ $TESTTMP/test-failure-globs.t.err
@@ -2,10 +2,10 @@
context
context
key: 1
- value: a
+ value: * (glob)
key: 2
- value: b
+ value: * (glob)
key: 3
- value: * (glob)
+ value: c
key: 4
- value: * (glob)
+ value: d
ERROR: test-failure-globs.t output changed
!
Failed test-failure-globs.t: output changed
# Ran 1 tests, 0 skipped, 1 failed.
python hash seed: * (glob)
[1]
$ rm test-failure-globs.t
test diff colorisation test diff colorisation
#if no-windows pygments #if no-windows pygments
$ rt test-failure.t --color always $ rt test-failure.t --color always
\x1b[38;5;124m--- $TESTTMP/test-failure.t\x1b[39m (esc) \x1b[38;5;124m--- $TESTTMP/test-failure.t\x1b[39m (esc)
\x1b[38;5;34m+++ $TESTTMP/test-failure.t.err\x1b[39m (esc) \x1b[38;5;34m+++ $TESTTMP/test-failure.t.err\x1b[39m (esc)
\x1b[38;5;90;01m@@ -1,4 +1,4 @@\x1b[39;00m (esc) \x1b[38;5;90;01m@@ -1,4 +1,4 @@\x1b[39;00m (esc)
this test is still more bytes than success. this test is still more bytes than success.
pad pad pad pad............................................................ pad pad pad pad............................................................
]]> </failure> ]]> </failure>
</testcase> </testcase>
</testsuite> </testsuite>
$ cat .testtimes $ cat .testtimes
test-empty.t * (glob) test-empty.t * (glob)
test-failure-globs.t * (glob)
test-failure-unicode.t * (glob) test-failure-unicode.t * (glob)
test-failure.t * (glob) test-failure.t * (glob)
test-success.t * (glob) test-success.t * (glob)
$ rt --list-tests $ rt --list-tests
test-failure-unicode.t test-failure-unicode.t
test-failure.t test-failure.t
test-success.t test-success.t