This is an archive of the discontinued Mercurial Phabricator instance.

Differential D3556

run-tests: update the test case name format
ClosedPublic

Authored by lothiraldan on May 13 2018, 5:28 PM.

Details

Reviewers
quark
pulkit
durin42
Group Reviewers
hg-reviewers
Commits
rHGb865bba56db1: run-tests: update the test case name format
Summary

Manually typing parenthesis and spaces will be tedious when trying to launch a
specific test case.

I'm proposing a simpler format that is less hard to remember and type right.
There was other possibilities envisaged like :: or !, I think # is
slight easier to type but I'm open to any suggestion on the new format.

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

lothiraldan created this revision.May 13 2018, 5:28 PM
quark accepted this revision.May 14 2018, 7:24 PM
quark added a subscriber: quark.

I like this change. Didn't realize it's this simple. Marking as accetped.

pulkit accepted this revision.May 17 2018, 2:37 PM
pulkit added a comment.May 17 2018, 2:43 PM

Getting this test failure at the tip of hg-committed with this series applied, can you have a look?

--- /home/foobar/repo/pushaccess/tests/test-run-tests.t
+++ /home/foobar/repo/pushaccess/tests/test-run-tests.t.err
@@ -1634,9 +1634,24 @@
    #endif
   
   ERROR: test-cases-abc.t#B output changed
-  !
+  !.
+  --- $TESTTMP/anothertests/cases/test-cases-abc.t
+  +++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err
+  @@ -7,7 +7,7 @@
+     $ V=C
+   #endif
+     $ echo $V | sed 's/A/C/'
+  -  C
+  +  B
+   #if C
+     $ [ $V = C ]
+   #endif
+  
+  ERROR: test-cases-abc.t#B output changed
+  !.
   Failed test-cases-abc.t#B: output changed
-  # Ran 1 tests, 0 skipped, 1 failed.
+  Failed test-cases-abc.t#B: output changed
+  # Ran 4 tests, 0 skipped, 2 failed.
   python hash seed: * (glob)
   [1]
 

ERROR: test-run-tests.t output changed
lothiraldan updated this revision to Diff 8868.May 22 2018, 7:48 AM
lothiraldan added a comment.May 22 2018, 7:51 AM

I fixed the issue, it was a last minute style fix that turns out to not be equivalent to the previous code.

Anyway, here is the updated version that was rebased on top of hg-commited.

durin42 accepted this revision.Jun 4 2018, 6:23 PM
This revision is now accepted and ready to land.Jun 4 2018, 6:23 PM
Closed by commit rHGb865bba56db1: run-tests: update the test case name format (authored by lothiraldan). · Explain WhyJun 4 2018, 6:33 PM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mtests/run-tests.py (6 lines)
Mtests/test-run-tests.t (30 lines)

Diff 8972

tests/run-tests.py

styles = { styles = {
skipped: '#e5e5e5', skipped: '#e5e5e5',
skippedname: '#00ffff', skippedname: '#00ffff',
failed: '#7f0000', failed: '#7f0000',
failedname: '#ff0000', failedname: '#ff0000',
} }
class TestRunnerLexer(lexer.RegexLexer): class TestRunnerLexer(lexer.RegexLexer):
testpattern = r'[\w-]+\.(t|py)( \(case [\w-]+\))?' testpattern = r'[\w-]+\.(t|py)(#[\w-]+)?'
tokens = { tokens = {
'root': [ 'root': [
(r'^Skipped', token.Generic.Skipped, 'skipped'), (r'^Skipped', token.Generic.Skipped, 'skipped'),
(r'^Failed ', token.Generic.Failed, 'failed'), (r'^Failed ', token.Generic.Failed, 'failed'),
(r'^ERROR: ', token.Generic.Failed, 'failed'), (r'^ERROR: ', token.Generic.Failed, 'failed'),
], ],
'skipped': [ 'skipped': [
(testpattern, token.Generic.SName), (testpattern, token.Generic.SName),
def __init__(self, path, *args, **kwds): def __init__(self, path, *args, **kwds):
# accept an extra "case" parameter # accept an extra "case" parameter
case = kwds.pop('case', None) case = kwds.pop('case', None)
self._case = case self._case = case
self._allcases = parsettestcases(path) self._allcases = parsettestcases(path)
super(TTest, self).__init__(path, *args, **kwds) super(TTest, self).__init__(path, *args, **kwds)
if case: if case:
self.name = '%s (case %s)' % (self.name, _strpath(case)) self.name = '%s#%s' % (self.name, _strpath(case))
self.errpath = b'%s.%s.err' % (self.errpath[:-4], case) self.errpath = b'%s.%s.err' % (self.errpath[:-4], case)
self._tmpname += b'-%s' % case self._tmpname += b'-%s' % case
self._have = {} self._have = {}
@property @property
def refpath(self): def refpath(self):
return os.path.join(self._testdir, self.bname) return os.path.join(self._testdir, self.bname)
if os.path.isdir(arg): if os.path.isdir(arg):
if not arg.endswith(b'/'): if not arg.endswith(b'/'):
arg += b'/' arg += b'/'
expanded_args.extend([arg + a for a in os.listdir(arg)]) expanded_args.extend([arg + a for a in os.listdir(arg)])
else: else:
expanded_args.append(arg) expanded_args.append(arg)
args = expanded_args args = expanded_args
testcasepattern = re.compile(r'([\w-]+\.t|py)( \(case ([\w-])+\))') testcasepattern = re.compile(r'([\w-]+\.t|py)(#([\w-])+)')
tests = [] tests = []
for t in args: for t in args:
case = None case = None
if not (os.path.basename(t).startswith(b'test-') if not (os.path.basename(t).startswith(b'test-')
and (t.endswith(b'.py') or t.endswith(b'.t'))): and (t.endswith(b'.py') or t.endswith(b'.t'))):
m = testcasepattern.match(t) m = testcasepattern.match(t)

tests/test-run-tests.t

#endif #endif
$ echo $V | sed 's/A/C/' $ echo $V | sed 's/A/C/'
- C - C
+ B + B
#if C #if C
$ [ $V = C ] $ [ $V = C ]
#endif #endif
ERROR: test-cases-abc.t (case B) output changed ERROR: test-cases-abc.t#B output changed
!. !.
Failed test-cases-abc.t (case B): output changed Failed test-cases-abc.t#B: output changed
# Ran 3 tests, 0 skipped, 1 failed. # Ran 3 tests, 0 skipped, 1 failed.
python hash seed: * (glob) python hash seed: * (glob)
[1] [1]
--restart works --restart works
$ rt --restart $ rt --restart
--- $TESTTMP/anothertests/cases/test-cases-abc.t --- $TESTTMP/anothertests/cases/test-cases-abc.t
+++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err +++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err
@@ -7,7 +7,7 @@ @@ -7,7 +7,7 @@
$ V=C $ V=C
#endif #endif
$ echo $V | sed 's/A/C/' $ echo $V | sed 's/A/C/'
- C - C
+ B + B
#if C #if C
$ [ $V = C ] $ [ $V = C ]
#endif #endif
ERROR: test-cases-abc.t (case B) output changed ERROR: test-cases-abc.t#B output changed
!. !.
Failed test-cases-abc.t (case B): output changed Failed test-cases-abc.t#B: output changed
# Ran 2 tests, 0 skipped, 1 failed. # Ran 2 tests, 0 skipped, 1 failed.
python hash seed: * (glob) python hash seed: * (glob)
[1] [1]
--restart works with outputdir --restart works with outputdir
$ mkdir output $ mkdir output
$ mv test-cases-abc.t.B.err output $ mv test-cases-abc.t.B.err output
$ rt --restart --outputdir output $ rt --restart --outputdir output
--- $TESTTMP/anothertests/cases/test-cases-abc.t --- $TESTTMP/anothertests/cases/test-cases-abc.t
+++ $TESTTMP/anothertests/cases/output/test-cases-abc.t.B.err +++ $TESTTMP/anothertests/cases/output/test-cases-abc.t.B.err
@@ -7,7 +7,7 @@ @@ -7,7 +7,7 @@
$ V=C $ V=C
#endif #endif
$ echo $V | sed 's/A/C/' $ echo $V | sed 's/A/C/'
- C - C
+ B + B
#if C #if C
$ [ $V = C ] $ [ $V = C ]
#endif #endif
ERROR: test-cases-abc.t (case B) output changed ERROR: test-cases-abc.t#B output changed
!. !.
Failed test-cases-abc.t (case B): output changed Failed test-cases-abc.t#B: output changed
# Ran 2 tests, 0 skipped, 1 failed. # Ran 2 tests, 0 skipped, 1 failed.
python hash seed: * (glob) python hash seed: * (glob)
[1] [1]
Test TESTCASE variable Test TESTCASE variable
$ cat > test-cases-ab.t <<'EOF' $ cat > test-cases-ab.t <<'EOF'
> $ dostuff() { > $ dostuff() {
> #endif > #endif
> EOF > EOF
$ rt test-cases-ab.t $ rt test-cases-ab.t
.. ..
# Ran 2 tests, 0 skipped, 0 failed. # Ran 2 tests, 0 skipped, 0 failed.
Support running a specific test case Support running a specific test case
$ rt "test-cases-abc.t (case B)" $ rt "test-cases-abc.t#B"
--- $TESTTMP/anothertests/cases/test-cases-abc.t --- $TESTTMP/anothertests/cases/test-cases-abc.t
+++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err +++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err
@@ -7,7 +7,7 @@ @@ -7,7 +7,7 @@
$ V=C $ V=C
#endif #endif
$ echo $V | sed 's/A/C/' $ echo $V | sed 's/A/C/'
- C - C
+ B + B
#if C #if C
$ [ $V = C ] $ [ $V = C ]
#endif #endif
ERROR: test-cases-abc.t (case B) output changed ERROR: test-cases-abc.t#B output changed
! !
Failed test-cases-abc.t (case B): output changed Failed test-cases-abc.t#B: output changed
# Ran 1 tests, 0 skipped, 1 failed. # Ran 1 tests, 0 skipped, 1 failed.
python hash seed: * (glob) python hash seed: * (glob)
[1] [1]
Support running multiple test cases in the same file Support running multiple test cases in the same file
$ rt "test-cases-abc.t (case B)" "test-cases-abc.t (case C)" $ rt test-cases-abc.t#B test-cases-abc.t#C
--- $TESTTMP/anothertests/cases/test-cases-abc.t --- $TESTTMP/anothertests/cases/test-cases-abc.t
+++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err +++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err
@@ -7,7 +7,7 @@ @@ -7,7 +7,7 @@
$ V=C $ V=C
#endif #endif
$ echo $V | sed 's/A/C/' $ echo $V | sed 's/A/C/'
- C - C
+ B + B
#if C #if C
$ [ $V = C ] $ [ $V = C ]
#endif #endif
ERROR: test-cases-abc.t (case B) output changed ERROR: test-cases-abc.t#B output changed
!. !.
Failed test-cases-abc.t (case B): output changed Failed test-cases-abc.t#B: output changed
# Ran 2 tests, 0 skipped, 1 failed. # Ran 2 tests, 0 skipped, 1 failed.
python hash seed: * (glob) python hash seed: * (glob)
[1] [1]
Support running invalid test cases Support running invalid test cases
$ rt "test-cases-abc.t (case B)" "test-cases-abc.t (case D)" $ rt test-cases-abc.t#B test-cases-abc.t#D
--- $TESTTMP/anothertests/cases/test-cases-abc.t --- $TESTTMP/anothertests/cases/test-cases-abc.t
+++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err +++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err
@@ -7,7 +7,7 @@ @@ -7,7 +7,7 @@
$ V=C $ V=C
#endif #endif
$ echo $V | sed 's/A/C/' $ echo $V | sed 's/A/C/'
- C - C
+ B + B
#if C #if C
$ [ $V = C ] $ [ $V = C ]
#endif #endif
ERROR: test-cases-abc.t (case B) output changed ERROR: test-cases-abc.t#B output changed
! !
Failed test-cases-abc.t (case B): output changed Failed test-cases-abc.t#B: output changed
# Ran 1 tests, 0 skipped, 1 failed. # Ran 1 tests, 0 skipped, 1 failed.
python hash seed: * (glob) python hash seed: * (glob)
[1] [1]
Test automatic pattern replacement Test automatic pattern replacement
================================== ==================================
$ cat << EOF >> common-pattern.py $ cat << EOF >> common-pattern.py