This is an archive of the discontinued Mercurial Phabricator instance.

Differential D4049

testrunner: allow multiple #testcases
ClosedPublic

Authored by martinvonz on Aug 1 2018, 7:52 PM.

Details

Reviewers
mharbison72
pulkit
Group Reviewers
hg-reviewers
Commits
rHG3086a8627b29: testrunner: allow multiple #testcases
Summary

This lets you have multiple #testcases declarations and they're taken
to be different dimensions of the test, so their cross product becomes
the total set of test cases. For example:

#testcases obsstore-on obsstore-off
#testcases manifest-flat manifest-tree

  $ hg rebase ...
  ...
  #if obsstore-on
    $ hg log ...
  #endif

Note that this is an excellent way to slow down the test suite, so use
it with care.

The feature is implemented by replacing most of the "case" variables
that were strings before by an array of strings with each item a
different dimension of the test case. The file names are created by
joining the dimensions by "#"
(e.g. test-foo.t#obsstore-on#manifest-flat).

Diff Detail

Repository
rHG Mercurial
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

martinvonz created this revision.Aug 1 2018, 7:52 PM
mharbison72 requested changes to this revision.Aug 2 2018, 1:07 AM
mharbison72 added a subscriber: mharbison72.

I think Windows will choke on ‘:’ in the file name.

This revision now requires changes to proceed.Aug 2 2018, 1:07 AM
martinvonz added a comment.Aug 2 2018, 1:08 AM
In D4049#62787, @mharbison72 wrote:

I think Windows will choke on ‘:’ in the file name.

What's a better character for Windows?

mharbison72 added a comment.Aug 2 2018, 1:13 AM

Not sure. If we can’t reuse ‘#’, maybe ‘_’ would be ok as a distinct visual identifier? I don’t think either are reserved characters.

martinvonz added a comment.Aug 2 2018, 1:21 AM
In D4049#62791, @mharbison72 wrote:

Not sure. If we can’t reuse ‘#’, maybe ‘_’ would be ok as a distinct visual identifier? I don’t think either are reserved characters.

I was thinking of using '#'. It would be kind of nice because we already use that for specifying test cases from the command line (e.g. run-tests.py test-foo.t#obsmarkers-on). It would seem consistent with that to use '#' before each dimension. However, in the output files (e.g. '.err'), we use a '.' as separator. I don't know if Windows is the reason for that. A quick internet search suggested it isn't. I'll switch to that and hopefully you or someone else can tell me if it seems to work.

martinvonz edited the summary of this revision. (Show Details)Aug 2 2018, 1:38 AM
martinvonz updated this revision to Diff 9784.
martinvonz added a comment.Aug 2 2018, 1:39 AM
In D4049#62792, @martinvonz wrote:
In D4049#62791, @mharbison72 wrote:

Not sure. If we can’t reuse ‘#’, maybe ‘_’ would be ok as a distinct visual identifier? I don’t think either are reserved characters.

I was thinking of using '#'. It would be kind of nice because we already use that for specifying test cases from the command line (e.g. run-tests.py test-foo.t#obsmarkers-on). It would seem consistent with that to use '#' before each dimension. However, in the output files (e.g. '.err'), we use a '.' as separator. I don't know if Windows is the reason for that. A quick internet search suggested it isn't. I'll switch to that and hopefully you or someone else can tell me if it seems to work.

I inserted a parent (D4052) that changes the format of .err files and changed the separator here to #. I like this much better. Hopefully it works on Windows.

lothiraldan added a subscriber: lothiraldan.Aug 2 2018, 5:29 AM
lothiraldan added inline comments.
tests/test-run-tests.t
912

Could you add a combination that failed so we test the test name of combination cases?

martinvonz marked an inline comment as done.Aug 2 2018, 11:00 AM
martinvonz updated this revision to Diff 9790.
martinvonz added a comment.Aug 2 2018, 11:00 AM

Added a failed test

mharbison72 added a comment.Aug 3 2018, 1:00 AM

I'm getting an import error, and I'm not sure how to figure out the parent revision from phab.

$ hg phabimport D4049
applying patch from stdin
patching file tests/run-tests.py
Hunk #2 FAILED at 1241
Hunk #7 FAILED at 2724
abort: patch failed to apply

$ hg config alias.phabimport
!"%HG%" phabread $1 --stack | "%HG%" import --bypass -
martinvonz added a comment.Aug 3 2018, 1:15 AM
In D4049#63018, @mharbison72 wrote:

I'm getting an import error, and I'm not sure how to figure out the parent revision from phab.

Make sure you're applying it onto @ from https://www.mercurial-scm.org/repo/hg-committed/. If not, you'll probably have to import D4052 before this patch (D4052 is in @ on hg-committed).

mharbison72 added a comment.Aug 3 2018, 1:15 AM

I manually fixed it up, and got the following diffs. I see the last parent is listed as D4067, which hasn't been accepted yet, so I'm assuming that's the issue (I'm way behind on the ML). And I guess --stack doesn't pick up parent dependencies, just what was submitted together?

--- c:/Users/Matt/projects/hg/tests/test-run-tests.t
+++ c:/Users/Matt/projects/hg/tests/test-run-tests.t.err
@@ -850,7 +850,7 @@
   > EOF

   --- $TESTTMP/test-cases.t
-  +++ $TESTTMP/test-cases.t.a.err
+  +++ $TESTTMP/test-cases.t#a.err
   @@ -1,6 +1,7 @@
    #testcases a b
    #if a
@@ -861,7 +861,7 @@
      $ echo 2
   Accept this change? [n] .
   --- $TESTTMP/test-cases.t
-  +++ $TESTTMP/test-cases.t.b.err
+  +++ $TESTTMP/test-cases.t#b.err
   @@ -5,4 +5,5 @@
    #endif
    #if b
@@ -1574,7 +1574,7 @@
   $ rt
   .
   --- $TESTTMP/anothertests/cases/test-cases-abc.t
-  +++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err
+  +++ $TESTTMP/anothertests/cases/test-cases-abc.t#B.err
   @@ -7,7 +7,7 @@
      $ V=C
    #endif
@@ -1597,7 +1597,7 @@
   $ rt --restart

   --- $TESTTMP/anothertests/cases/test-cases-abc.t
-  +++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err
+  +++ $TESTTMP/anothertests/cases/test-cases-abc.t#B.err
   @@ -7,7 +7,7 @@
      $ V=C
    #endif
@@ -1619,10 +1619,12 @@

   $ mkdir output
   $ mv test-cases-abc.t.B.err output
+  mv: cannot stat `test-cases-abc.t.B.err': $ENOENT$
+  [1]
   $ rt --restart --outputdir output
-
+  .
   --- $TESTTMP/anothertests/cases/test-cases-abc.t
-  +++ $TESTTMP/anothertests/cases/output/test-cases-abc.t.B.err
+  +++ $TESTTMP/anothertests/cases/output/test-cases-abc.t#B.err
   @@ -7,7 +7,7 @@
      $ V=C
    #endif
@@ -1636,8 +1638,9 @@
   ERROR: test-cases-abc.t#B output changed
   !.
   Failed test-cases-abc.t#B: output changed
-  # Ran 2 tests, 0 skipped, 1 failed.
-  python hash seed: * (glob)
+  # Ran 3 tests, 0 skipped, 1 failed.
+  python hash seed: * (glob)
+  running all tests
   [1]

 Test TESTCASE variable
@@ -1665,7 +1668,7 @@
   $ rt "test-cases-abc.t#B"

   --- $TESTTMP/anothertests/cases/test-cases-abc.t
-  +++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err
+  +++ $TESTTMP/anothertests/cases/test-cases-abc.t#B.err
   @@ -7,7 +7,7 @@
      $ V=C
    #endif
@@ -1688,7 +1691,7 @@
   $ rt test-cases-abc.t#B test-cases-abc.t#C

   --- $TESTTMP/anothertests/cases/test-cases-abc.t
-  +++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err
+  +++ $TESTTMP/anothertests/cases/test-cases-abc.t#B.err
   @@ -7,7 +7,7 @@
      $ V=C
    #endif
@@ -1711,7 +1714,7 @@
   $ rt test-cases-abc.t#B test-cases-abc.t#D

   --- $TESTTMP/anothertests/cases/test-cases-abc.t
-  +++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err
+  +++ $TESTTMP/anothertests/cases/test-cases-abc.t#B.err
   @@ -7,7 +7,7 @@
      $ V=C
    #endif
@@ -1745,7 +1748,7 @@
   $ rt test-cases-advanced-cases.t

   --- $TESTTMP/anothertests/cases/test-cases-advanced-cases.t
-  +++ $TESTTMP/anothertests/cases/test-cases-advanced-cases.t.case-with-dashes.err
+  +++ $TESTTMP/anothertests/cases/test-cases-advanced-cases.t#case-with-dashes.err
   @@ -1,3 +1,3 @@
    #testcases simple case-with-dashes casewith_-.chars
      $ echo $TESTCASE
@@ -1755,7 +1758,7 @@
   ERROR: test-cases-advanced-cases.t#case-with-dashes output changed
   !
   --- $TESTTMP/anothertests/cases/test-cases-advanced-cases.t
-  +++ $TESTTMP/anothertests/cases/test-cases-advanced-cases.t.casewith_-.chars.err
+  +++ $TESTTMP/anothertests/cases/test-cases-advanced-cases.t#casewith_-.chars.err
   @@ -1,3 +1,3 @@
    #testcases simple case-with-dashes casewith_-.chars
      $ echo $TESTCASE
@@ -1773,7 +1776,7 @@
   $ rt "test-cases-advanced-cases.t#case-with-dashes"

   --- $TESTTMP/anothertests/cases/test-cases-advanced-cases.t
-  +++ $TESTTMP/anothertests/cases/test-cases-advanced-cases.t.case-with-dashes.err
+  +++ $TESTTMP/anothertests/cases/test-cases-advanced-cases.t#case-with-dashes.err
   @@ -1,3 +1,3 @@
    #testcases simple case-with-dashes casewith_-.chars
      $ echo $TESTCASE
@@ -1790,7 +1793,7 @@
   $ rt "test-cases-advanced-cases.t#casewith_-.chars"

   --- $TESTTMP/anothertests/cases/test-cases-advanced-cases.t
-  +++ $TESTTMP/anothertests/cases/test-cases-advanced-cases.t.casewith_-.chars.err
+  +++ $TESTTMP/anothertests/cases/test-cases-advanced-cases.t#casewith_-.chars.err
   @@ -1,3 +1,3 @@
    #testcases simple case-with-dashes casewith_-.chars
      $ echo $TESTCASE

ERROR: test-run-tests.t output changed
!
mharbison72 added a comment.Aug 3 2018, 1:18 AM
In D4049#63019, @martinvonz wrote:
In D4049#63018, @mharbison72 wrote:

I'm getting an import error, and I'm not sure how to figure out the parent revision from phab.

Make sure you're applying it onto @ from https://www.mercurial-scm.org/repo/hg-committed/. If not, you'll probably have to import D4052 before this patch (D4052 is in @ on hg-committed).

Maybe needs a push? I'm seeing @ on D4048 on hg-committed.

martinvonz added a comment.Aug 3 2018, 1:20 AM
In D4049#63024, @mharbison72 wrote:
In D4049#63019, @martinvonz wrote:
In D4049#63018, @mharbison72 wrote:

I'm getting an import error, and I'm not sure how to figure out the parent revision from phab.

Make sure you're applying it onto @ from https://www.mercurial-scm.org/repo/hg-committed/. If not, you'll probably have to import D4052 before this patch (D4052 is in @ on hg-committed).

Maybe needs a push? I'm seeing @ on D4048 on hg-committed.

Sorry, I was just misremembering that D4052 had been queued. I don't know why phabricator doesn't understand that this patch is on top of D4052. Anyway, try applying D4052 first, then this one.

quark added a subscriber: quark.Aug 3 2018, 2:00 AM
In D4049#63026, @martinvonz wrote:

Sorry, I was just misremembering that D4052 had been queued. I don't know why phabricator doesn't understand that this patch is on top of D4052. Anyway, try applying D4052 first, then this one.

There is no formal API to set dependency. So you need to click "Edit Related Revisions... -> Edit Parent Revisions" on the right sidebar manually.

mharbison72 accepted this revision.Aug 3 2018, 2:01 AM
In D4049#63026, @martinvonz wrote:
In D4049#63024, @mharbison72 wrote:
In D4049#63019, @martinvonz wrote:
In D4049#63018, @mharbison72 wrote:

I'm getting an import error, and I'm not sure how to figure out the parent revision from phab.

Make sure you're applying it onto @ from https://www.mercurial-scm.org/repo/hg-committed/. If not, you'll probably have to import D4052 before this patch (D4052 is in @ on hg-committed).

Maybe needs a push? I'm seeing @ on D4048 on hg-committed.

Sorry, I was just misremembering that D4052 had been queued. I don't know why phabricator doesn't understand that this patch is on top of D4052. Anyway, try applying D4052 first, then this one.

That did the trick, and this runs clean on Windows, thanks.

It seems to add ~4 seconds to test-run-tests.t, but that's probably not a big deal if it doesn't affect all of the other tests like this.

martinvonz added a parent revision: D4052: testrunner: use "#" for "test cases" suffix in .err filename too.Aug 3 2018, 2:02 AM
In D4049#63029, @mharbison72 wrote:
In D4049#63026, @martinvonz wrote:
In D4049#63024, @mharbison72 wrote:
In D4049#63019, @martinvonz wrote:
In D4049#63018, @mharbison72 wrote:

I'm getting an import error, and I'm not sure how to figure out the parent revision from phab.

Make sure you're applying it onto @ from https://www.mercurial-scm.org/repo/hg-committed/. If not, you'll probably have to import D4052 before this patch (D4052 is in @ on hg-committed).

Maybe needs a push? I'm seeing @ on D4048 on hg-committed.

Sorry, I was just misremembering that D4052 had been queued. I don't know why phabricator doesn't understand that this patch is on top of D4052. Anyway, try applying D4052 first, then this one.

That did the trick, and this runs clean on Windows, thanks.

Great. Thanks for checking!

It seems to add ~4 seconds to test-run-tests.t, but that's probably not a big deal if it doesn't affect all of the other tests like this.

That's probably just because it adds a recursive call to run 4 tests. I doubt it has any measurable impact on other tests.

martinvonz added a comment.Aug 3 2018, 2:04 AM
In D4049#63027, @quark wrote:
In D4049#63026, @martinvonz wrote:

Sorry, I was just misremembering that D4052 had been queued. I don't know why phabricator doesn't understand that this patch is on top of D4052. Anyway, try applying D4052 first, then this one.

There is no formal API to set dependency. So you need to click "Edit Related Revisions... -> Edit Parent Revisions" on the right sidebar manually.

Thanks! I had never noticed that sidebar.

mharbison72 added a comment.Aug 3 2018, 2:07 AM
In D4049#63027, @quark wrote:
In D4049#63026, @martinvonz wrote:

Sorry, I was just misremembering that D4052 had been queued. I don't know why phabricator doesn't understand that this patch is on top of D4052. Anyway, try applying D4052 first, then this one.

There is no formal API to set dependency. So you need to click "Edit Related Revisions... -> Edit Parent Revisions" on the right sidebar manually.

That's the closest thing I found to determining what the parent was. Even if there's no formal API to set that dependency, is there an API to read the parent? I'm wondering if an --exact switch is possible on phabread, similar to --stack, but pull in all parents recursively.

quark added a comment.Aug 3 2018, 2:16 AM

--stack should work as expected if dependency is set manually.

pulkit added a subscriber: pulkit.Aug 3 2018, 9:59 AM

Thanks to @mharbison72 for review and testing it on windows. Queuing this, many thanks!

pulkit accepted this revision.Aug 3 2018, 10:00 AM
pulkit added inline comments.Aug 3 2018, 10:06 AM
tests/run-tests.py
2705–2706

Fixed this bad indent in flight.

Closed by commit rHG3086a8627b29: testrunner: allow multiple #testcases (authored by martinvonz). · Explain WhyAug 3 2018, 10:27 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathPackages
Mtests/run-tests.py (60 lines)
Mtests/test-run-tests.t (34 lines)

Diff 9825

tests/run-tests.py

f.close() f.close()
return entries return entries
def parsettestcases(path): def parsettestcases(path):
"""read a .t test file, return a set of test case names """read a .t test file, return a set of test case names
If path does not exist, return an empty set. If path does not exist, return an empty set.
""" """
cases = set() cases = []
try: try:
with open(path, 'rb') as f: with open(path, 'rb') as f:
for l in f: for l in f:
if l.startswith(b'#testcases '): if l.startswith(b'#testcases '):
cases.update(l[11:].split()) cases.append(sorted(l[11:].split()))
except IOError as ex: except IOError as ex:
if ex.errno != errno.ENOENT: if ex.errno != errno.ENOENT:
raise raise
return cases return cases
def getparser(): def getparser():
"""Obtain the OptionParser used by the CLI.""" """Obtain the OptionParser used by the CLI."""
parser = argparse.ArgumentParser(usage='%(prog)s [options] [tests]') parser = argparse.ArgumentParser(usage='%(prog)s [options] [tests]')
NEEDESCAPE = re.compile(br'[\x00-\x08\x0b-\x1f\x7f-\xff]').search NEEDESCAPE = re.compile(br'[\x00-\x08\x0b-\x1f\x7f-\xff]').search
ESCAPESUB = re.compile(br'[\x00-\x08\x0b-\x1f\\\x7f-\xff]').sub ESCAPESUB = re.compile(br'[\x00-\x08\x0b-\x1f\\\x7f-\xff]').sub
ESCAPEMAP = dict((bchr(i), br'\x%02x' % i) for i in range(256)) ESCAPEMAP = dict((bchr(i), br'\x%02x' % i) for i in range(256))
ESCAPEMAP.update({b'\\': b'\\\\', b'\r': br'\r'}) ESCAPEMAP.update({b'\\': b'\\\\', b'\r': br'\r'})
def __init__(self, path, *args, **kwds): def __init__(self, path, *args, **kwds):
# accept an extra "case" parameter # accept an extra "case" parameter
case = kwds.pop('case', None) case = kwds.pop('case', [])
self._case = case self._case = case
self._allcases = parsettestcases(path) self._allcases = {x for y in parsettestcases(path) for x in y}
super(TTest, self).__init__(path, *args, **kwds) super(TTest, self).__init__(path, *args, **kwds)
if case: if case:
self.name = '%s#%s' % (self.name, _strpath(case)) casepath = _strpath('#'.join(case))
self.errpath = b'%s#%s.err' % (self.errpath[:-4], case) self.name = '%s#%s' % (self.name, casepath)
self._tmpname += b'-%s' % case self.errpath = b'%s#%s.err' % (self.errpath[:-4], casepath)
self._tmpname += b'-%s' % casepath
self._have = {} self._have = {}
@property @property
def refpath(self): def refpath(self):
return os.path.join(self._testdir, self.bname) return os.path.join(self._testdir, self.bname)
def _run(self, env): def _run(self, env):
with open(self.path, 'rb') as f: with open(self.path, 'rb') as f:
self._have[allreqs] = (True, None) self._have[allreqs] = (True, None)
return True, None return True, None
def _iftest(self, args): def _iftest(self, args):
# implements "#if" # implements "#if"
reqs = [] reqs = []
for arg in args: for arg in args:
if arg.startswith(b'no-') and arg[3:] in self._allcases: if arg.startswith(b'no-') and arg[3:] in self._allcases:
if arg[3:] == self._case: if arg[3:] in self._case:
return False return False
elif arg in self._allcases: elif arg in self._allcases:
if arg != self._case: if arg not in self._case:
return False return False
else: else:
reqs.append(arg) reqs.append(arg)
return self._hghave(reqs)[0] return self._hghave(reqs)[0]
def _parsetest(self, lines): def _parsetest(self, lines):
# We generate a shell script which outputs unique markers to line # We generate a shell script which outputs unique markers to line
# up script results with our source. These markers include input # up script results with our source. These markers include input
if self._debug: if self._debug:
script.append(b'set -x\n') script.append(b'set -x\n')
if self._hgcommand != b'hg': if self._hgcommand != b'hg':
script.append(b'alias hg="%s"\n' % self._hgcommand) script.append(b'alias hg="%s"\n' % self._hgcommand)
if os.getenv('MSYSTEM'): if os.getenv('MSYSTEM'):
script.append(b'alias pwd="pwd -W"\n') script.append(b'alias pwd="pwd -W"\n')
if self._case: if self._case:
casestr = '#'.join(self._case)
if isinstance(self._case, str): if isinstance(self._case, str):
quoted = shellquote(self._case) quoted = shellquote(casestr)
else: else:
quoted = shellquote(self._case.decode('utf8')).encode('utf8') quoted = shellquote(casestr.decode('utf8')).encode('utf8')
script.append(b'TESTCASE=%s\n' % quoted) script.append(b'TESTCASE=%s\n' % quoted)
script.append(b'export TESTCASE\n') script.append(b'export TESTCASE\n')
n = 0 n = 0
for n, l in enumerate(lines): for n, l in enumerate(lines):
if not l.endswith(b'\n'): if not l.endswith(b'\n'):
l += b'\n' l += b'\n'
if l.startswith(b'#require'): if l.startswith(b'#require'):
if os.path.isdir(arg): if os.path.isdir(arg):
if not arg.endswith(b'/'): if not arg.endswith(b'/'):
arg += b'/' arg += b'/'
expanded_args.extend([arg + a for a in os.listdir(arg)]) expanded_args.extend([arg + a for a in os.listdir(arg)])
else: else:
expanded_args.append(arg) expanded_args.append(arg)
args = expanded_args args = expanded_args
testcasepattern = re.compile(br'([\w-]+\.t|py)(#([a-zA-Z0-9_\-\.]+))') testcasepattern = re.compile(br'([\w-]+\.t|py)(#([a-zA-Z0-9_\-\.#]+))')
tests = [] tests = []
for t in args: for t in args:
case = None case = []
if not (os.path.basename(t).startswith(b'test-') if not (os.path.basename(t).startswith(b'test-')
and (t.endswith(b'.py') or t.endswith(b'.t'))): and (t.endswith(b'.py') or t.endswith(b'.t'))):
m = testcasepattern.match(t) m = testcasepattern.match(t)
if m is not None: if m is not None:
t, _, case = m.groups() t, _, casestr = m.groups()
if casestr:
case = casestr.split('#')
else: else:
continue continue
if t.endswith(b'.t'): if t.endswith(b'.t'):
# .t file may contain multiple test cases # .t file may contain multiple test cases
cases = sorted(parsettestcases(t)) casedimensions = parsettestcases(t)
if cases: if casedimensions:
if case is not None and case in cases: cases = []
tests += [{'path': t, 'case': case}] def addcases(case, casedimensions):
elif case is not None and case not in cases: if not casedimensions:
cases.append(case)
else:
for c in casedimensions[0]:
addcases(case + [c], casedimensions[1:])
addcases([], casedimensions)
if case and case in cases:
cases = [case]
elif case:
# Ignore invalid cases # Ignore invalid cases
pass cases = []
else: else:
pass
tests += [{'path': t, 'case': c} for c in sorted(cases)] tests += [{'path': t, 'case': c} for c in sorted(cases)]
pulkitUnsubmitted
Not Done

Fixed this bad indent in flight.

pulkit: Fixed this bad indent in flight.
else: else:
tests.append({'path': t}) tests.append({'path': t})
else: else:
tests.append({'path': t}) tests.append({'path': t})
return tests return tests
def _runtests(self, testdescs): def _runtests(self, testdescs):
def _reloadtest(test, i): def _reloadtest(test, i):
# convert a test back to its description dict # convert a test back to its description dict
desc = {'path': test.path} desc = {'path': test.path}
case = getattr(test, '_case', None) case = getattr(test, '_case', [])
if case: if case:
desc['case'] = case desc['case'] = case
return self._gettest(desc, i) return self._gettest(desc, i)
try: try:
if self.options.restart: if self.options.restart:
orig = list(testdescs) orig = list(testdescs)
while testdescs: while testdescs:
desc = testdescs[0] desc = testdescs[0]
# desc['path'] is a relative path # desc['path'] is a relative path
if 'case' in desc: if 'case' in desc:
errpath = b'%s#%s.err' % (desc['path'], desc['case']) casestr = '#'.join(desc['case'])
errpath = b'%s#%s.err' % (desc['path'], casestr)
else: else:
errpath = b'%s.err' % desc['path'] errpath = b'%s.err' % desc['path']
errpath = os.path.join(self._outputdir, errpath) errpath = os.path.join(self._outputdir, errpath)
if os.path.exists(errpath): if os.path.exists(errpath):
break break
testdescs.pop(0) testdescs.pop(0)
if not testdescs: if not testdescs:
print("running all tests") print("running all tests")

tests/test-run-tests.t

> $ echo $NAME > $ echo $NAME
> A (a !) > A (a !)
> B (b !) > B (b !)
> EOF > EOF
$ rt test-cases.t $ rt test-cases.t
.. ..
# Ran 2 tests, 0 skipped, 0 failed. # Ran 2 tests, 0 skipped, 0 failed.
When using multiple dimensions of "#testcases" in .t files
$ cat > test-cases.t <<'EOF'
> #testcases a b
> #testcases c d
> #if a d
> $ echo $TESTCASE
> a#d
> #endif
> #if b c
> $ echo yes
> no
> #endif
> EOF
lothiraldanUnsubmitted
Done

Could you add a combination that failed so we test the test name of combination cases?

lothiraldan: Could you add a combination that failed so we test the test name of combination cases?
$ rt test-cases.t
..
--- $TESTTMP/test-cases.t
+++ $TESTTMP/test-cases.t#b#c.err
@@ -6,5 +6,5 @@
#endif
#if b c
$ echo yes
- no
+ yes
#endif
ERROR: test-cases.t#b#c output changed
!.
Failed test-cases.t#b#c: output changed
# Ran 4 tests, 0 skipped, 1 failed.
python hash seed: * (glob)
[1]
$ rm test-cases.t#b#c.err
$ rm test-cases.t $ rm test-cases.t
(reinstall) (reinstall)
$ mv backup test-failure.t $ mv backup test-failure.t
No Diff No Diff
=============== ===============