This is an archive of the discontinued Mercurial Phabricator instance.

Differential D3699

run-tests: follow-up on the test-case format
ClosedPublic

Authored by lothiraldan on Jun 7 2018, 3:19 PM.

Download Raw Diff

Details

Reviewers

indygreg

Group Reviewers

hg-reviewers

Commits

rHGf2e3196a34f9: run-tests: follow-up on the test-case format

Summary

It turns out the original regex doesn't support real test cases names like the
one Mercurial is using. Update the regex to being able to precisely select
them on the command line.

Diff Detail

Repository

rHG Mercurial

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

lothiraldan created this revision.Jun 7 2018, 3:19 PM

Herald added a reviewer: hg-reviewers. · View Herald TranscriptJun 7 2018, 3:19 PM

Herald added a subscriber: mercurial-devel. · View Herald Transcript

lothiraldan added a child revision: D3700: run-tests: add support for external test result.Jun 7 2018, 3:19 PM

indygreg accepted this revision.Jun 7 2018, 7:44 PM

This revision is now accepted and ready to land.Jun 7 2018, 7:44 PM

Closed by commit rHGf2e3196a34f9: run-tests: follow-up on the test-case format (authored by lothiraldan). · Explain WhyJun 7 2018, 7:48 PM

This revision was automatically updated to reflect the committed changes.

mharbison72 added a subscriber: mharbison72.Jun 8 2018, 9:50 PM

mharbison72 added inline comments.

tests/test-run-tests.t
1669	Windows can't create the test directory with all of the reserved characters here. Are these necessary?

test-run-tests.t:1669
+ --- $TESTTMP/anothertests/cases/test-cases-advanced-cases.t
+ +++ $TESTTMP/anothertests/cases/test-cases-advanced-cases.t.casewith!@#$%^&*()chars.err
+ @@ -1,3 +1,3 @@

Windows can't create the test directory with all of the reserved characters here. Are these necessary?

Maybe no? I think alphanumerics + '-' should be enough, and allowing shell
meta characters is potentially unsafe.

In D3699#58174, @yuja wrote:

test-run-tests.t:1669
+ --- $TESTTMP/anothertests/cases/test-cases-advanced-cases.t
+ +++ $TESTTMP/anothertests/cases/test-cases-advanced-cases.t.casewith!@#$%^&*()chars.err
+ @@ -1,3 +1,3 @@

Windows can't create the test directory with all of the reserved characters here. Are these necessary?

Maybe no? I think alphanumerics + '-' should be enough, and allowing shell
meta characters is potentially unsafe.

I checked core test files and some extensions test files. Alphanumerics + '-" would works for most of them. Some tests in the hg-experimental repository have a . in their test case name. I would also add _ for allowing all naming cases. Does that sounds good?

@mharbison72 I was not aware that this patch had been merged, is it breaking the windows build right now?

I have sends https://phab.mercurial-scm.org/D3721 to restrict the characters allowed in test-cases, we might want to add a warning or an error when such invalid names are used so the user won't get confused that he cannot select them

In D3699#58394, @lothiraldan wrote:

@mharbison72 I was not aware that this patch had been merged, is it breaking the windows build right now?

Yes.

https://buildbot.mercurial-scm.org/builders/Win7%20x86_64%20hg%20tests/builds/754/steps/run-tests.py%20%28python%202.7.13%29/logs/stdio

Revision Contents
Changeset List

			Path	Packages
M			tests/run-tests.py (4 lines)
M			tests/test-run-tests.t (77 lines)

Status	Author	Revision
Closed	lothiraldan	D3701 run-tests: extract onStart and onEnd into the test result
Closed	lothiraldan	D3700 run-tests: add support for external test result
Closed	lothiraldan	D3699 run-tests: follow-up on the test-case format

Diff 8995

tests/run-tests.py

	styles = {			styles = {
	skipped: '#e5e5e5',			skipped: '#e5e5e5',
	skippedname: '#00ffff',			skippedname: '#00ffff',
	failed: '#7f0000',			failed: '#7f0000',
	failedname: '#ff0000',			failedname: '#ff0000',
	}			}

	class TestRunnerLexer(lexer.RegexLexer):			class TestRunnerLexer(lexer.RegexLexer):
	testpattern = r'[\w-]+\.(t\|py)(#[\w-]+)?'			testpattern = r'[\w-]+\.(t\|py)(#[^\s]+)?'
	tokens = {			tokens = {
	'root': [			'root': [
	(r'^Skipped', token.Generic.Skipped, 'skipped'),			(r'^Skipped', token.Generic.Skipped, 'skipped'),
	(r'^Failed ', token.Generic.Failed, 'failed'),			(r'^Failed ', token.Generic.Failed, 'failed'),
	(r'^ERROR: ', token.Generic.Failed, 'failed'),			(r'^ERROR: ', token.Generic.Failed, 'failed'),
	],			],
	'skipped': [			'skipped': [
	(testpattern, token.Generic.SName),			(testpattern, token.Generic.SName),
	if os.path.isdir(arg):			if os.path.isdir(arg):
	if not arg.endswith(b'/'):			if not arg.endswith(b'/'):
	arg += b'/'			arg += b'/'
	expanded_args.extend([arg + a for a in os.listdir(arg)])			expanded_args.extend([arg + a for a in os.listdir(arg)])
	else:			else:
	expanded_args.append(arg)			expanded_args.append(arg)
	args = expanded_args			args = expanded_args

	testcasepattern = re.compile(r'([\w-]+\.t\|py)(#([\w-])+)')			testcasepattern = re.compile(r'([\w-]+\.t\|py)(#([^\s]+))')
	tests = []			tests = []
	for t in args:			for t in args:
	case = None			case = None

	if not (os.path.basename(t).startswith(b'test-')			if not (os.path.basename(t).startswith(b'test-')
	and (t.endswith(b'.py') or t.endswith(b'.t'))):			and (t.endswith(b'.py') or t.endswith(b'.t'))):

	m = testcasepattern.match(t)			m = testcasepattern.match(t)

tests/test-run-tests.t


	ERROR: test-cases-abc.t#B output changed			ERROR: test-cases-abc.t#B output changed
	!.			!.
	Failed test-cases-abc.t#B: output changed			Failed test-cases-abc.t#B: output changed
	# Ran 2 tests, 0 skipped, 1 failed.			# Ran 2 tests, 0 skipped, 1 failed.
	python hash seed: * (glob)			python hash seed: * (glob)
	[1]			[1]

	Support running invalid test cases			Support ignoring invalid test cases

	$ rt test-cases-abc.t#B test-cases-abc.t#D			$ rt test-cases-abc.t#B test-cases-abc.t#D

	--- $TESTTMP/anothertests/cases/test-cases-abc.t			--- $TESTTMP/anothertests/cases/test-cases-abc.t
	+++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err			+++ $TESTTMP/anothertests/cases/test-cases-abc.t.B.err
	@@ -7,7 +7,7 @@			@@ -7,7 +7,7 @@
	$ V=C			$ V=C
	#endif			#endif
	$ echo $V \| sed 's/A/C/'			$ echo $V \| sed 's/A/C/'
	- C			- C
	+ B			+ B
	#if C			#if C
	$ [ $V = C ]			$ [ $V = C ]
	#endif			#endif

	ERROR: test-cases-abc.t#B output changed			ERROR: test-cases-abc.t#B output changed
	!			!
	Failed test-cases-abc.t#B: output changed			Failed test-cases-abc.t#B: output changed
	# Ran 1 tests, 0 skipped, 1 failed.			# Ran 1 tests, 0 skipped, 1 failed.
	python hash seed: * (glob)			python hash seed: * (glob)
	[1]			[1]

				Support running complex test cases names

				$ cat > test-cases-advanced-cases.t <<'EOF'
				> #testcases simple case-with-dashes casewith!@#$%^&*()chars
				> $ echo $TESTCASE
				> simple
				> EOF

				$ cat test-cases-advanced-cases.t
				#testcases simple case-with-dashes casewith!@#$%^&*()chars
				$ echo $TESTCASE
				simple

				$ rt test-cases-advanced-cases.t

				--- $TESTTMP/anothertests/cases/test-cases-advanced-cases.t
				+++ $TESTTMP/anothertests/cases/test-cases-advanced-cases.t.case-with-dashes.err
				@@ -1,3 +1,3 @@
				#testcases simple case-with-dashes casewith!@#$%^&*()chars
				$ echo $TESTCASE
				- simple
				+ case-with-dashes

				ERROR: test-cases-advanced-cases.t#case-with-dashes output changed
				!
				--- $TESTTMP/anothertests/cases/test-cases-advanced-cases.t
				+++ $TESTTMP/anothertests/cases/test-cases-advanced-cases.t.casewith!@#$%^&*()chars.err
				mharbison72Unsubmitted Not Done Windows can't create the test directory with all of the reserved characters here. Are these necessary? mharbison72: Windows can't create the test directory with all of the reserved characters here. Are these…
				@@ -1,3 +1,3 @@
				#testcases simple case-with-dashes casewith!@#$%^&*()chars
				$ echo $TESTCASE
				- simple
				+ casewith!@#$%^&*()chars

				ERROR: test-cases-advanced-cases.t#casewith!@#$%^&*()chars output changed
				!.
				Failed test-cases-advanced-cases.t#case-with-dashes: output changed
				Failed test-cases-advanced-cases.t#casewith!@#$%^&*()chars: output changed
				# Ran 3 tests, 0 skipped, 2 failed.
				python hash seed: * (glob)
				[1]

				$ rt "test-cases-advanced-cases.t#case-with-dashes"

				--- $TESTTMP/anothertests/cases/test-cases-advanced-cases.t
				+++ $TESTTMP/anothertests/cases/test-cases-advanced-cases.t.case-with-dashes.err
				@@ -1,3 +1,3 @@
				#testcases simple case-with-dashes casewith!@#$%^&*()chars
				$ echo $TESTCASE
				- simple
				+ case-with-dashes

				ERROR: test-cases-advanced-cases.t#case-with-dashes output changed
				!
				Failed test-cases-advanced-cases.t#case-with-dashes: output changed
				# Ran 1 tests, 0 skipped, 1 failed.
				python hash seed: * (glob)
				[1]

				$ rt "test-cases-advanced-cases.t#casewith!@#$%^&*()chars"

				--- $TESTTMP/anothertests/cases/test-cases-advanced-cases.t
				+++ $TESTTMP/anothertests/cases/test-cases-advanced-cases.t.casewith!@#$%^&*()chars.err
				@@ -1,3 +1,3 @@
				#testcases simple case-with-dashes casewith!@#$%^&*()chars
				$ echo $TESTCASE
				- simple
				+ casewith!@#$%^&*()chars

				ERROR: test-cases-advanced-cases.t#casewith!@#$%^&*()chars output changed
				!
				Failed test-cases-advanced-cases.t#casewith!@#$%^&*()chars: output changed
				# Ran 1 tests, 0 skipped, 1 failed.
				python hash seed: * (glob)
				[1]

	Test automatic pattern replacement			Test automatic pattern replacement
	==================================			==================================

	$ cat << EOF >> common-pattern.py			$ cat << EOF >> common-pattern.py
	> substitutions = [			> substitutions = [
	> (br'foo-(.*)\\b',			> (br'foo-(.*)\\b',
	> br'\$XXX=\\1\$'),			> br'\$XXX=\\1\$'),
	> (br'bar\\n',			> (br'bar\\n',

Diff	ID	Description	Created	Lint	Unit
Base		Base
Diff 1	8991		Jun 7 2018, 3:19 PM	★	★
Diff 2	8995	rHGf2e3196a34f967e0a02570e1945e3c89c0e3db61	Jun 7 2018, 3:09 PM	★	★