No tests run using exclude tags about gradle-scalatest

[maiflai]

That's odd - taking https://github.com/xanderdunn/reinforcementLearning as an example I find that test runs 1 test which passes.

When I uncomment the second test in src/test/scala/ticTacToeTests.scala it runs and fails:

:test
Discovery starting.
Discovery completed in 126 milliseconds.
Run starting. Expected test count is: 2
TicTacToeSpec:
Tic-tac-toe learning
NeuralNetSpec:
A NeuralNet
- should have greater than 79.0% X wins for Neural Net vs. Random (11 minutes, 56 seconds)
  + X won 88.485% of games 
- should have greater than 90% stalemates for Neural vs. Neural *** FAILED *** (12 minutes, 29 seconds)
  0.0 was not greater than 0.9 (ticTacToeTests.scala:79)
Run completed in 12 minutes, 29 seconds.
Total number of tests run: 2
Suites: completed 3, aborted 0
Tests: succeeded 1, failed 1, canceled 0, ignored 0, pending 0
*** 1 TEST FAILED ***
TicTacToeSpec:
Tic-tac-toe learning 
- should have greater than 90% stalemates for Neural vs. Neural *** FAILED *** (12 minutes, 29 seconds)
  0.0 was not greater than 0.9 (ticTacToeTests.scala:79)
:test FAILED

Is this an appropriate repository to use as a working example? If so, please could you push any updates and I'll see if I can reproduce the problem.

[xanderdunn]

Yeah, I get the same results that you do for master on GitHub.

I've pushed my changes to the unitTests branch. With this code, I run gradle clean && gradle test and it outputs:

:test
Discovery starting.
Discovery completed in 94 milliseconds.
Run starting. Expected test count is: 2
Run completed in 156 milliseconds.
Total number of tests run: 0
Suites: completed 1, aborted 0
Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
No tests were executed.

BUILD SUCCESSFUL

The most significant difference related to unit tests that I can think of is that I created custom tag classes and tagged all of my unit tests. If I remove the exclude tag lines from my build.gradle, then it runs all the tests as expected.

Thanks a lot for your help!

[maiflai]

Sorry - is this a typo?

I think you mean that you want to include org.scalatest.tags.UnitTest, but the currentbuild.gradle marks them as exclude.

Perhaps this is what it should be?

test {
    tags {
        include 'org.scalatest.tags.UnitTest' // include Unit
        exclude 'org.scalatest.tags.CoverageAcceptanceTest' // exclude Coverage
    }
}

I guess that scalatest is reporting a single suite to indicate the wrapping suite (the DiscoverySuite).

[xanderdunn]

No, this is not a typo. I have two test targets: test and testScoverage.

I want test to run nothing but those tests with the tag NonCoverageAcceptanceTest. Hence, I excluded all other test tags.

I want testScoverage to run those tests tagged CoverageAcceptanceTest or UnitTest.

[maiflai]

Hmm - yes - from the documentation I see that:

If tags to include is not specified, then all tests except those mentioned in the tags to exclude (and in the org.scalatest.Ignore tag), will be executed

So I agree that excluding those two tags should have the desired effect.

I am however getting very strange results running the tests by hand (i.e. without Gradle).

Scalatest appears to randomly detect tests when using the PS argument which is intended to enable parallel execution of Suites (I notice that you are using ParallelTestExecution in order to enable this at a test level rather than Suite).

gradle clean test -i prints the command line before running the test. Can you remove the PS argument locally, run the command and see if this helps? Unfortunately PS is always specified, it's just the number which can currently be controlled...

[maiflai]

And it seems that Pn works, it's just PSn that doesn't.

Sorry, I'll have to pick this up tomorrow, I suspect it will require an update to the plugin.

Please could you confirm these findings if you get a mo?

Thanks,
Stu

[maiflai]

I reviewed the git log - PS is used to ensure that the tests are reported in a sane manner when they are run in parallel. I don't think I can remove it.

I did remove the use of ParallelTestExecution in your tests and the build seems to fail reliably. Are you ok with this workaround?

I think this is actually an issue in the underlying scalatest library. I imagine you'd rather continue with working on the problem you're trying to solve, rather than debugging a tool? I'm happy to chase this down with the scalatest guys, unless you fancy it...

Cheers,
Stu

[xanderdunn]

@maiflai Thanks so much for your help! Yeah, this is very helpful. It's a huge performance gain for me to be able to run the tests in parallel, but as a temporary workaround I'll run them without ParallelTestExecution.

It would be awesome if you followed up with the scalatest team, thanks!

Since this doesn't appear to be an issue with gradle-scalatest, shall I close this?

[maiflai]

Thanks - closing this issue now that I've raised scalatest/scalatest#706

[xanderdunn]

@maiflai Thanks!

[jeksmith]

Greetings. I know this issue was closed almost 5 years ago, but it is currently a thorn in our side. And still a problem even with Scalatest 3.2.0. I think we'd be happy just risking it with -P instead of -PS. @maiflai Any way to get that with the current version 0.26? Or would that require a code change?
Thanks.