Comments (14)
@axiopisty Using my fork, my maven config is as follows. All I did in my fork was to write out to a file the word "failure" and then read it in the verify stage. This was just a rough prototype because I wanted to gage interest from the maintainers if they even wanted something like this as a feature.
In order to make this a viable solution there would have to be some more work done in understanding how the maven-failsafe-plugin
writes the reports it reads from and just implement a solution that is similar in the scalatest-maven-plugin
. The one thing I don't know is if this "fail-fast" behavior (of throwing a MojoFailureException
whenever a failure occurs) was implemented for a reason and that changing it would go against someone's vision for the project. However, I think a change like this would only increase compatibility and given how prevalent docker is today, I think it would be a good design decision that would help what I presume to be a lot of people.
<plugin>
<groupId>org.scalatest</groupId>
<artifactId>scalatest-maven-plugin</artifactId>
<version>3.0.0</version>
<configuration>
<runVerifyOnFailure>true</runVerifyOnFailure>
<systemProperties>
<java.awt.headless>true</java.awt.headless>
<mysql.port>${mysql.port}</mysql.port>
</systemProperties>
</configuration>
<executions>
<execution>
<id>test</id>
<phase>integration-test</phase>
<goals>
<goal>test</goal>
</goals>
</execution>
<execution>
<id>verify</id>
<goals>
<goal>verify</goal>
</goals>
</execution>
</executions>
</plugin>
from scalatest-maven-plugin.
I'm running into a similar issue with docker tests. It would be useful to have this functionality
from scalatest-maven-plugin.
@nab0310 I'm experiencing the same problem.
Will you provide an example of your configuration of the scalatest-maven-plugin
that enabled you to work around this?
My scenario is as follows:
I have a multi-module maven project. The main module produces a docker image as an artifact. Another module (the last one to run in my build) is used to build and run end-to-end tests using a combination of the fabric8/docker-maven-plugin
to start the container built from my main artifact along with all the other containers it requires as dependencies, and the scalatest-maven-plugin
to execute my end-to-end integration tests.
Looking at the project build from the perspective of maven phases results in the following simplified view:
Maven Phase -> Plugin:goal
package
->docker-maven-plugin:docker.build
pre-integration-tests
->docker-maven-plugin:docker.start
integration-test
->scalatest-maven-plugin:test
post-integration-tests
->docker-maven-plugin:docker.stop
verify
-> (this is where themaven-failsafe-plugin
fails the build if tests failed)
Using the scalatest-maven-plugin
, if tests fail, the docker containers don't get cleaned up in the post-integration-tests
phase.
This issue gets a big +1 vote from me. Fixing it would make this plugin work more like the maven-failsafe-plugin
(which could be used as a reference for how to implement this feature request) rather than the maven-surefire-plugin
.
from scalatest-maven-plugin.
I did a little research on the maven-failsafe-plugin
to see how it delays reporting the failed build status until the verify phase. Basically, the maven-failsafe-plugin
is a simple plugin closely related to the maven-surefire-plugin
, they are maintained within the same code base. So how does it work?
The maven-failsafe-plugin
has only 2 goals (mojos), the IntegrationTestMojo bound to the integration-test maven phase. This mojo runs the tests and creates a report summary file called failsafe-summary.xml
which is simply a file which reports the status of the tests that were executed.
It contains details such as how many tests:
- completed
- had errors
- had failures
- were skipped
Based on configuration there could be many summary reports of this style. All these are created by the IntegrationTestMojo during the integration-test phase. So, simply put, that Mojo does not fail the build by throwing MojoFailureException
if there are any problems (tests which either had errors or failed). It just writes the results to a file.
Then there is the VerifyMojo
. That goal is bound to the verify phase. All it does (over simplification here) is to read the test summary files that were generated by the IntegrationTestMojo
and check to see if any of the tests would have caused the build to fail. If any did, then it will fail the build by throwing MojoFailureException
.
Failing the build in this manner allows the post-integration-tests
phase to execute and clean up any resources that were allocated in the pre-integration-tests
phase.
The scalatest-maven-plugin
could easily implement a similar approach. All it would need to do is create a summary report as output for the TestMojo
and ReporterMojo
. Those mojos should never throw MojoFailureException
. Then a VerifyMojo
should be created to read the summary reports and throw MojoFailureException
if the build needs to be failed.
from scalatest-maven-plugin.
@nab0310 I think it would be helpful if you were to open a PR with your changes so we can review the changes you made on your branch. If you do, please reference this issue in your PR.
from scalatest-maven-plugin.
I opened the PR
from scalatest-maven-plugin.
@nab0310 @axiopisty I wonder if it is possible if we can produce the same failsure-summary.xml and let maven-failsafe-plugin's VerifyMojo to do its task? The first to achieve that perhaps is that we should not require to disable surefire?
from scalatest-maven-plugin.
@cheeseng Perhaps this plugin could create the xml file and delegate the verification to the failsafe plugin. But it seems to me that it might be easier to just create the functionality directly within this plugin rather than trying to couple everything together with the failsafe plugin. I doubt the maintainers of the surefire and failsafe plugins (which are both maintained from the same source repository) would be considering the xml files they generate and read to be part of some public API. So I tend to think it would be easier to just replicate the idea and implement a solution directly within the scalatest-maven-plugin.
from scalatest-maven-plugin.
@cheeseng I doubt that maintainers of the scalatest-maven-plugin would want to start receiving issues being reported by people who might be using both the scalatest-maven-plugin and the surefire/failsafe plugins. To quote Offspring: "You gotta keep 'em separated!"
from scalatest-maven-plugin.
@axiopisty That's true, looking at @nab0310 's it appears to be very near we can have it already. The PR looks good just I don't have a good way to test it.
from scalatest-maven-plugin.
I wonder if there might be examples of test cases in the failsafe repository.
from scalatest-maven-plugin.
@axiopisty I added a comment to the PR here:
https://github.com/scalatest/scalatest-maven-plugin/pull/60/files#r911599853
I don't quite get how verify stage work yet, by any chance do you mind to check if that's intended behavior to always throw the exception?
Cheers.
from scalatest-maven-plugin.
We stopped using this plugin in our project for a variety of reasons. Our project has been completely rewritten and doesn't even use scala anymore. So I don't have any easy way to contribute anything other than what I already have in comments. Sorry.
from scalatest-maven-plugin.
@axiopisty No worry, thanks for your comments so far, they are useful! :)
from scalatest-maven-plugin.
Related Issues (20)
- Make the plugin toolchains aware HOT 3
- Disabling Failsafe doesn't appear to be necessary HOT 1
- Scalatest Maven Plugin Roadmap?
- Make "Given, when, then" visible
- Adopt the code in Surefire Provider HOT 2
- broken with Scala 2.13 HOT 4
- Mismach of errors between maven and scalatest-maven-plugin, 'There are test failures' VS 'java.lang.ClassNotFoundException' HOT 9
- Build fails on jdk 11
- Integration tests fail due to missing parameter
- How to run or re-run failed tests HOT 1
- How to run single test case from the command line?
- Make the resolution fo java location consistent with maven ecosystem HOT 2
- how to set `<environmentVariables>` HOT 3
- How to include scala test results in maven site? HOT 1
- Specify URL in the pom.xml HOT 1
- Dependency missing for `com/vladsch/flexmark/util/ast/Node` HOT 2
- is version 2.1.0-SNAP2 a snapshot? HOT 5
- Is it possible to set the -P options when running scalatest HOT 3
- Feature request: forkMode always HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from scalatest-maven-plugin.