temyers / cucumber-jvm-parallel-plugin Goto Github PK
View Code? Open in Web Editor NEWMaven plugin to help running Cucumber features in parallel
License: Apache License 2.0
Maven plugin to help running Cucumber features in parallel
License: Apache License 2.0
If there are failures in the integration tests, the TravisCI build should be marked as failed.
Even though I have this:
<includes>
<include>**/*IT.class</include>
</includes>
The Tests are not being found. Can we just use the "Test.java" suffix instead? I can make a pull request for this, but please confirm the work allowed
The plugin doesn't work with the xml format described in the readme, namely:
<plugin>
...
<executions>
...
<configuration>...</configuration>
</executions>
</plugin>
But I find that this works:
<plugin>
...
<configuration>...</configuration>
<executions>...</executions>
</plugin>
I'd like to ask for an enhancement to the plugin. Currently when running tests in parallel on our CI server each generated unit test is being executed regardless of whether it matches filter tags. So, what is happening is as follows:
There is a delay when no matches are found before it exits. Whilst this is not significant for a small number of feature files. The cumulative total of the delay for unnecessarily executing tests is starting to become significant (~12 mins). I suspect the problem lies with cucumber-jvm itself. However, it would be helpful if additional configuration could be added to the plugin whereby some pre-processing of feature files occurs. So, test classes will only be generated for those that match filter tags that have been supplied.
I use testng, having a xml test suite.
the test suite have several feature, each feature have it's test. just like this:
<suite name="regression_suite" verbose="1" parallel="false" thread-count="3"> <test name="regression1"> <parameter name="browser" value="ie"/> <parameter name="browser_version" value="8" /> <classes> <class name="F00001"/> </classes> </test> <test name="regression2"> <parameter name="browser" value="ie"/> <parameter name="browser_version" value="9" /> <classes> <class name="F00002"/> </classes> </test> <test name="regression3"> <parameter name="browser" value="ie"/> <parameter name="browser_version" value="10" /> <classes> <class name="F00003"/> </classes> </test> </suite>
sometimes the cases (F00003) running in the end would fail and the log shows glue code was not found while the cases running in the beginning with the same glue code pass
The cucumber-jvm-parallel-plugin allows arbitrary code injection via tags element of the configuration.
<execution>
<id>generateRunners</id>
<phase>validate</phase>
<goals><goal>generateRunners</goal></goals>
<configuration>
<glue>com.(redact).automation</glue>
<outputDirectory>${project.build.directory}/generated-test-src/cuke</outputDirectory>
<featuresDirectory>src/test/resources/features/</featuresDirectory>
<cucumberOutputDir>target/cucumber-parallel</cucumberOutputDir>
<format>junit,json</format>
<monochrome>true</monochrome>
<tags>**CODE-TO-INJECT**</tags>
<filterFeaturesByTags>true</filterFeaturesByTags>
<namingScheme>simple</namingScheme>
</configuration>
The contents of the tags element is inserted directly into the generated code AS-IS and un-sanitised.
The Arbitrary string could include code. The tags should be sanitised.
@RunWith(Cucumber.class)
@CucumberOptions(strict = true,
features = {"classpath:features/LandingPage.feature"},
format = {"junit:target/cucumber-parallel/1.junit", "json:target/cucumber-parallel/1.json", "pretty"},
monochrome = true,
tags = {**CODE-TO-INJECT**},
glue = { "com.(redact).automation" })
public class Parallel01IT {
}
I'm seeing that the plug-in is always generating 10 runners even though there are only four feature files.
It looks like it is creating one runner per scenario going by the number of duplicate runners (it is creating the class ok).
I have the plug-in set-up with:
parallelScheme = FEATURE
filterFeaturesByTags = true
tags = ~ignore
It doesn't matter if filterFeaturesByTags = false or tags is empty.
I have tried versions 2.0.0 and 2.0.1
Would be cool if there were some way to push the concurrency of the cucumber features even more. And one way to do this is to split up a feature file into smaller parts.
Take the following feature
Feature: Upload photos
As a user,
I should be able to upload my photos
Background:
Given I have an account
And I have a couple of photos on my device
And I have logged into my account
Scenario Outline: As a user I should be able to upload one or more photos
When I select <num> photos for upload
Then I expect <photos> photos to appear as uploaded
And I expect <events> events to appear in the event view
Examples:
| num | photos | events |
| 1 | 1 | 1 |
| 4 | 4 | 1 |
Scenario: As a user I should not be able to upload duplicate photos
When I upload the same photo twice
Then I expect 1 photos to appear as uploaded
And I expect 1 events to appear in the event view
If i'm to run this scenario then this would take quite a while as it really is comprised of 3 test runs.
The way this could be improved was if this plugin could create three feature files from this single file where the feature file "1.feature" would contain:
Feature: Upload photos
As a user,
I should be able to upload my photos
Background:
Given I have an account
And I have a couple of photos on my device
And I have logged into my account
Scenario Outline: As a user I should be able to upload one or more photos
When I select <num> photos for upload
Then I expect <photos> photos to appear as uploaded
And I expect <events> events to appear in the event view
Examples:
| num | photos | events |
| 1 | 1 | 1 |
Then the next "2.feature" file contains
Feature: Upload photos
As a user,
I should be able to upload my photos
Background:
Given I have an account
And I have a couple of photos on my device
And I have logged into my account
Scenario Outline: As a user I should be able to upload one or more photos
When I select <num> photos for upload
Then I expect <photos> photos to appear as uploaded
And I expect <events> events to appear in the event view
Examples:
| num | photos | events |
| 4 | 4 | 1 |
And then lastly "3.feature" contains
Feature: Upload photos
As a user,
I should be able to upload my photos
Background:
Given I have an account
And I have a couple of photos on my device
And I have logged into my account
Scenario: As a user I should not be able to upload duplicate photos
When I upload the same photo twice
Then I expect 1 photos to appear as uploaded
And I expect 1 events to appear in the event view
And then finally we'd use the existing functionality of the plugin to create dedicated test runners for each feature file.
WDYT? Too tall an order? or is it doable? Heck, maybe it would be better to implement this directly into cucumber.
Now it is always using the default character encoding of the system.
i have my feature files inside multiple directory as below /src/test/resources/com/automationrhapsody/cucumber/parallel/tests/wikipedia
in my pom.xml if i specify the entire path it does not work as expected
<configuration>
<glue>com.automationrhapsody.cucumber.parallel.tests.wikipedia</glue>
<featuresDirectory>src/test/resources/com/automationrhapsody/cucumber/parallel/tests/wikipedia</featuresDirectory>
<cucumberOutputDir>target/cucumber-parallel</cucumberOutputDir>
<format>json,html</format>
<tags>"~@ignored"</tags>
</configuration>
Genrerated Runner class is
@RunWith(Cucumber.class)
@CucumberOptions(strict = true,
features = {"classpath:wikipedia/main-page.feature"},
format = {"json:target/cucumber-parallel/2.json", "html:target/cucumber-parallel/2.html", "pretty"},
monochrome = false,
tags = {"~@ignored"},
glue = { "com.automationrhapsody.cucumber.parallel.tests.wikipedia" })
public class Parallel02IT {
}
classpath is in-correct it should have been src/test/resources/com/automationrhapsody/cucumber/parallel/tests/wikipedia/main-page.feature
Hi,
Cucumber Parallel plugin causing compilation issues on windows while execution.
Parallel java files created during execution are showing path in wrong format. forward slash '/' should be used in specifying the path instead of the backward slash for the format attribute.
format = {"json:D:\MNSProject\test\shopping-test-framework\shopping-tests-api-ls/target/cucumber-reports/1.json", "pretty"},
This throws illegal escape character errors.
Hello,
Version 1.2.1. I got errors in generated Java code, when generating names using option namingScheme='feature-title' features, for files like "some.test.feature" The '.' is not a valid character in a Java class name.
Here is what I would like to achieve:
How do I generate runners with no tags?
I have found that if I set the following config:
<configuration>
<tags></tags>
or
<configuration>
<tags> </tags>
Then cucumber-jvm-parallel-plugin
generates runners like:
@CucumberOptions(
tags = {"@complete", "@accepted"},
However, if I use the following command-line property, it does seem to work:
mvn -Dcucumber.tags= test-compile
I am using ver 1.2.1
Given the configuration:
<plugin>
<groupId>com.github.temyers</groupId>
<artifactId>cucumber-jvm-parallel-plugin</artifactId>
<version>1.2.1</version>
<executions>
<execution>
<id>generateAndroidRunners</id>
<phase>validate</phase>
<goals>
<goal>generateRunners</goal>
</goals>
<configuration>
<configuration>
...
<tags>"@android"</tags>
<namingScheme>simple</namingScheme>
</configuration>
</configuration>
</execution>
<execution>
<id>generateIOSRunners</id>
<phase>validate</phase>
<goals>
<goal>generateRunners</goal>
</goals>
<configuration>
...
<tags>"@ios"</tags>
<namingScheme>simple</namingScheme>
</configuration>
</execution>
</executions>
</plugin>
If i have a single feature in the project this configuration would only create Parallel01IT.java, where i would expect this to create two files, one with the IOS configuration one with the Android configuration.
However the workaround for now is to have the naming scheme on one of these executions set to feature-title.
I guess something like #19 would be a solution for this, or a naming scheme which is a bit more intelligent.
If you use custom report formatter like
<configuration>
...
<format>com.some.Formatter</format>
...
</configuration>
It causes generated classes look like this:
@RunWith(Cucumber.class)
@CucumberOptions(
plugin = {"com.some.Formatter:target/cucumber-parallel/1.com.some.Formatter"},
public class Parallel01IT {
}
on linux, Parallel01IT is created for feature2.feature
on Windows, Parallel01IT is created for feature1.feature
Hi,
I've started to work on a small enhancement that would use the feature file names to generate classes with that name (camelCased), as the current behavior generates 1 class per feature file, but with a generic name.
This is the main change :
vincent-fuchs@c3590d3
The problem is that currently it breaks most of the existing tests, since the output file is different.
Do you think this could be useful ? maybe we can add a flag in the config to enable that feature (disabled by default), so that it doesn't impact existing users...
If you're ready to take that feature, I'll fix the tests in my PR, then submit.
Hey @temyers ,
Are there any possibilities to generate runners per tags in feature?
That will be another workaround of splitting scenarios on parts.
Like:
Scenario Outline: func test
Given some <data>
And test func
@theTagPart2
Examples:
|data|
|any|
@theTagPart1
Examples:
|data|
|some|
As result we will have 2 runners with 2 different tags.
Currently there is no way to set java package for runners.
Hi! I've experienced issue with screenshots display in the Cucumber-JVM-Reports:
if there are 2 failed tests in a suite, 1 screenshot for each test and screenshots look the same visually and a size is the same, then link to the 2nd screenshot doesn't work: link to the 2nd screenshot expands/collapses the 1st screenshot.
Is it a feature or a bug?
I have a Cucumber/Selenium framework and have each test designed to extend a class in order to initialize the web driver upon suite execution and then subsequently destroy the web driver after suite execution.
Is there any way to get the generated runner classes to extend a class?
Any way to inject code into the body of the generated runner class, in order to add @BeforeClass and @afterclass hooks?
I understand that my overall design may need to change in order to support parallelization with Selenium,
but was hoping there'd be a way to manipulate the generated runner class.
Thank you.
When run on Windows, feature locations include '' characters, rather than '/'
In order to support contributions,
As the project owner
I want a consistent codeing style to make merge/review easier.
You should update the readme to say 1.1.2
HI,
for browser compatibility testing using selenium grid i need to pass different parameter for each generate test.
Would you let me know how can i achieve this ?
Hi, Is there way to generate single rerun.txt file after the parallel tests are completed so we can just rerun all failed tests as post single thread instead rerun it in parallel mode. For example. currently when we run parallel tests using this plugin, we don't have the capability to add "rerun:rerun.txt" plugin to all the runners. I know that this pull request is open related to rerun functionality. Please kindly let us know if this pull request will fix our issue. Btw thanks for building this plugin, It's been helping us everyday with our test execution.
Thanks
Mohammad
Hi,
I setup the plug-in as described @ https://github.com/temyers/cucumber-jvm-parallel-plugin.
However, I am facing following error, when trying maven build. The issue is same with v1.2.0, 1.2.1 and 1.3.0 as well.
Failed to execute goal com.github.temyers:cucumber-jvm-parallel-plugin:1.3.0:generateRunners (generateRunners) on project colspace: Execution generateRunners of goal com.github.temyers:cucumber-jvm-parallel-plugin:1.3.0:generateRunners failed: A required class was missing while executing com.github.temyers:cucumber-jvm-parallel-plugin:1.3.0:generateRunners: org/apache/velocity/context/Context
Please help!!
Hi all,
Basically problem is described in title:
As far as I know there are 2 ways to add listeners to TestNG tests: 1. Using annotations in runner (which is not an option cuz we're not able to customize runners)
2. Using testng.xml - and in this case we're required to provide package or location of test classes using package hierarchy.
As I understand there is no way to add listeners when running testNG?
@temyers is there a way we can re-run failed scenarios?
At the moment, users can choose between junit and testng formats, which changes the runner-template.vm file used to generate the java.
We have the requirement for our runner file to have some additional annotations (eg the test classes extend from a BaseTest class), and so I've forked this project and implemented a custom template.
(fork is here: https://github.com/Frameworkium/cucumber-jvm-parallel-plugin )
I've tried to create a pull request that allows users to specify a parameter in their pom - alongside the standard parameters for the plugin - to a vm file in their project. For example, it might be in their /resources/ folder: /resources/custom-runner.vm
However, this doesn't work, as Velocity is unable to find the file on the class path (as presumably it's still looking in it's /resources/ folder?)
What do I need to add to make this work, or could a new feature be implemented to enable this?
The configuration option of tags complicates custom run configurations in maven.
so say i have the property "cucumber.tags", which is injected into the tags configuration of the plugin and i'd like to run my project with mvn -Dcucumber.tags="@completed", "~@ignored"
then this doesn't work properly.
I've also tried to use singular quotes around the whole expression and escaping the double quotes. The result always seems to be that the last double quote is consumed.
Ideally i'd like to be able to run with mvn -Dcucumber.tags=@completed,~@ignored
where the plugin then correctly transforms this into "@completed","~@ignored" in when creating the runners.
I want to use before and after class annotations of junit to execute clean-up actions such as close browser after test. However, the parallel class created by this plugin does not create before and after class in cucumberrunnertest file. I am able to execute clean-up as another cucumber step at the end of my tests but it will be skipped in case of a test failure. So is there a way I can always execute test clean-up regardless of test pass or failure with this plugin?
At the moment the json report outputs to cucumber-parallel. This should be made configurable
I get the following error when attempting to use 1.2.0
[ERROR] Failed to execute goal com.github.temyers:cucumber-jvm-parallel-plugin:1.2.0:generateRunners (generateRunners) on project test-project: Execution generateRunners of goal com.github.temyers:cucumber-jvm-parallel-plugin:1.2.0:generateRunners failed: A required class was missing while executing com.github.temyers:cucumber-jvm-parallel-plugin:1.2.0:generateRunners: com/google/common/base/CaseFormat
[ERROR] -----------------------------------------------------
[ERROR] realm = plugin>com.github.temyers:cucumber-jvm-parallel-plugin:1.2.0
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] = file:/C:/Users/kthomson/.m2/repository/com/github/temyers/cucumber-jvm-parallel-plugin/1.2.0/cucumber-jvm-parallel-plugin-1.2.0.jar
[ERROR] urls[1] = file:/C:/Users/kthomson/.m2/repository/commons-io/commons-io/2.4/commons-io-2.4.jar
[ERROR] urls[2] = file:/C:/Users/kthomson/.m2/repository/org/apache/velocity/velocity/1.7/velocity-1.7.jar
[ERROR] urls[3] = file:/C:/Users/kthomson/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar
[ERROR] urls[4] = file:/C:/Users/kthomson/.m2/repository/commons-lang/commons-lang/2.4/commons-lang-2.4.jar
[ERROR] urls[5] = file:/C:/Users/kthomson/.m2/repository/org/codehaus/plexus/plexus-utils/1.1/plexus-utils-1.1.jar
[ERROR] Number of foreign imports: 1
[ERROR] import: Entry[import from realm ClassRealm[maven.api, parent: null]]
[ERROR]
[ERROR] -----------------------------------------------------: com.google.common.base.CaseFormat
There should be a fature to set a custom name on the json report at the CucumberOptions plugin. Otherwise the reports are named with the fork number and it's not easy to identify the remote device where the test was actually executed.
Would it be possible to rename it with the device platform and browser getting them thorugh system or env properties?
I'm happy to help on this if someone provides some guidance
In a large project, if you only want to run small number of features with a specific tag, classes are still generated for all feature files.
When run in parallel, each Test case has a setup/tear down time which can lead to significant overhead.
Add support for filtering the generated output files to only those features that contain the tags to run.
Hallo,
I think the plugin has a little bug with a great impact on me. All the feature files have to be placed under "/src/test/resources/features/". But I have another folder structure like "/src/test/resources/com/company/test/testcases/". In this case no tests will be found with the given tag and so no tests will run in parallel. So, I have to copy all my existing feature files to "/src/test/resources/features/" to run the tests in parallel. But this is not possible because other code is searching for files in my existing folder structure.
It will be better that all feature files under "/src/test/resources/" will be runnable with this plugin.
THX Martin
Added following comment from @bhattankit in issue#15
This could be a good addition. Can we also add functionality to add suffix to the generated class. The reason for this is that we have a couple maven profile that isolates the kind of tests to run.
hi,
i'm in a situation where i have static webdriver because of specific design (not using cucumber dependency injection). i'm experiencing problems with parallel execution (still under investigation) and my first worry is about the static web driver.
i know that using the cucumber-jvm-parallel-plugin , it uses surefire forks(JVMs) instead of threads which means separate JVM for each feature file BUT it is configured to reuse the forks which may prove problematic with static variables. is it true that its not possible to set reuseForks=false ? (will be ignored by the plugin?)
not reusing the JVM forks will ensure full separation in parallel execution in the current design I have.
Thanks
@temyers
in the pom.xml i have not added
<executions>
<execution>
<id>generateRunners</id>
<phase>validate</phase>
<goals>
<goal>generateRunners</goal>
</goals>
<configuration>
<outputDirectory>src/test/java/output</outputDirectory>
<glue>com.test.cucumber.stepdefinitions</glue>
<featuresDirectory>src/test/resources/features/</featuresDirectory>
<cucumberOutputDir>target</cucumberOutputDir>
<format>json,html</format>
</configuration>
</execution>
</executions>
when i run mvn clean validate, i get @completed and @Accepted tag in the Parallel01IT class
@RunWith(Cucumber.class)
@CucumberOptions(strict = true,
features = {"classpath:features/Sample.feature"},
format = {"json:target/1.json", "html:target/1.html", "pretty"},
monochrome = false,
tags = {"@complete", "@accepted"},
plugin = {"com.cucumber.listener.ExtentCucumberFormatter:"},
glue = { "com.test.cucumber.stepdefinitions" })
public class Parallel01IT {
}
Can you please help me on this?
I would like to run the 1.3.0-SNAPSHOT version of the plugin, do you have a repository for development builds?
I tried using jitpack for this, however this failed:
Our project has many features directories, and in addition there are some nested features directories, so under /features/ there are the names of all of the subdirectories and within them there are feature files as well as directories containing feature files. Is making cucumber-jvm-parallel-plugin more flexible in this direction something that you would consider taking on, or should we just put together an implementation and submit a pull request?
When I using the command 'mvn generate-test-sources' everything is working fine, but when I use the command 'mvn test' the generate classes creates duplicates.
Hey @temyers,
I tried new code you merged, Its good in compact have great functionality, however there is issue while generating jUnit Runner class.
<tags>"@google"</tags>
<parallelScheme>SCENARIO</parallelScheme>
Expected :
In feature property if feature file path followed by line number as follow
features = {"D:/Test/resources/features/Test.feature:**40**"}
runner class should not include
tags = {"@google"},
pom.xml
<parallelScheme>FEATURE</parallelScheme>
<tags>"@google"</tags>
<filterFeaturesByTags>true</filterFeaturesByTags>
There is only one feature file has tag @google. but runner classes are generated for non matching or no tags on features file. which is unexpected.
Feature with only matching tag should get filtered.
** Edited
sorry it was mistake from my side. Configuration property
<filterFeaturesByTags>true</filterFeaturesByTags> must be true with
<parallelScheme>FEATURE</parallelScheme>
<tags>"@google"</tags>
Still if we use following combination runner should not generate runner for all other non matching or empty Feature tag match.
<filterFeaturesByTags>false</filterFeaturesByTags>
<parallelScheme>FEATURE</parallelScheme>
<tags>"@google"</tags>
I have my feature files in src/it/resources
, as per the Maven standard layout
If I configure cucumber-jvm-parallel-plugin
like:
<configuration>
<!-- The diectory, which must be in the root of the runtime classpath, containing your feature files. -->
<featuresDirectory>src/it/resources</featuresDirectory>
... then it finds all my feature files, but generates invalid runner files which look like:
@RunWith(Cucumber.class)
@CucumberOptions(strict = true,
features = {"classpath:resources/uk/co/me/my-feature.feature"},
format = {"json:target/cucumber-parallel/3.json", "pretty"},
monochrome = true,
tags = {},
glue = { "uk.co.me" })
public class Myfeature03IT {
Which fails at runtime because the feature path should be "classpath:uk/co/me/my-feature.feature
", not "classpath:resources/uk/co/me/my-feature.feature
"
Do you know how I can fix this?
TIA
In the Parallel01IT.java i wouls want to include plugin ={"com.cucumber.listener.ExtentCucumberFormatter:"}
Ideally the Runner would look like
@RunWith(Cucumber.class)
@CucumberOptions(strict = true,
features = {"classpath:features/MyFeature.feature"},
format = {"json:target//1.json", "html:target//1.html", "pretty"},
monochrome = false,
tags = {"@accepted"},
plugin = {"com.cucumber.listener.ExtentCucumberFormatter:"},
glue = { "com.cucumber.stepdefinitions" })
public class Parallel01IT {
}
how can i add this ?
I am not sure if this is a bug or a setting I am missing but my browsers remain open until all runners complete their tests. I end up having 20+ open browsers. Is there any setting to setup the browser to close after each runner complete the task?
This was especially visible when I used <parallelScheme>SCENARIO</parallelScheme>
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.