zebrunner / java-agent-testng

Zebrunner Agent: Java TestNG reporting integration

For details check our official documentation: https://zebrunner.com/documentation/reporting/carina-testng

java-agent-testng's Issues

Test sessions are not tracked in Zebrunner properly for failed tests with not unique method name and parameters defined on suite level

On one of the projects we have test method defined in next way:

@Parameters({"TUID", "userType", "startPage"})
    @Test()
    public void verifyItemSelection(String TUID, String userType, String startPage) {
        TestRail.setTestCaseId(TUID);
        ......

Then on suite level we have multiple test items with different parameters, e.g.:

<test name="name1" enabled="true">
        <parameter name="TUID" value="3709692,3705554"/>
        <parameter name="userType" value="type1"/>
        <parameter name="filterOption" value="filter1"/>
        <parameter name="startPage" value="page1"/>
        <classes>
            <class name="classname1">
                <methods>
                    <include name="verifyAppFilterResultsTest"/>
                </methods>
            </class>
        </classes>
    </test>

    <test name="name2" enabled="true">
        <parameter name="TUID" value="3709696,3705558"/>
        <parameter name="userType" value="type2"/>
        <parameter name="filterOption" value="filter2"/>
        <parameter name="startPage" value="page2"/>
        <classes>
            <class name="classname2">
                <methods>
                    <include name="verifyAppFilterResultsTest"/>
                </methods>
            </class>
        </classes>
    </test>

As a result in Zebrunner report for failed tests which are organised this way we have not realistic amount of tracked sessions. because test results page also includes sessions and logs related to other test items (see screenshot)

ConcurrentModificationException appears during test start registering

Introduce ability to skip retry logic for failing tests which have known issues in association

Scenario is next:

• we have a test suite with number of failures which are mapped to tickets (bugs) in Jira
• those failures are already mapped in test runs in zebrunner
• during the next run such tests fail because of bug reason (with error stacktrace already saved in zebrunner)
• right now such tests are restarted again in case if retry_count parameter > 0

The idea is to disable retrying logic for tests like this as it doesn't make any sense.
So on test failure agent should check whether such error with same stacktrace already linked to a bug and skip retrying if condition has been met.

Benefits:

• saving of overall execution time especially if amount of failures linked to bugs is sufficient and count of retry attempts in configuration is high.

Environment variables and program arguments configurations for TestNG agent

I've read the document about Zebrunner TestNG agent but I don't know the way to configure environment variables and program arguments. Is there any detail document about them?

BeforeClass has no access to the test method, it leads to NullPointerException on rerun

Launch time execution is displayed incorrectly while running several XML suites at once

Launch time execution in Zebrunner is displayed incorrectly while running several XML suites at once

Steps to reproduce:

1. There are several XML TestNG suites to execute
2. Run them together using the command:

mvn clean test -DsuiteXmlFile=suite1.xml,suite2.xml,suite3.xml,suite4.xml

3. When launch is finished, check its time execution

Actual Result: Tests were executing about 30 minutes, but launch time execution in Zebrunner is displayed 1 hour 21 minutes (elapsed value returned by BE). If sum up time execution of all tests - the result is correct (30 minutes).
The issue happens due to sending finish launch request 4 times by TestNG agent (number of requests equals to number of executed suites).
If execute only one suite, the issue is not reproducible.

TestNG version: 7.6.1
Zebrunner TestNG agent version: 1.9.5

Video recording function when running with agent testNG

Could I use video recording function? I've read the doc but only screenshot available.