ybonjour / test-store

Stores your test results

test-store

Stores your test results

Run test-store server

1. Install docker and docker-compose
2. Download docker-compose-prod.yml
3. Run docker-compose -f /path/to/docker-compose-prod.yml up -d
4. Access test-store on http://localhost or http://localhost:8080 if you want to use the new UI.

Store test results from your gradle build

Adapt your build.gradle according to the following template

buildscript {
    repositories {
        maven {
            url 'https://plugins.gradle.org/m2/'
        }
    }
    dependencies {
        classpath 'gradle.plugin.ch.yvu.teststore:gradle-plugin:0.12'
    }
}

apply plugin: 'ch.yvu.teststore'

teststore {
    host '<your test store server>'
    port  8080
    testSuite '<your test suite id>'
    revision System.properties['revision']
    xmlReports ~/.*\/test-results\/TEST-.*\.xml/
    changesFile 'git.log'
    changesUrlTemplate 'https://github.com/ybonjour/test-store/commit/${revision}'
    incremental false
}

Store results at the end of a run

Run ./gradlew storeResults to store your test results once you have executed your tests. It will collect all test results (in JUnit XML format) according to the pattern specified in teststore.xmlReports in your build.gradle file and send them to the test-store server.

Store results as soon as they are available

If you want to store the result of a test as soon as it finishes, you can set incremental to true. In this case you don't need to use the storeResults task.

test-store's Issues

Use dockerize

Use dockerize (https://github.com/jwilder/dockerize) to

• create the nginx test-store.conf using an environment variable to externalize the backend host
• replace the wait-for-*.sh scripts

gradle-plugin: Replace left shift with doLast

The left shift operator (<<) is deprecated in gradle 3. It should be replaced with doLast. This needs to be adapted in the gradle plugin

Make tests selectable

When I have a failed test, I'd like to copy its name to my IDE in order to find and analyse it locally. Currently, this is difficult since the names are Hyperlinks.

Result list: Show time when run was executed

In the results lists I want to see the date and time when the run was executed.

Run list: Optimize loading time

I want to see the latest 10 runs in less than 1 second over 3g in the run list.

Currently the latest 200 runs are loaded in one request from the server.
This turns into loading time of > 5 seconds when on a slow connection (e.g. 3g).
There shall still only be 200 runs shown on the page, but they shall be loaded in multiple requests.

Test data statistics - Ideas

This ticket serves as a collection of ideas for further statistical analysis of the test data:

Result List: Group by class name

I want to be able to easier see if multiple tests of the same class have failed.

The pattern for test names in test store is <ClassName>#<testName>.
The test results with the same result (passed/failed/retried) shall be grouped by class name.
Example:
ch.yvu.teststore.MyTest1
- test1: Failed
- test2: Failed
ch.yvu.teststore.MyTest2
- test1: Failed

ch.yvu.teststoreMyTest1
- test13 Passed

Classpath conflict when using with org.asciidoctor:asciidoctorj-pdf

Scenario:

buildscript {
    dependencies {
        classpath 'gradle.plugin.ch.yvu.teststore:gradle-plugin:0.14'
        classpath 'org.asciidoctor:asciidoctorj-pdf:1.5.0-alpha.16'
    }
}

plugins {
    id 'org.asciidoctor.convert' version '1.5.6'
}

apply plugin: 'org.asciidoctor.convert'

asciidoctorj {
    version = 'v1.6.0-alpha.5'
}

asciidoctor {
    backends 'pdf'
    attributes   \
          'imagesdir': './images'
}

Command:

./gradlew asciidoctor

Result:

FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':asciidoctor'.
> org.yaml.snakeyaml.events.DocumentStartEvent.getVersion()Lorg/yaml/snakeyaml/DumperOptions$Version;

I guess, the classpath collision is caused by the org.codehaus.groovy.modules.http-builder.http-builder dependency used in this plugin.

Suggestion: use shading OR explicit dependencies (no re-packaging)

Improve performance of test statistics page

The test statistics page (that shows all the tests of a test suite) should show the first tests in < 1 sec (over the wired internal network).
To achieve this the tests shall be loaded in chunks (pages).

Deeplink to Result List of latest run

I want to directly access the Result list of the latest run:
a) thorugh a deep link and
b) from the sidebar

Result List: Optimize loading time

I want to be able to see failed tests in the result list faster in the Result List.

Currently all the tests are loaded in one request from the server.
The tests should be loaded incrementally with the failed tests first.

Result List: Order by result time

I want to sort the results in the result list by execution time, because I want to see in which order the tests have been executed. If the tests are poorly isolated, this is needed to debug test failures.

Allow Ruby scripts as migrations

Currently for migrations there are only cql scripts allowed. This does sometimes not provide enough flexibility. It should be possible to also execute ruby scripts as part of the migration.