dakusui / jcunit

A Combinatorial Testing + Model-based Testing framework on top of Java+JUnit. Model your application as code and let JCUnit do the rest.

Design Your Test as Code: The JCUnit 0.10.x

is an open source model-based testing (MBT) framework powered by combinatorial interaction testing (CIT) technique. It supports various models to provide a uniformed testing style. It employs CIT to balance the coverage and test suite size. The documentation is found here.

Maven Coordinate for JCUnit

<dependency>
    <groupId>com.github.dakusui</groupId>
    <artifactId>jcunit</artifactId>
    <version>{jcunit-version}</version>
</dependency>

JCUnit 0.10.x is under development. The most recent stable major version of JCUnit is 0.8.x. Please check

• MVNrepository

• 0.8.x-develop branch

JCUnit 0.10.x: Key Features to Come

0.10.x will be the last version line previous to 1.0.x, which is intended for the "true" public usage and broader audience.

The project board is found here.

• Enhanced Regex Support: (#152, #155)

• JUnit5: support (#142)

• Temporary removal of FSM support: (It will be brought back in 1.0.x, #169).

• PICT integration: (#179)

• Simplified Internal Structure for integrating external CIT tools: (Stretch goal, #175).

• Fault Localizer: (Stretch goal, #144)

Limitations

• Issue-185: JCUnit test fails with 4.13.1: This is coming from a design change introduced in JUnit’s side. A workaround is described in the ticket.

• Issue-125: Provide a way to execute a test method for a test case or all the generated test cases: If you have multiple test methods in your test class from an IDE such as IntelliJ, you cannot run only one of them. JCUnit has a helper class to work around the situation.

References

PICT

PICT is the most popular tool for Combinatorial Interaction Testing (CIT). It can generate a covering array from a text file described in its own human-readable notation. It was developed by Jacek Czerwonka and now an open source project from Microsoft.

JCUnit 0.10.x (or later) uses PICT for covering array generation for the part (factors) it can be used and then combine it with another covering array generated by JCUnit’s internal covering array generator for the rest (#179).

• "Pairwise Testing in Real World", J Czerwonka, 24th Pacific Northwest Software Quality Conference, 2006

• PICT github project, Microsoft

JCUnit

We have published some key ideas of JCUnit as academic papers and a blog.

Publications

• Test Design as Code: JCUnit; A paper on this product; Presented at IEEE International Conference on Software Testing 2017

• Reduce Test Cost by Reusing Test Oracles through Combinatorial Join; A paper about a technique this product utilizes for building a covering array; Presented at 2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)

• Accelerating covering array generation by combinatorial join for industry scale software testing; A paper about a technique to construct a covering array from existing ones;

• JCUnit blog; Japanese; 日本語

JCUnit Project

• Quick Start Guide for JCUnit (t.b.d.)

• JCUnit 0.10.x Project Board

Build Instructions

The JCUnit requires Java SE 8 to build. Please don’t forget installing and using it for building. This is intentional decision to make the testing compatible for as many product as possible.

• Java Platform, Standard Edition (Java SE) 8

• AdoptOpenJDK

Build

$ mvn clean compile

Conduct Unit Test

$ mvn clean compile test

Generating Site

$ mvn clean package site

Generates the site under target/site in the following structure.

target/
  site/
    apidocs/     - JavaDoc
    jacoco/      - JaCoCo report (code coverage)
    pit-reports/ - PIT report (mutation coverage)

The mutation testing is configured to cover only small amount of classes to save build time. To configure the coverage, refer to [mutationTesting].

Deploying Site

$ mvn clean package site

Misc

Conduct Mutation Test

$ mvn clean compile test org.pitest:pitest-maven:mutationCoverage

Currently, this may take very long time (≥ 2 hours), if you include all the classes of this product. By default, the pom.xml is configured to exercise the mutation testing over small number of classes so that you customize the intended coverage.

Under the plugin configuration for org.pitest:pitest-maven, you can find a following element.

pom.xml configuration for org.pitest:pitest-maven plugin

    <targetClasses>
        <param>com.github.dakusui.jcunit8.exceptions.*</param>
    </targetClasses>

To cover all the JCUnit classes, you can modify it like following.

pom.xml configuration for org.pitest:pitest-maven plugin

    <targetClasses>
        <param>com.github.dakusui.*</param>
    </targetClasses>

After a successful execution, it generates a pitest report under a directory target/pit-reports/{yyyyMMddHHmmss}.

Build Javadoc

$ mvn clean javadoc:javadoc

This generates JavaDoc under target/site/apidocs.

Build the Site

$ mvn clean package site

Publish the Site

$ mvn clean package site site-deploy

Copyright and license

Copyright 2013 Hiroshi Ukai.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this work except in compliance with the License. You may obtain a copy of the License in the LICENSE file, or at:

• http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

jcunit's Issues

Refactor covering array generation mechanism

Following is a diagram that illustrates intended refactoring.

Enhance JCUnit API to support FSMs

JCUnit already has API which enables users to generate test suites programmatically, but it doesn't support some important features like FSM etc.

Let's support it.

Design page is found here

Support Nested FSM feature

FSM is a so powerful idea that you can model broad range of software systems.
If we can mode a software product using it, and let JCUnit generate test suite and execute it over the SUT, wouldn't it sound nice?

The enhancement is on-going on this branch: https://github.com/dakusui/jcunit/tree/nestedfsm

Small refactorings, test enhancements, or improvements

A place holder ticket for small refactorings, test enhancements, or improvements.

Separate annotation system from the other parts of JCUnit

Separate annotation system from the other parts of JCUnit and allow users to

• reuse test suite generator
• create their own annotation systems.

(FSM)Multi-threading support

FSM/JCUnit treats each state in each FSM as an independent factor. Therefore it guarantees a fair coverage for multi-threading situation where

Step. 1, client 1 requests op.A the server and at the same time client 2 requests Op.B,
Step. 2, ...

with relatively small number of test cases since you can use pairwise/t-wise technique as its test suite generation method.

AETG algorithm support

IPO algorithm isn't good at handling constraints. Probably AETG can handle it better since it generates a complete tuple every time.

This feature is not only useful for FSM support but also useful for regular test suite generations.

Create a simple builder API

Create a simple builder API

A logic to construct a scenario sequence for test preparation sometimes creates unintentional one

A logic to construct a scenario sequence for test preparation sometimes creates unintentional one

Consider using Guice for assigning fields' values

Since guice provides pretty decent API for assigning values to fields based on type information.
Potentially it's worth considering use it instead of the home-brew mechanism which is being used currently.

(FSM)Ensure FSMUtils.resetStories is called before each test method is called every time

(FSM)Ensure FSMUtils.resetStories is called before each test method is called every time to reset story's state (isPerformed)

Potentially, nested stories will not be executed if there are more than one method in a test object.

Make generated test suite reusable (experimental feature)

As discussed in this blog, it would be useful to make generated test suite reusable.

As of now, JCUnit has "Recorder" and "Replayer" mechanism , but

• This is a feature to reuse output of SUT as test expectation. Not generated test suite.
• The documentation about it is not sufficient.
• It requires code change on record/replay.
• How it should behave when the test suite definition (what factors the test class has, what levels each factor has, etc.) is changed isn't defined.
• How to detect those changes (or a user should be responsible for it) is not defined.

At least JCUnit needs documented best practice for this.

(FSM)Simplify test suite generation

right now (Sep/2015), FSM/JCUnit chooses states not only events as factors. But, except for the first one, states should always be determined by the previous state and an input symbol (event) given to the FSM. As discussed in [Introduction to Combinatorial Testing]4, in this situation we can exclude them during test suite generation from factors and after the process finishes we can append states determined by previous states and input to each test case. By this optimization, we should be able to improve FSM/JCUnit's performance.

Add tests for "local constraint" feature.

Add tests for "local constraint" feature. Current existing tests do not have sufficient coverage.

Support constraint handling for AETG

Support constraint handling for AETG

Refine constraint checking mechanism

Refine constraint checking mechanism.
Right now ConstraintChecker desinged to check all constraints for a factor space at once.
But test suite generator implementations (CoveringArrayEngine) normally need to access each of them individually to generate a good test suite.
Also such implementations need to know which factors each constraint relies on.

Following is a diagram that illustrates current class structure of JCUnit's constraint handling mechanism.

ASIS

    +-------------------+           +-----------------+
    |CoveringArrayEngine|---------->|<<interface>>    |
    +-------------------+ 1       1 |ConstraintChecker|
                                    +-----------------+
                                    |check(TestCase t)|
                                    +-----------------+
                                             |
                                             A
                                             |
                                    +----------------------+        +-----------------+
                                    |SmartConstraintChecker|<>----->|Constraint       |
                                    +----------------------+  1   * +-----------------+
                                                                    |check(TestCase t)|
                                                                    +-----------------+

As seen, CoveringArrayEngines only knows top-level interface of constraint checker. If an unassigned value is used by any of constraints, JCUnit's IPO CA generator traverses all the possible test cases recursively assuming there is a level(s) for the factor that makes the on-going test case valid. If it can't find any, the test case will be removed.
This is too funky and inefficient.

Instead, we should have a structure like following, where covering array engines can access all the necessary information to figure out valid test cases under a set of constraints.

TOBE

    +-------------------+           +-----------------+        +-----------------+
    |CoveringArrayEngine|---------->|<<interface>>    |<>----->|Constraint       |
    +-------------------+ 1       1 |ConstraintChecker|  1   * +-----------------+
                                    +-----------------+        |check(TestCase t)|
                                    |check(TestCase t)|        |getFactors(): F[]| returns involved factors
                                    +-----------------+        +-----------------+
                                             |
                                             A
                                             |
                                 +----------------------+
                                 |SmartConstraintChecker|
                                 +----------------------+

Design page is found here

Keep document updated

(A place holder ticket for document updated)

Make it possible to specify factors used by a constraint

Make it possible to specify factors used by a constraint.

@Uses({"a", "b"})
@Condition(constraint = true)
public boolean isConstraintSatisfied() {
  return   this.a == ... &&  this.b == ...
}

If a or b is not yet set by JCUnit, the framework will not call the method.

Support Theories runner

Support Theories runner of JUnit.

• About Theories runner: http://junit.org/apidocs/org/junit/experimental/theories/Theories.html

This has already been done in following commits.

• 4a8ad05
• e6b2a34
• 2255f89

Can I Use Without JUnit?

Would it be possible for me to use this tool to generate the combination table but not use JUnit to execute? I read the section about generating the test. Will that work without JUnit present?

Add BDD support

I would like to be able to write a test case like below,

public class ManageLorries {
  @Given("I have gone to the new lorry page")
  @When({"I fill in 'Name' with 'name 1'", "I press 'Create'"})
  @Then("I should see 'name 1 - this is from before filter'")
  public void registerNewLorry() {
    ...
  }
}

Expose default values of @FactorField

Expose default values of @FactorField somehow.

(FSM)Local constraints

probably we want to define constraints applied to parameters of a method.

Levels provider instantiation should comply with regular plugin handling mechanism

Currently Levels providers are directly instantiated by JCUnit. But they should be created by "regular" way where users can implement their own levels providers easily (at least in comparison).

Enhance unit tests

Enhance unit tests toward 0.6.0 release.

Use variable names "degree" and "strength" consistently (Refactoring)

Currently I have been using variable names "degree" or "strength" both for how many factors whose all possible combinations of levels to be covered covered in a test suite, but there is no policy.
Maybe I should always use "strength" since I haven't seen the usage of "degree" as much as the other's.

Reduce number of test executions where possible

It is a possible situation where a test method uses some of factors defined in a test class.
In such a case, if the levels of the factors are exactly the same, the test shouldn't be executed repeatedly.

One idea is to allow users to declare factors used in a test method by using a new annotation '@uses'

@Test
@Uses("factor1", "factor3)
public void test() {
    use this.factor1 and 2
}

Clean up code

Clean up code thoroughly toward the stable release 0.6.x.
It would be the last opportunity to do so wildly.

Use validation framework of JUnit more extensively

Right now JCUnit is utilizing JUnit 4.12's validation framework only partially.
It should be used more extensively and consistently.

Organize documentation

JCUnit is becoming bigger and its documentation is so, too.
Before releasing 0.6.0, documentations should be organized better.

@ActionSpec should only need to be attached to top level.

@actionspec should only need to be attached to top level but this this property is not tested.
And FSM related tests are giving it to all the overriding methods.

public enum IncorrectSpec implements FSMSpec<ModifiedMealyMachine> {
  @StateSpec I {
  },
  @StateSpec S {
    @ActionSpec // <-- This is not necessary
    public Expectation<ModifiedMealyMachine> getS(Expectation.Builder<ModifiedMealyMachine> b) {
        ....
    }

    @ActionSpec // <-- This is not necessary
    public Expectation<ModifiedMealyMachine> getI(Expectation.Builder<ModifiedMealyMachine> b) {
      return b.valid(S).build();
    }
  };

...

  @ActionSpec(alias="method")
  public Expectation<ModifiedMealyMachine> method(Expectation.Builder<ModifiedMealyMachine> b, String s) {
    ...
  }

The location to store previous run can't be changed

The location to store previous run is set to a path 'new File(".jcunit").getAbsolutePath()'.
But users need to change it in case they want to include the test results as a part of test definitions and submit it to source code repository for instance.

Simplify annotation system

As of now, JCUnit's annotation system became more complicated than it should be.
Simplify this.

In the following example, users should not need to write @GenerateCoveringArrayWith annotation every time because the setting is pretty typical. Let's make them (values set to checker and reporters) default of those attributes.

@RunWith(JCUnit.class)
@GenerateCoveringArrayWith(
    checker = @Checker(value = SmartConstraintChecker.class),
    reporters = {
        @Reporter(value = CombinatorialMetrics.class, args = { @Value("2") })
    })
public class QuadraticEquationSolverTest8 {
  @FactorField
  public int a;
  @FactorField
  public int b;
  @FactorField
  public int c;

  @Uses({ "a" })
  @Condition(constraint = true)
  public boolean aIsNonZero() {
    return this.a != 0;
  }

  ...

  @Test(expected = IllegalArgumentException.class)
  @When({ "!aIsNonZero" })
  public void solveEquation1$thenThrowIllegalArgumentException() {
    new QuadraticEquationSolver(
        a,
        b,
        c).solve();
  }

  ...

Design page is found here

Refine exception wrapping mechanism

Right now Checks.rethrow wraps and rethrow checked exceptions from lower levels.
Instead we shoud introduce Checks.wrap checked exception and 'return' wrapping runtime exception and callers should be responsible for throwing returned runtime exception.
By this we will not need to suppress around warnings caused by the rethrow method (because it makes java compiler not understand the method is aborted at the points where the method is called)

Output formatting part should be separated from core logic

Currently output formatting logic is embedded into core testing logic and not pluggable.
This should be separated from the core logic and pluggable.

Make it possible to customize how objects are compared in tests

Users want to customize how previous objects and current objects are compared in automatic regression tests, where the outputs of the last run are recorded and used for comparisons from the next run on.
Using 'equals' method is not enough since users might want to test objects that do not override equals method.

(FSM)Support static method

Support static method.
Probably just doing a static method call if a Class object is passed as SUT would be sufficient.
(java.lang.Class is marked final.)

Implement "smart" negative test generation feature

By letting users write an enum such as following,

  public enum QuadraticEquationConstraint implements Constraint {
     A_IS_NON_ZERO("aIsNonZero") {
       @Override
       public boolean check(...) {...
       }
     },
     DISCRIMINANT_NON_NEGATIVE("discriminantIsNonNegative") { ... },
     A_IS_IN_RANGE("coefficientsAreValid") { ... },
     B_IS_IN_RANGE("coefficientsAreValid") { ... },
     C_IS_IN_RANGE("coefficientsAreValid") { ... },;
  }

if JCUnit generates a sub test suite for 'negetive' testing, it will be very useful.
The sub test suite should

• contain test cases that violate all the conditions defined by check method of the enum.
• each of those test cases should violate one and only one constraint.
• A factor level not covered by positive tests, it should be covered by at least one of negative test cases. (All the factor levels should be covered by 'negative' and/or 'positive' testing.)

Define a behavior when @Out annotation is specified for non-primitive/non-collection fields

Currently if an @out annotated field is non-primitive/non-collection, the value will not be stored (only ) is recorded and therefore tests for the fields will always pass, which isn't preferable.

We should define clear behavior for these fields.
Option 1. Binary serializer should be used instead.
Option 2. JCUnit should complain that non serializable objects can't be used.

Refine test runner using JUnit 4.12 feature

Refine test runner using JUnit 4.12 feature.
Current mechanism is doing dirty since it is based on JUnit available at the moment JCUnit's development started.

Create (or enhance) a "smart" constraint manager to use @Condition annotated methods as constraints

Create (or enhance) a "smart" constraint manager to use @condition annotated methods as constraints.

Defining constraints between arguments for actions in FSM

I have a FSM which can perform two actions, which both take arguments in common. Those arguments have constraints, which renders some combinations invalid. How would I be able to define those constraints?

Introduce @InitSpec feature to instantiate a new FSM

Introduce @InitSpec feature to instantiate a new FSM.
This might be useful to implement feature that creates a new child FSM instance when FSMs are nested and child is not invoked by parent.

(FSM)Coverage report (Experimental feature)

generating test suite which covers all the possible value-pairs under complicated constraints is a very time consuming task. Instead, relying on random generation and assessing how much possible value pairs are covered might be more practical and good enough.

MethodLevelsProvider doesn't work.

MethodLevelsProvider doesn't work anymore.
Implement from attribute in @FactorField annotation.

(FSM)Support overloading methods with the same number of arguments

Right now FSM/JCUnit doesn't support those methods. But it would be nice to have.

'Load' and 'store' function objects are not printed correctly

'Load' and 'store' function objects are not printed correctly in 'ALL TEST RULES' section.
It's printed like below,

JCUNIT - [01] 13/ 0 - (eq,(100,CalcTest1.obj,c),(get,CalcTest1.obj,c))

And '100' is the last used value in the test case in the session, while 'ALL TEST RULES' section is describing the entire test session.
This should be something like below,

JCUNIT - [01] 13/ 0 - (eq,(load,{test name},CalcTest1.obj,c),(get,CalcTest1.obj,c))

Make input history feature support "aliases".

Make input history feature support "aliases" for method parameters and actions (methods).

0.6.x and sources that use 0.6.x cannot be compiled by JDK 1.7.0 older than 1.7.0_80

0.6.x and sources that use 0.6.x cannot be compiled by JDK 1.7.0 older than 1.7.0_80 with following error.

annotation com.github.dakusui.jcunit.runners.standard.annotations.Checker is missing value for the attribute <clinit>

This is caused by a following bug in javac which is introduced somewhere in 1.7.0 and fixed in JDK 1.7.0_80

• http://www.oracle.com/technetwork/java/javase/2col/7u80-bugfixes-2494167.html
• http://bugs.java.com/view_bug.do?bug_id=8013485

As of now, this is only reproduced by older 1.7.0 and not reproduced by 1.6.0_45/1.8.0_77.

(FSM)Handle a history of inputs to FSMs

When we want to test output of FSM, we sometimes want to build an expectation from a history of inputs.
E.g., Suppose we want to test a StringBuilder, internal state is I only, but the output of build (toString()) is determined by the inputs by the time the method is called (a history).

We want a way to let users model/represent this.