Coder Social home page Coder Social logo

dakusui / jcunit Goto Github PK

View Code? Open in Web Editor NEW
76.0 4.0 9.0 26.29 MB

A Combinatorial Testing + Model-based Testing framework on top of Java+JUnit. Model your application as code and let JCUnit do the rest.

License: Apache License 2.0

Java 98.98% Shell 1.02%
model-based-testing combinatorial-testing junit

jcunit's Introduction

Design Your Test as Code: The JCUnit 0.10.x

JCnit

is an open source model-based testing (MBT) framework powered by combinatorial interaction testing (CIT) technique. It supports various models to provide a uniformed testing style. It employs CIT to balance the coverage and test suite size. The documentation is found here.

Maven Coordinate for JCUnit
<dependency>
    <groupId>com.github.dakusui</groupId>
    <artifactId>jcunit</artifactId>
    <version>{jcunit-version}</version>
</dependency>

JCUnit 0.10.x is under development. The most recent stable major version of JCUnit is 0.8.x. Please check

JCUnit 0.10.x: Key Features to Come

0.10.x will be the last version line previous to 1.0.x, which is intended for the "true" public usage and broader audience.

The project board is found here.

  • Enhanced Regex Support: (#152, #155)

  • JUnit5: support (#142)

  • Temporary removal of FSM support: (It will be brought back in 1.0.x, #169).

  • PICT integration: (#179)

  • Simplified Internal Structure for integrating external CIT tools: (Stretch goal, #175).

  • Fault Localizer: (Stretch goal, #144)

Limitations

References

PICT

PICT is the most popular tool for Combinatorial Interaction Testing (CIT). It can generate a covering array from a text file described in its own human-readable notation. It was developed by Jacek Czerwonka and now an open source project from Microsoft.

JCUnit 0.10.x (or later) uses PICT for covering array generation for the part (factors) it can be used and then combine it with another covering array generated by JCUnit’s internal covering array generator for the rest (#179).

JCUnit

We have published some key ideas of JCUnit as academic papers and a blog.

Publications

Build Instructions

The JCUnit requires Java SE 8 to build. Please don’t forget installing and using it for building. This is intentional decision to make the testing compatible for as many product as possible.

Build
$ mvn clean compile
Conduct Unit Test
$ mvn clean compile test
Generating Site
$ mvn clean package site

Generates the site under target/site in the following structure.

target/
  site/
    apidocs/     - JavaDoc
    jacoco/      - JaCoCo report (code coverage)
    pit-reports/ - PIT report (mutation coverage)

The mutation testing is configured to cover only small amount of classes to save build time. To configure the coverage, refer to [mutationTesting].

Deploying Site
$ mvn clean package site

Misc

Conduct Mutation Test
$ mvn clean compile test org.pitest:pitest-maven:mutationCoverage

Currently, this may take very long time (≥ 2 hours), if you include all the classes of this product. By default, the pom.xml is configured to exercise the mutation testing over small number of classes so that you customize the intended coverage.

Under the plugin configuration for org.pitest:pitest-maven, you can find a following element.

pom.xml configuration for org.pitest:pitest-maven plugin
    <targetClasses>
        <param>com.github.dakusui.jcunit8.exceptions.*</param>
    </targetClasses>

To cover all the JCUnit classes, you can modify it like following.

pom.xml configuration for org.pitest:pitest-maven plugin
    <targetClasses>
        <param>com.github.dakusui.*</param>
    </targetClasses>

After a successful execution, it generates a pitest report under a directory target/pit-reports/{yyyyMMddHHmmss}.

Build Javadoc
$ mvn clean javadoc:javadoc

This generates JavaDoc under target/site/apidocs.

Build the Site
$ mvn clean package site
Publish the Site
$ mvn clean package site site-deploy

Copyright 2013 Hiroshi Ukai.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this work except in compliance with the License. You may obtain a copy of the License in the LICENSE file, or at:

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

jcunit's People

Contributors

dakusui avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

jcunit's Issues

Enhance JCUnit API to support FSMs

JCUnit already has API which enables users to generate test suites programmatically, but it doesn't support some important features like FSM etc.

Let's support it.

Design page is found here

(FSM)Multi-threading support

FSM/JCUnit treats each state in each FSM as an independent factor. Therefore it guarantees a fair coverage for multi-threading situation where

Step. 1, client 1 requests op.A the server and at the same time client 2 requests Op.B,
Step. 2, ...

with relatively small number of test cases since you can use pairwise/t-wise technique as its test suite generation method.

AETG algorithm support

IPO algorithm isn't good at handling constraints. Probably AETG can handle it better since it generates a complete tuple every time.

This feature is not only useful for FSM support but also useful for regular test suite generations.

Consider using Guice for assigning fields' values

Since guice provides pretty decent API for assigning values to fields based on type information.
Potentially it's worth considering use it instead of the home-brew mechanism which is being used currently.

Make generated test suite reusable (experimental feature)

As discussed in this blog, it would be useful to make generated test suite reusable.

As of now, JCUnit has "Recorder" and "Replayer" mechanism , but

  • This is a feature to reuse output of SUT as test expectation. Not generated test suite.
  • The documentation about it is not sufficient.
  • It requires code change on record/replay.
  • How it should behave when the test suite definition (what factors the test class has, what levels each factor has, etc.) is changed isn't defined.
  • How to detect those changes (or a user should be responsible for it) is not defined.

At least JCUnit needs documented best practice for this.

(FSM)Simplify test suite generation

right now (Sep/2015), FSM/JCUnit chooses states not only events as factors. But, except for the first one, states should always be determined by the previous state and an input symbol (event) given to the FSM. As discussed in [Introduction to Combinatorial Testing]4, in this situation we can exclude them during test suite generation from factors and after the process finishes we can append states determined by previous states and input to each test case. By this optimization, we should be able to improve FSM/JCUnit's performance.

Refine constraint checking mechanism

Refine constraint checking mechanism.
Right now ConstraintChecker desinged to check all constraints for a factor space at once.
But test suite generator implementations (CoveringArrayEngine) normally need to access each of them individually to generate a good test suite.
Also such implementations need to know which factors each constraint relies on.

Following is a diagram that illustrates current class structure of JCUnit's constraint handling mechanism.

ASIS

    +-------------------+           +-----------------+
    |CoveringArrayEngine|---------->|<<interface>>    |
    +-------------------+ 1       1 |ConstraintChecker|
                                    +-----------------+
                                    |check(TestCase t)|
                                    +-----------------+
                                             |
                                             A
                                             |
                                    +----------------------+        +-----------------+
                                    |SmartConstraintChecker|<>----->|Constraint       |
                                    +----------------------+  1   * +-----------------+
                                                                    |check(TestCase t)|
                                                                    +-----------------+

As seen, CoveringArrayEngines only knows top-level interface of constraint checker. If an unassigned value is used by any of constraints, JCUnit's IPO CA generator traverses all the possible test cases recursively assuming there is a level(s) for the factor that makes the on-going test case valid. If it can't find any, the test case will be removed.
This is too funky and inefficient.

Instead, we should have a structure like following, where covering array engines can access all the necessary information to figure out valid test cases under a set of constraints.

TOBE

    +-------------------+           +-----------------+        +-----------------+
    |CoveringArrayEngine|---------->|<<interface>>    |<>----->|Constraint       |
    +-------------------+ 1       1 |ConstraintChecker|  1   * +-----------------+
                                    +-----------------+        |check(TestCase t)|
                                    |check(TestCase t)|        |getFactors(): F[]| returns involved factors
                                    +-----------------+        +-----------------+
                                             |
                                             A
                                             |
                                 +----------------------+
                                 |SmartConstraintChecker|
                                 +----------------------+

Design page is found here

Make it possible to specify factors used by a constraint

Make it possible to specify factors used by a constraint.

@Uses({"a", "b"})
@Condition(constraint = true)
public boolean isConstraintSatisfied() {
  return   this.a == ... &&  this.b == ...
}

If a or b is not yet set by JCUnit, the framework will not call the method.

Can I Use Without JUnit?

Would it be possible for me to use this tool to generate the combination table but not use JUnit to execute? I read the section about generating the test. Will that work without JUnit present?

Add BDD support

I would like to be able to write a test case like below,

public class ManageLorries {
  @Given("I have gone to the new lorry page")
  @When({"I fill in 'Name' with 'name 1'", "I press 'Create'"})
  @Then("I should see 'name 1 - this is from before filter'")
  public void registerNewLorry() {
    ...
  }
}

Use variable names "degree" and "strength" consistently (Refactoring)

Currently I have been using variable names "degree" or "strength" both for how many factors whose all possible combinations of levels to be covered covered in a test suite, but there is no policy.
Maybe I should always use "strength" since I haven't seen the usage of "degree" as much as the other's.

Reduce number of test executions where possible

It is a possible situation where a test method uses some of factors defined in a test class.
In such a case, if the levels of the factors are exactly the same, the test shouldn't be executed repeatedly.

One idea is to allow users to declare factors used in a test method by using a new annotation '@uses'

@Test
@Uses("factor1", "factor3)
public void test() {
    use this.factor1 and 2
}

Clean up code

Clean up code thoroughly toward the stable release 0.6.x.
It would be the last opportunity to do so wildly.

Organize documentation

JCUnit is becoming bigger and its documentation is so, too.
Before releasing 0.6.0, documentations should be organized better.

@ActionSpec should only need to be attached to top level.

@actionspec should only need to be attached to top level but this this property is not tested.
And FSM related tests are giving it to all the overriding methods.

public enum IncorrectSpec implements FSMSpec<ModifiedMealyMachine> {
  @StateSpec I {
  },
  @StateSpec S {
    @ActionSpec // <-- This is not necessary
    public Expectation<ModifiedMealyMachine> getS(Expectation.Builder<ModifiedMealyMachine> b) {
        ....
    }

    @ActionSpec // <-- This is not necessary
    public Expectation<ModifiedMealyMachine> getI(Expectation.Builder<ModifiedMealyMachine> b) {
      return b.valid(S).build();
    }
  };

...

  @ActionSpec(alias="method")
  public Expectation<ModifiedMealyMachine> method(Expectation.Builder<ModifiedMealyMachine> b, String s) {
    ...
  }

The location to store previous run can't be changed

The location to store previous run is set to a path 'new File(".jcunit").getAbsolutePath()'.
But users need to change it in case they want to include the test results as a part of test definitions and submit it to source code repository for instance.

Simplify annotation system

As of now, JCUnit's annotation system became more complicated than it should be.
Simplify this.

In the following example, users should not need to write @GenerateCoveringArrayWith annotation every time because the setting is pretty typical. Let's make them (values set to checker and reporters) default of those attributes.

@RunWith(JCUnit.class)
@GenerateCoveringArrayWith(
    checker = @Checker(value = SmartConstraintChecker.class),
    reporters = {
        @Reporter(value = CombinatorialMetrics.class, args = { @Value("2") })
    })
public class QuadraticEquationSolverTest8 {
  @FactorField
  public int a;
  @FactorField
  public int b;
  @FactorField
  public int c;

  @Uses({ "a" })
  @Condition(constraint = true)
  public boolean aIsNonZero() {
    return this.a != 0;
  }

  ...

  @Test(expected = IllegalArgumentException.class)
  @When({ "!aIsNonZero" })
  public void solveEquation1$thenThrowIllegalArgumentException() {
    new QuadraticEquationSolver(
        a,
        b,
        c).solve();
  }

  ...

Design page is found here

Refine exception wrapping mechanism

Right now Checks.rethrow wraps and rethrow checked exceptions from lower levels.
Instead we shoud introduce Checks.wrap checked exception and 'return' wrapping runtime exception and callers should be responsible for throwing returned runtime exception.
By this we will not need to suppress around warnings caused by the rethrow method (because it makes java compiler not understand the method is aborted at the points where the method is called)

Make it possible to customize how objects are compared in tests

Users want to customize how previous objects and current objects are compared in automatic regression tests, where the outputs of the last run are recorded and used for comparisons from the next run on.
Using 'equals' method is not enough since users might want to test objects that do not override equals method.

(FSM)Support static method

Support static method.
Probably just doing a static method call if a Class object is passed as SUT would be sufficient.
(java.lang.Class is marked final.)

Implement "smart" negative test generation feature

By letting users write an enum such as following,

  public enum QuadraticEquationConstraint implements Constraint {
     A_IS_NON_ZERO("aIsNonZero") {
       @Override
       public boolean check(...) {...
       }
     },
     DISCRIMINANT_NON_NEGATIVE("discriminantIsNonNegative") { ... },
     A_IS_IN_RANGE("coefficientsAreValid") { ... },
     B_IS_IN_RANGE("coefficientsAreValid") { ... },
     C_IS_IN_RANGE("coefficientsAreValid") { ... },;
  }

if JCUnit generates a sub test suite for 'negetive' testing, it will be very useful.
The sub test suite should

  • contain test cases that violate all the conditions defined by check method of the enum.
  • each of those test cases should violate one and only one constraint.
  • A factor level not covered by positive tests, it should be covered by at least one of negative test cases. (All the factor levels should be covered by 'negative' and/or 'positive' testing.)

(FSM)Coverage report (Experimental feature)

generating test suite which covers all the possible value-pairs under complicated constraints is a very time consuming task. Instead, relying on random generation and assessing how much possible value pairs are covered might be more practical and good enough.

'Load' and 'store' function objects are not printed correctly

'Load' and 'store' function objects are not printed correctly in 'ALL TEST RULES' section.
It's printed like below,

JCUNIT - [01] 13/ 0 - (eq,(100,CalcTest1.obj,c),(get,CalcTest1.obj,c))

And '100' is the last used value in the test case in the session, while 'ALL TEST RULES' section is describing the entire test session.
This should be something like below,

JCUNIT - [01] 13/ 0 - (eq,(load,{test name},CalcTest1.obj,c),(get,CalcTest1.obj,c))

0.6.x and sources that use 0.6.x cannot be compiled by JDK 1.7.0 older than 1.7.0_80

0.6.x and sources that use 0.6.x cannot be compiled by JDK 1.7.0 older than 1.7.0_80 with following error.


annotation com.github.dakusui.jcunit.runners.standard.annotations.Checker is missing value for the attribute <clinit>

This is caused by a following bug in javac which is introduced somewhere in 1.7.0 and fixed in JDK 1.7.0_80

As of now, this is only reproduced by older 1.7.0 and not reproduced by 1.6.0_45/1.8.0_77.

(FSM)Handle a history of inputs to FSMs

When we want to test output of FSM, we sometimes want to build an expectation from a history of inputs.
E.g., Suppose we want to test a StringBuilder, internal state is I only, but the output of build (toString()) is determined by the inputs by the time the method is called (a history).

We want a way to let users model/represent this.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.