dakusui / jcunit Goto Github PK

By letting users write an enum such as following,

  public enum QuadraticEquationConstraint implements Constraint {
     A_IS_NON_ZERO("aIsNonZero") {
       @Override
       public boolean check(...) {...
       }
     },
     DISCRIMINANT_NON_NEGATIVE("discriminantIsNonNegative") { ... },
     A_IS_IN_RANGE("coefficientsAreValid") { ... },
     B_IS_IN_RANGE("coefficientsAreValid") { ... },
     C_IS_IN_RANGE("coefficientsAreValid") { ... },;
  }

if JCUnit generates a sub test suite for 'negetive' testing, it will be very useful.
The sub test suite should

• contain test cases that violate all the conditions defined by check method of the enum.
• each of those test cases should violate one and only one constraint.
• A factor level not covered by positive tests, it should be covered by at least one of negative test cases. (All the factor levels should be covered by 'negative' and/or 'positive' testing.)

Support static method.
Probably just doing a static method call if a Class object is passed as SUT would be sufficient.
(java.lang.Class is marked final.)

Create (or enhance) a "smart" constraint manager to use @condition annotated methods as constraints.

'Load' and 'store' function objects are not printed correctly in 'ALL TEST RULES' section.
It's printed like below,

JCUNIT - [01] 13/ 0 - (eq,(100,CalcTest1.obj,c),(get,CalcTest1.obj,c))

And '100' is the last used value in the test case in the session, while 'ALL TEST RULES' section is describing the entire test session.
This should be something like below,

JCUNIT - [01] 13/ 0 - (eq,(load,{test name},CalcTest1.obj,c),(get,CalcTest1.obj,c))

Support Theories runner of JUnit.

• About Theories runner: http://junit.org/apidocs/org/junit/experimental/theories/Theories.html

This has already been done in following commits.

• 4a8ad05
• e6b2a34
• 2255f89

Following is a diagram that illustrates intended refactoring.

0.6.x and sources that use 0.6.x cannot be compiled by JDK 1.7.0 older than 1.7.0_80 with following error.

annotation com.github.dakusui.jcunit.runners.standard.annotations.Checker is missing value for the attribute <clinit>

This is caused by a following bug in javac which is introduced somewhere in 1.7.0 and fixed in JDK 1.7.0_80

• http://www.oracle.com/technetwork/java/javase/2col/7u80-bugfixes-2494167.html
• http://bugs.java.com/view_bug.do?bug_id=8013485

As of now, this is only reproduced by older 1.7.0 and not reproduced by 1.6.0_45/1.8.0_77.

Add tests for "local constraint" feature. Current existing tests do not have sufficient coverage.

I have a FSM which can perform two actions, which both take arguments in common. Those arguments have constraints, which renders some combinations invalid. How would I be able to define those constraints?

Introduce @InitSpec feature to instantiate a new FSM.
This might be useful to implement feature that creates a new child FSM instance when FSMs are nested and child is not invoked by parent.

JCUnit is becoming bigger and its documentation is so, too.
Before releasing 0.6.0, documentations should be organized better.

FSM is a so powerful idea that you can model broad range of software systems.
If we can mode a software product using it, and let JCUnit generate test suite and execute it over the SUT, wouldn't it sound nice?

The enhancement is on-going on this branch: https://github.com/dakusui/jcunit/tree/nestedfsm

Refine constraint checking mechanism.
Right now ConstraintChecker desinged to check all constraints for a factor space at once.
But test suite generator implementations (CoveringArrayEngine) normally need to access each of them individually to generate a good test suite.
Also such implementations need to know which factors each constraint relies on.

Following is a diagram that illustrates current class structure of JCUnit's constraint handling mechanism.

ASIS

    +-------------------+           +-----------------+
    |CoveringArrayEngine|---------->|<<interface>>    |
    +-------------------+ 1       1 |ConstraintChecker|
                                    +-----------------+
                                    |check(TestCase t)|
                                    +-----------------+
                                             |
                                             A
                                             |
                                    +----------------------+        +-----------------+
                                    |SmartConstraintChecker|<>----->|Constraint       |
                                    +----------------------+  1   * +-----------------+
                                                                    |check(TestCase t)|
                                                                    +-----------------+

As seen, CoveringArrayEngines only knows top-level interface of constraint checker. If an unassigned value is used by any of constraints, JCUnit's IPO CA generator traverses all the possible test cases recursively assuming there is a level(s) for the factor that makes the on-going test case valid. If it can't find any, the test case will be removed.
This is too funky and inefficient.

Instead, we should have a structure like following, where covering array engines can access all the necessary information to figure out valid test cases under a set of constraints.

TOBE

    +-------------------+           +-----------------+        +-----------------+
    |CoveringArrayEngine|---------->|<<interface>>    |<>----->|Constraint       |
    +-------------------+ 1       1 |ConstraintChecker|  1   * +-----------------+
                                    +-----------------+        |check(TestCase t)|
                                    |check(TestCase t)|        |getFactors(): F[]| returns involved factors
                                    +-----------------+        +-----------------+
                                             |
                                             A
                                             |
                                 +----------------------+
                                 |SmartConstraintChecker|
                                 +----------------------+

Design page is found here

Currently output formatting logic is embedded into core testing logic and not pluggable.
This should be separated from the core logic and pluggable.

Clean up code thoroughly toward the stable release 0.6.x.
It would be the last opportunity to do so wildly.

Create a simple builder API

@actionspec should only need to be attached to top level but this this property is not tested.
And FSM related tests are giving it to all the overriding methods.

public enum IncorrectSpec implements FSMSpec<ModifiedMealyMachine> {
  @StateSpec I {
  },
  @StateSpec S {
    @ActionSpec // <-- This is not necessary
    public Expectation<ModifiedMealyMachine> getS(Expectation.Builder<ModifiedMealyMachine> b) {
        ....
    }

    @ActionSpec // <-- This is not necessary
    public Expectation<ModifiedMealyMachine> getI(Expectation.Builder<ModifiedMealyMachine> b) {
      return b.valid(S).build();
    }
  };

...

  @ActionSpec(alias="method")
  public Expectation<ModifiedMealyMachine> method(Expectation.Builder<ModifiedMealyMachine> b, String s) {
    ...
  }

A place holder ticket for small refactorings, test enhancements, or improvements.

Make input history feature support "aliases" for method parameters and actions (methods).

The location to store previous run is set to a path 'new File(".jcunit").getAbsolutePath()'.
But users need to change it in case they want to include the test results as a part of test definitions and submit it to source code repository for instance.

Enhance unit tests toward 0.6.0 release.

Currently if an @out annotated field is non-primitive/non-collection, the value will not be stored (only ) is recorded and therefore tests for the fields will always pass, which isn't preferable.

We should define clear behavior for these fields.
Option 1. Binary serializer should be used instead.
Option 2. JCUnit should complain that non serializable objects can't be used.

I would like to be able to write a test case like below,

public class ManageLorries {
  @Given("I have gone to the new lorry page")
  @When({"I fill in 'Name' with 'name 1'", "I press 'Create'"})
  @Then("I should see 'name 1 - this is from before filter'")
  public void registerNewLorry() {
    ...
  }
}

Since guice provides pretty decent API for assigning values to fields based on type information.
Potentially it's worth considering use it instead of the home-brew mechanism which is being used currently.

When we want to test output of FSM, we sometimes want to build an expectation from a history of inputs.
E.g., Suppose we want to test a StringBuilder, internal state is I only, but the output of build (toString()) is determined by the inputs by the time the method is called (a history).

We want a way to let users model/represent this.

A logic to construct a scenario sequence for test preparation sometimes creates unintentional one

As discussed in this blog, it would be useful to make generated test suite reusable.

As of now, JCUnit has "Recorder" and "Replayer" mechanism , but

• This is a feature to reuse output of SUT as test expectation. Not generated test suite.
• The documentation about it is not sufficient.
• It requires code change on record/replay.
• How it should behave when the test suite definition (what factors the test class has, what levels each factor has, etc.) is changed isn't defined.
• How to detect those changes (or a user should be responsible for it) is not defined.

At least JCUnit needs documented best practice for this.

generating test suite which covers all the possible value-pairs under complicated constraints is a very time consuming task. Instead, relying on random generation and assessing how much possible value pairs are covered might be more practical and good enough.

Currently Levels providers are directly instantiated by JCUnit. But they should be created by "regular" way where users can implement their own levels providers easily (at least in comparison).

Separate annotation system from the other parts of JCUnit and allow users to

• reuse test suite generator
• create their own annotation systems.

Make it possible to specify factors used by a constraint.

@Uses({"a", "b"})
@Condition(constraint = true)
public boolean isConstraintSatisfied() {
  return   this.a == ... &&  this.b == ...
}

If a or b is not yet set by JCUnit, the framework will not call the method.

Users want to customize how previous objects and current objects are compared in automatic regression tests, where the outputs of the last run are recorded and used for comparisons from the next run on.
Using 'equals' method is not enough since users might want to test objects that do not override equals method.

FSM/JCUnit treats each state in each FSM as an independent factor. Therefore it guarantees a fair coverage for multi-threading situation where

Step. 1, client 1 requests op.A the server and at the same time client 2 requests Op.B,
Step. 2, ...

with relatively small number of test cases since you can use pairwise/t-wise technique as its test suite generation method.

(A place holder ticket for document updated)

JCUnit already has API which enables users to generate test suites programmatically, but it doesn't support some important features like FSM etc.

Let's support it.

Design page is found here

Would it be possible for me to use this tool to generate the combination table but not use JUnit to execute? I read the section about generating the test. Will that work without JUnit present?

right now (Sep/2015), FSM/JCUnit chooses states not only events as factors. But, except for the first one, states should always be determined by the previous state and an input symbol (event) given to the FSM. As discussed in [Introduction to Combinatorial Testing]4, in this situation we can exclude them during test suite generation from factors and after the process finishes we can append states determined by previous states and input to each test case. By this optimization, we should be able to improve FSM/JCUnit's performance.

It is a possible situation where a test method uses some of factors defined in a test class.
In such a case, if the levels of the factors are exactly the same, the test shouldn't be executed repeatedly.

One idea is to allow users to declare factors used in a test method by using a new annotation '@uses'

@Test
@Uses("factor1", "factor3)
public void test() {
    use this.factor1 and 2
}

Right now JCUnit is utilizing JUnit 4.12's validation framework only partially.
It should be used more extensively and consistently.

Right now FSM/JCUnit doesn't support those methods. But it would be nice to have.

Right now Checks.rethrow wraps and rethrow checked exceptions from lower levels.
Instead we shoud introduce Checks.wrap checked exception and 'return' wrapping runtime exception and callers should be responsible for throwing returned runtime exception.
By this we will not need to suppress around warnings caused by the rethrow method (because it makes java compiler not understand the method is aborted at the points where the method is called)

As of now, JCUnit's annotation system became more complicated than it should be.
Simplify this.

In the following example, users should not need to write @GenerateCoveringArrayWith annotation every time because the setting is pretty typical. Let's make them (values set to checker and reporters) default of those attributes.

@RunWith(JCUnit.class)
@GenerateCoveringArrayWith(
    checker = @Checker(value = SmartConstraintChecker.class),
    reporters = {
        @Reporter(value = CombinatorialMetrics.class, args = { @Value("2") })
    })
public class QuadraticEquationSolverTest8 {
  @FactorField
  public int a;
  @FactorField
  public int b;
  @FactorField
  public int c;

  @Uses({ "a" })
  @Condition(constraint = true)
  public boolean aIsNonZero() {
    return this.a != 0;
  }

  ...

  @Test(expected = IllegalArgumentException.class)
  @When({ "!aIsNonZero" })
  public void solveEquation1$thenThrowIllegalArgumentException() {
    new QuadraticEquationSolver(
        a,
        b,
        c).solve();
  }

  ...

Design page is found here

Expose default values of @FactorField somehow.

probably we want to define constraints applied to parameters of a method.

(FSM)Ensure FSMUtils.resetStories is called before each test method is called every time to reset story's state (isPerformed)

Potentially, nested stories will not be executed if there are more than one method in a test object.

MethodLevelsProvider doesn't work anymore.
Implement from attribute in @FactorField annotation.

Support constraint handling for AETG

Currently I have been using variable names "degree" or "strength" both for how many factors whose all possible combinations of levels to be covered covered in a test suite, but there is no policy.
Maybe I should always use "strength" since I haven't seen the usage of "degree" as much as the other's.

Refine test runner using JUnit 4.12 feature.
Current mechanism is doing dirty since it is based on JUnit available at the moment JCUnit's development started.

IPO algorithm isn't good at handling constraints. Probably AETG can handle it better since it generates a complete tuple every time.

This feature is not only useful for FSM support but also useful for regular test suite generations.