dbenhur / lambdatest

Functional testing for Scala.

LambdaTest

LambdaTest is a testing library for Scala code. Although it is itself fully functional/immutable, it can be used for testing any Scala code.

LambdaTest has the following features.

• Written entirely in Scala.
• Fully functional using immutable testing state.
• Simple with relatively little code (about 1K lines of Scala).
• Easy to customize and extend.
• Can include ScalaCheck properties as tests.
• Tests can be run either via SBT or directly.

Comparison with ScalaTest and Specs2

Feature	LambdaTest	ScalaTest	Specs2
lines of code	~1K	~ 400K	~50K
written in	Scala	Scala/Java	Scala/Java
functional/immutable	yes	no	no
easy to customize	yes	no	no
easy to generate tests	yes	no	no
wrap code	wrappers	before/after	before/after
supports ScalaCheck	yes	yes	yes
tagged/ignored tests	yes	yes	yes
timed tests	yes	yes	no
performance tests	yes	no	no
major dependencies	only ScalaCheck	lots	lots

LambdaTest has a simple clean fully functional/immutable API that makes it easy to define new kinds of assertions and compound testing forms (such as wrappers). This simple API also provides the option of having code that generates an entire test suite rather than having to write each test manually.

Wrappers offer many advantages over use of before and after.

• Single function (wrapper) rather than 2 (before and after).
• Does not require mutable state.
• Can catch exceptions.
• Can hide state needed both before and after.
• Can be applied at any level.

All these features are discussed below with links to demo code that illustrate the features.

Concepts

• Reporter. Used to report test results.
• State. All state needed to run tests is stored in objects of type LambdaState. State objects are immutable.
• Action. Actions transform an old state to a new state and are objects of type LambdaAct. Each LambdaAct object contains an ordered list of transforms. Each transform maps an old state to a new state. A LambdaAct with exactly one transform is called a single action, with no transforms is call an empty action and with two or more transforms is called a multiple action. Actions can be combined using the infix + operator.

Actions are created using functions that create objects of type LambdaAct. These functions are also informally refered to as actions.

Multiple actions can also be specified as a list of actions. This is done via an implicit conversion from Seq[LambdaAct] to LambdaAct. See the Generate demo for an example that used this conversion.

Documentation

• This README file.
• ScalaDoc for APIs.
• Sample demos in src/test/scala/demo
• Description of and defaults for options in src/main/resources/reference.conf.

Quick Start

Include LambdaTest jar

"com.fortysevendeg" % "lambda-test_2.12" % "1.2.1" % "test"

"com.fortysevendeg" % "lambda-test_2.11" % "1.1.2" % "test"

Each test file should

include com.fortyseven.lambdatest._

Each test file class should extend the LambdaTest trait.

Each test class must define a local val act whose value is the tests to be run.

If you want to run tests via SBT then the build.sbt file should specify

testFrameworks += new TestFramework("com.fortysevendeg.lambdatest.sbtinterface.LambdaFramework")

To see a simple example look at the Example demo.

Actions

Actions are created using functions contained in

src/main/scala/com/fortysevendeg/lambdatest/package.scala

Additional actions can be easily defined.

Simple Actions

Simple actions do not contain other actions.

• assert. Tests a boolean predicate. See the Example demo.
• assertEq. Tests that two values are equal. See the Example demo.
• assertEx. Test that an exception is raised. See the Except demo
• assertSC. Used to test a ScalaCheck property. See the ScalaCheck demo.
• exec. Used to insert Scala code. See the Mutable demo.
• assertPerf. Runs an expression multiple times and reports mean, max, and stdDev. Can assert a maximum mean and max. See the Timing demo.

Compound Actions

Compound actions contain other actions within themselves.

• label. Introduced a labeled block of code. See the Example demo.
• test. Defines a named test. See the Example demo.
• changeOptions. Used to change options within its body. See the FailOnly test.
• timer. Reports the execution time of its body. See the Timing demo.
• assertTiming. Reports the execution time of its body. Fails if a max time is exceeded. See the Timing demo.
• nest. Used to nest declarations in immutable tests. See the Immutable demo.

Assertion Nesting Rule

• Assertions must either directly or indirectly inside a test.

See Bad demo for what not to do.

Running Tests

• Via SBT. Classes that extend the LambdaTest trait can be run using the SBT test commands.
• Directly. This is done using the run function. See the companion objects in the demos. Note these companion objects are not needed for running via SBT.

Immutablility and Side Effects

The testing system itself contains no mutable data. It does however have one kind of side effect: tests results are output. It would have been more pure to move the output entirely after the core functionallity, but it is nice to see test output incrementally as tests are run so a little purity is sacrificed.

Although LambdaTest is ideal for testing pure functional code, it can also be used to test code with mutable state and side effects. See the Mutable demo.

Including Scala Code

Suppose we have a test that contains several assertions. We can add arbitrary code as follows.

• Before all assertions. Put the code in the body of the test action. See the Mutable demo.
• Between assertions (mutable). Add exec actions between assertions. See the Mutable demo.
• Between assertions (immutable). Use nesting. See the Immutable demo.
• After all assertions. The can be done using a user defined compound action called a wrapper. See the Wrap demo. Wrappers are used instead of the before and after features of other testing frameworks. Wrappers avoid the need to use mutable state and can easily capture exceptions

Exceptions

Expected exceptions can be checked using the assertEx action.

Unexpected exceptions are caught and treated as test failures. Where possible an unexpected exception will not stop later tests from being run.

Tags

The label and test actions can have sets of tags that can be used to selectively execute only some tests. There are options includeTags and excludeTags that can be set to select the subset of tests to be run.

By default, including the tag "ignore" on a label or test causes that action to be excluded.

See the Tag demo.

Timing

There are several actions that can be used to time code. Some are assertions that fail if the code runs too long.

See the Timing demo.

Parallel Execution

The run function and the label and test actions have an optional parallel parameter.

By default those actions directly inside are run sequentially in order.

If parallel is true, actions directly inside are run in parallel. Note that the output still occurs in the specified order.

See the Parallel demo.

Extensions

LambdaTest is designed to be easily extensible.

• Test generation. The clean simple design of LambdaTest makes generation of tests easy. See the Generate demo for some simple examples.
• New actions. New actions are easily defined. See the code in package.scala for examples. Actions typically use the methods of LambdaState and the eval method of LambdaAct. These methods should not be called directly in user test code. Wrappers are one kind of user defined action (see the Wrap demo).
• Alternate reporters. Reporters collect and display the results of tests. Reporters are included for both StdOut (for tests run directly) and SBT (for tests run using the SBT test command). Custom reporters can created by extending the LambdaReporter trait.

Contributing

Comments, suggestions and pull requests are welcome. Since the goals are to keep this system small and extensible, additions of most features that could be easily added as extensions are not wanted. Instead they should be placed in other optional libraries.