Coder Social home page Coder Social logo

greghaskins / spectrum Goto Github PK

View Code? Open in Web Editor NEW
143.0 17.0 23.0 1.09 MB

A BDD-style test runner for Java 8. Inspired by Jasmine, RSpec, and Cucumber.

License: MIT License

Java 100.00%
java junit bdd testing java-8 unit-testing rspec jasmine gherkin test-runner

spectrum's Introduction

Spectrum

Build Status Codecov MIT License Download Gitter

A colorful BDD-style test runner for Java

Spectrum is inspired by the behavior-driven testing frameworks Jasmine and RSpec, bringing their expressive syntax and functional style to Java tests. It is a custom runner for JUnit, so it works with many development and reporting tools out of the box.

Spectrum with Eclipse via JUnit

Getting Started

Spectrum 1.2.0 is available as a package on JCenter and Maven Central.

Examples

Spectrum supports Specification-style tests similar to RSpec and Jasmine:

@RunWith(Spectrum.class)
public class Specs {{

  describe("A list", () -> {

    List<String> list = new ArrayList<>();

    afterEach(list::clear);

    it("should be empty by default", () -> {
      assertThat(list.size(), is(0));
    });

    it("should be able to add items", () -> {
      list.add("foo");
      list.add("bar");

      assertThat(list, contains("foo", "bar"));
    });

  });
}}

And also Gherkin-style tests similar to Cucumber:

@RunWith(Spectrum.class)
public class Features {{

  feature("Lists", () -> {

    scenario("adding items", () -> {

      Variable<List<String>> list = new Variable<>();

      given("an empty list", () -> {
        list.set(new ArrayList<>());
      });

      when("you add the item 'foo'", () -> {
        list.get().add("foo");
      });

      and("you add the item 'bar'", () -> {
        list.get().add("bar");
      });

      then("it contains both foo and bar", () -> {
        assertThat(list.get(), contains("foo", "bar"));
      });

    });

  });
}}

For more details and examples, see the documentation.

Can I Contribute?

Yes please! See CONTRIBUTING.md.

spectrum's People

Contributors

evertones avatar greghaskins avatar pjk25 avatar richdouglasevans avatar stuart-pollock avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

spectrum's Issues

Support running tests in a random order?

Hey friends!

I rather like this project a lot -- it makes it very easy to write BDD style tests in Java. Many thanks for writing and maintaining it.

I care very deeply about test ordering and randomization - rspec, jasmine, cedar, quick, etc all support test randomization as a way of detecting test pollution.

Is this a feature that is already in Spectrum? I couldn't find any documentation or code to that provided any hint as to how to do this, so I have to assume it doesn't work.

Would this be a desired feature, if I possibly found the time to contribute a PR? Any thoughts as to how it might work?

Thank you!

Add `timeout` pre-condition

Similar to JUnit's @Test(timeout=1234) this is a suggestion to extend the current pre-condition functionality a bit like this:

it("will run but not for long", with(timeout(1234), ()->{
   // spec code
}));

This is quite easy to do, since the PreConditionBlock can just handle it for us, wrapping the real block in a timing out sentinel. But there's a wrinkle. What does:

describe("this has a timeout so what does it mean?", with(timeout(1234), ()-> {
   // specs and describes go in here
   it("tests", () -> {});
}));

If we think it means that the whole parent block runs within a timeout, then the behaviour would happen for free. I think that's the least valuable and least expected answer. More likely it means that the above is the equivalent of:

// any time out put in a describe is sent down to its next most atomic children and not executed at parent level
describe("this has a timeout so what does it mean?", ()-> {
   // specs and describes go in here
   it("tests",  with(timeout(1234, () -> {}));
}));

Thoughts on:

  • Is this valuable as a feature (clue: yes :) )
  • How do we think it should pass timeouts around the tree?
  • Is the above syntax and re-use of the PreConditions a good approach?

Installation instructions

Greg,

I realize I should know this. But I'm new to JAVA. I'm coming from Ruby & Node which has build tools that I understand very well. However, I don't know gradle or maven or ant at all.

Would it be possible to have some installation instructions included in the readme? I looked for this package on maven, but alas.

Any help would be appreciated!

Should Spectrum support environments running Java 6?

The test runner itself (everything in src/main/java) only uses features that are backward-compatible with Java 6. The idea was to keep Spectrum available as an option in scenarios where Java 8 isn't allowed and/or feasible (like testing Android apps and in many enterprise environments that aren't "ready" for Java 8 yet).

The specs themselves can be written in Java 6, but of course lambdas aren't available so they don't look as pretty:

describe("specs using Java 6", new Block() {
  @Override
  public void run() throws Throwable {

    it("can use anonymous objects", new Block() {
        @Override
        public void run() throws Throwable {
            assertEquals(1, foo);
        }
    });
  }
});

I see a few choices:

  1. Officially support Java 6 using anonymous objects (as above). While we don't have a Java 6 automated build to verify it (yet), I believe this will work just fine in the current state. This would mean keeping the production code in Java 6 mode.
  2. Officially support Java 6 using Retrolambda. This would allow us to use some of the Java 8 language features (but not APIs), while compiling to bytecode that runs on older JVMs.
  3. Forget compatibility with anything less than Java 8. This will make our development lives easier, but may eliminate the tool from consideration for some users.

Thoughts? Ideas?

Test suite uses inconsistent terminology for suite/spec

Let's standardize on what Jasmine does: describe declares a "suite" and it declares a "spec".

This will make it less confusing for future users and contributors, and is especially important for ExampleSpec since that is what people see first when they find Spectrum on GitHub.

To do:

  • Read/seach through Spectrum's tests to find usages and mis-usages of "suite", "spec", and "test"
  • Update test code to reflect proper terms

Use tags for Spectrum's own test suite

Some of Spectrum's tests are more integration-focused and slower to run (e.g. those for Spring, etc.). It would be cool to use the tagging feature to split those out so they can get run on CI without slowing down the local TDD cycle. It's not really that slow yet, but down the line I see the possibility of integrating more static analysis (such as degraph) and other types of tests, which would make sense to keep split from the unit test suite. Plus, dogfooding is good.

Allow Spring @Autowired dependency injections and Spring Boot integration

In case other users find it interesting you could include that in your Documentation:

I have created a BDDTest class that runs with Spectrum.java and also initializes Spring ApplicationContext

@RunWith(Spectrum.class)
@Ignore
@SpringBootTest
public class BDDTest {

    //global autowired objects
    @Autowired
    protected DbC conf;


    @Before
    public void beforeTest() {
        // do stuff
    }
    @After
    public void afterTest() {
        // do stuff
    }

    private TestContextManager testContextManager;

    public BDDTest() {
        try {
            this.testContextManager = new TestContextManager(getClass());
            this.testContextManager.prepareTestInstance(this);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }


}

classes that are BDD-style can extend BDDTest like that:

public class AttributeControllerTest extends BDDTest {
    @Autowired
    private IAttributeService attributeService;
    private MockMvc mockMvc;

    {
        beforeAll(() -> {
            this.graph = graphService.getGraph();
            mockMvc = MockMvcBuilders
                    .standaloneSetup(new AttributeController(attributeService))
                    .setControllerAdvice(new GlobalExceptionHandler())
                    .build();
        });

        describe("after creation of attribute", () -> {
            Attribute storedAttribute = attributeService.storeAndFetch(new Attribute());

            describe( "fetchByUuid", () -> {
                it("should return 200", () -> {
                    mockMvc.perform(get("/api/attribute/" + storedAttribute.getUuid())).andExpect(status().isOk());
             });
        });
   }
}

Haven't tried it yet but constructor Dependency Injection using @Autowired should work too

Hope someone finds this helpful!

Idea: RSpec-style aroundEach and aroundAll

RSpec provides the around hook that can be useful for situations where a normal beforeEach/afterEach doesn't offer enough flexibility. For example, it may be easier or even required to run a database transaction in a single block, as opposed to splitting into before/after steps.

The around hook is also a powerful extension point, since you can access metadata/tags/etc of the example being run. Many RSpec plugins leverage around to mix-in enhanced behavior like profiling, framework integration, etc.

Doing this in Spectrum might drive the introduction of a proper Public model for specs and suites beyond just a runnable Block. They have names, tags, and other useful metadata already. I would not expose the current internal classes as-is, though.

JUnit 5

We don't have an immediate need for JUnit 5, but as Spring moves forward we might want to start using it. Are there any plans to make Spectrum work with JUnit 5?

features listed on README not released yet under tag v1.0.0

The latest released git tag is 1.0.0, but there're a lot of amazing features listed on README which are not under any tag.

Wouldn't that be better to remove those features from README until they are released or add notes telling the user those features are not under the latest released tag yet?

There are about ~ 100 commits on master branch since v1.0.0. When are you guys going to release a new tag with all those features included?

Test modularization and dependency injection

The most obvious way to modularize tests is to have lots of test classes. This should be the default. With JUnit rules mix-ins there's further modularization.

However, it may be the intent of a test author to create a singular test suite, which is simply too long to express in a single place. Or there may be a reusable test that, dependent on data being set up, operates differently.

Here's the proposal

  • Write a demonstration of dependency injecting suppliers/rules etc from the surrounding test class into a new Test class object, which writes its tests in its parameterised constructor - this is no code change for Spectrum, since you should be able to do it already
    • The class may be a suite or a spec - it wouldn't matter
    • Actually it could be a method too - any method would do
  • Make it easy to add in classes NOT decorated with the @RunWith(Spectrum.class) as sub-suites within the test - this allows us to weave together a user-facing suite (in terms of runner output) from several small classes.

If we add tricks like parallelised running, this sort of option will really really be useful.

Dependency injection

@RunWith(Spectrum.class)
public class MyTest {{
    final Supplier<Thing> thing = let(() -> new Thing());

    new TestWhichUsesThing(thing);
    OtherTestWhichUsesThing.testsFor(thing);
}}

class TestWhichUsesThing (
    TestWhichUsesThing(Supplier<Thing> thing) {
        describe("suite", () -> {
            it("uses thing", () -> { thing.thingy(); });
        });
    }
}

class OtherTestWhichUsesThing(
    static void testFor(Supplier<Thing> thing) {
        describe("suite", () -> {
            it("uses thing", () -> { thing.thingy(); });
        });
    }
}

Suites

Just have an "include" keyword or something

// this is a super suite and it's super sweet
@RunWith(Spectrum.class)
public class MyTest {{
    include(MyOtherTest.class);
    include(MyOtherOtherTest.class);
}}

this may be as simple to implement as

//spectrum class
public static void include(final Class<?> testClass) throws Throwable {
    new ConstructorBlock(testClass).run();
}

Integration with Mockito

Hi Greg,

I have been having at Spectrum and really like the way the tests get grouped together. I wondered if there was any support for mockito annotations, or if you knew a way to get them working.

I have managed to get a Mockito test by explicitly constructing the mocks and then adding a manual call to validateMockitoUsage and reset in the afterEach, it just feels quite hard work.

Normally I would add in a Mockito JUnit rule, annotate each mock and then construct the service in a before method. Having to manually construct the mocks is not too bad, but It would be great if there was some way to trigger the validate and reset at the end of each test.

Thanks,

Peter.

When describe block contains () I am getting <no name> in IntelliJ

describe("after creation of attribute", () -> {
        describe( "latest()", () -> {
                it("should return 200", () -> {
                    mockMvc.perform(get("/api/latest/attribute")).andExpect(status().isOk());
                });
        });
});

if I replace latest() with latest everything works fine!

image

image

Thank you!

Error when using Spectrum in Android instrumented tests

Just moving this file
to this directory

when running this command: gradle connectedCheck
I get the following error:

:app:transformClassesWithDexForDebugAndroidTestDex: Error converting bytecode to dex:
Cause: Dex cannot parse version 52 byte code.
This is caused by library dependencies that have been compiled using Java 8 or above.
If you are using the 'java' gradle plugin in a library submodule add
targetCompatibility = '1.7'
sourceCompatibility = '1.7'
to that submodule's build.gradle file.
	UNEXPECTED TOP-LEVEL EXCEPTION:
java.lang.RuntimeException: Exception parsing classes
		at com.android.dx.command.dexer.Main.processClass(Main.java:775)
		at com.android.dx.command.dexer.Main.processFileBytes(Main.java:741)
		at com.android.dx.command.dexer.Main.access$1200(Main.java:88)
		at com.android.dx.command.dexer.Main$FileBytesConsumer.processFileBytes(Main.java:1683)
		at com.android.dx.cf.direct.ClassPathOpener.processArchive(ClassPathOpener.java:284)
		at com.android.dx.cf.direct.ClassPathOpener.processOne(ClassPathOpener.java:166)
		at com.android.dx.cf.direct.ClassPathOpener.process(ClassPathOpener.java:144)
		at com.android.dx.command.dexer.Main.processOne(Main.java:695)
		at com.android.dx.command.dexer.Main.processAllFiles(Main.java:592)
		at com.android.dx.command.dexer.Main.runMonoDex(Main.java:321)
		at com.android.dx.command.dexer.Main.run(Main.java:292)
		at com.android.builder.internal.compiler.DexWrapper.run(DexWrapper.java:54)
		at com.android.builder.core.DexByteCodeConverter.lambda$dexInProcess$0(DexByteCodeConverter.java:173)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
		at java.lang.Thread.run(Thread.java:745)
Caused by: com.android.dx.cf.iface.ParseException: bad class file magic (cafebabe) or version (0034.0000)
		at com.android.dx.cf.direct.DirectClassFile.parse0(DirectClassFile.java:476)
		at com.android.dx.cf.direct.DirectClassFile.parse(DirectClassFile.java:406)
		at com.android.dx.cf.direct.DirectClassFile.parseToInterfacesIfNecessary(DirectClassFile.java:388)
		at com.android.dx.cf.direct.DirectClassFile.getMagic(DirectClassFile.java:251)
		at com.android.dx.command.dexer.Main.parseClass(Main.java:787)
		at com.android.dx.command.dexer.Main.access$1600(Main.java:88)
		at com.android.dx.command.dexer.Main$ClassParserTask.call(Main.java:1722)
		at com.android.dx.command.dexer.Main.processClass(Main.java:773)
		... 16 more

1 error; aborting
 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:transformClassesWithDexForDebugAndroidTest'.
> com.android.build.api.transform.TransformException: java.lang.RuntimeException: java.lang.RuntimeException: com.android.ide.common.process.ProcessException: java.util.concurrent.ExecutionException: com.android.ide.common.process.ProcessException: Return code 1 for dex process

Allow grouping of describe blocks using @Test annotations so IDEs can provide selective test running at suite level

How about supporting this hybrid approach? This way there won't be too much need of fit, fdescribes etc and people will be able to use Spectrum without losing anything - Not sure if this is related to #79

    @Test
    public void something_suite() {
        describe("something",() -> {
            it("somethning", () -> {
                Assert.assertEquals(1, 1);
            });
        });
    }

If this can be done somehow I will migrate all my tests to use Spectrum!

Organize test suite to use Spectrum to test Spectrum

Spectrum was originally bootstrapped by test-driving with traditional JUnit tests. Most of those tests live on in packages like given.a.spec.with.nested.describe.blocks. Later on, Spectrum became feature-rich enough to be able to test itself (dogfood), for example FixturesSpec and of course ExampleSpec).

This mix of testing styles makes it hard to figure out what is tested where and how. It should be possible to convert all the old vanilla JUnit tests over to Spectrum, but if not, that exercise may reveal important functionality gaps.

I've been trying to follow these rough guidelines:

  1. Prefer testing "happy path" Spectrum behavior with direct examples (similar to the Jasmine docs). Just call the functions in the appropriate context and assert that they behave correctly.
  2. Avoid having "expected" failures or ignored specs in Spectrum's test suite. Test those "sad path" behaviors through indirection with SpectrumRunner instead. For an example, see RunnerSpec
  3. Try to organize suite files around functional boundaries (e.g. a suite about focused specs in one file, another about ignored specs, etc.)
  4. When adding features, always include an example in ExampleSpec and update the README. Test all the edge cases in the suite itself, but it's good to have an overview that is big and visible.

Run specs or suites in Parallel

I intend to build some Acceptance tests for a batch processing system. These generally have the form of:

Given we drop file X into the input
When the system has a result in the output
Then it is a good one

As you might imagine, each test uploading files or waiting for propagation time is very slow in terms of elapsed time, even though the system under test could receive all inputs at once and run them all in parallel. While we could write the whole test as a tangle of parallel concerns:

Given we drop file X into the input
And we drop file Y into the input
When the system has results in the output for X AND Y
Then X is a good one
And Y is a good one

The best option would be to write the tests to run in parallel. Spectrum style it might look like this:

feature("unconflicting use cases that can share the system", with(parallelRunning(), () -> {
   scenario("the X files", () -> {
       given("we drop the file into the input", () -> {
          system.drop("X.file");
       });
      when("the system has results", () -> {
          system.waitFor("X.result");
      });
      then("the result is good", () -> {
          validator.check("X.result", "X.expectedResult");
      });
   }); 
   
   // this scenario is expected to run in parallel with the other
   scenario("the y files", () -> {
       given(...);
       when(...);
       then(...);
   });
});

This will play havoc with the output to Eclipse or IntelliJ's runner, though potentially a RunNotifier proxy could be put in the middle to avoid that.

Executing tests in parallel would be impaired by `synchronized` block in `Spectrum`

In a situation where something like JUnit's ParallelComputer were used, or where other test runners were parallelising the construction of test classes run with the Spectrum runner, the fact that the test class construction happens inside a synchronized block will constrain the execution time.

This is relatively easy to fix. It's in a synchronized block to protect a stack from being bombarded by different threads. By introducing a thread-independent singleton into the mix, this will become thread-safe. This is what I propose to do.

Mockito annotations support?

We have found that using @Mock and @InjectMocks annotations don't work with spectrum, resulting in the mock causing a NullPointerException.

It seems that the runner needs to do all the injections and mocking pretty early in the test execution, so that all instances are injected and/or mocked.

This issue is somewhat related to this issue, but perhaps it's helpful to bring it up here separately.

Thanks for the great framework,

@ishustava && @larham

Suite and spec names with parentheses do weird things in Eclipse

describe("A suite with (parentheses)", () -> {
    it("will look weird in Eclipse", () -> {  });
});

Eclipse interprets this as

parentheses
  will look weird in Eclipse

instead of

A suite with (parentheses)
  will look weird in Eclipse

This probably has to do with how JUnit Descriptions are coupled to testName(className) format.

Deprecate and remove the `value()` helper in favor of `let()`

See the comments in #36. The Value class was a workaround for Java's requirement that closures only use final variables. It works, but isn't very clean. Instead, let() is more in line with the RSpec API, and has the additional feature of re-initializing between specs.

I propose removing Value prior to the 1.0.0 release.

Specs with the same suite + spec name get mis-reported in Eclipse

May not be Eclipse-specific, but specs with the same JUnit description spec name(suite name) get coalesced by Eclipse. This is probably a limitation of JUnit's Description and Eclipse's runner that Spectrum will need to work around. Related to #3 .

For example:

@RunWith(Spectrum.class)
public class WeirdSpecs {{

    describe("a thing", () -> {
        it("should do something", () -> {
            assertThat("failing", false);
        });
    });

    describe("a thing", () -> {
        it("should do something", () -> {
            assertThat("passing", true);
        });
    });

}}

The above gets reported as simultaneously passing, failing, and being incomplete:

screen shot 2016-02-01 at 1 33 02 am

Publish to JCenter

So that Spectrum can be more easily incorporated into Maven, Gradle, and Ivy projects.

REQUEST: Include the variable class in the next release

This really a request but are you planning on including the variable class in your next release? We use spectrum in our java and android projects and I've run across lots of situations where it would come in handy. Thanks!

aroundAll runs even when all specs are ignored

This is a low-priority one, but should probably get addressed at some point. At the moment, there isn't a simple implementation fix, because Suite objects don't know which of their children are ignored or not. I think the underlying implementation will evolve to be based more around tags anyway; fixing this issue will be simpler after that.

For reference, this is the failing test:

describe("a suite with only ignored specs", () -> {

  it("should not run aroundAll", () -> {
    ArrayList<String> steps = new ArrayList<>();
    SpectrumHelper.run(() -> {

      aroundAll(block -> {
        steps.add("aroundAll");
        block.run();
      });

      xit("foo", () -> {
      });

    });

    assertThat(steps, is(empty()));
  });

});

afterAll blocks run with incorrect order, and stop on first failure

I noticed this while implementing a proper fix for #40. They should run in the same order as afterEach blocks (reverse declaration order), and all afterAll blocks should always run, regardless of whether there are errors in other afterAll blocks.

I already have a fix, which is part of another PR.

Cucumber-style code generator from feature files

The GherkinSyntax capability of Spectrum makes it a substitute for

  • runtime parsing of feature files
  • "glue code" which binds the Gherkin syntax to runtime code

In Spectrum, both of the above are expressed in the initialisation of a test class in a single place.

In CucumberJVM, the "glue code" (e.g. @Given("there are cukes") public void there_are_cukes() {}) is not hand-written, it's generated. This is how you bind code to the feature files and this is how it's easy for testers or BA's to update feature files and for them to be used in tests.

While Spectrum can in a lot of situations be simply easier and neater than using Cucumber, there's one big difference in the process. With Spectrum as it stands a developer has to manually convert a Gherkin spec into the Java code and if the tester/BA were to change the original Spec, nobody would necessarily notice and it would be a manual conversion if they were.

Proposal: provide a means of scanning a source of .feature files and mapping them to the respective test classes and spec definitions within those classes in Spectrum. This would allow:

  • start with feature file and generate skeleton of test
  • scan that the tests match the feature files
  • provide diffs between tests and feature files, proposing the code change to put the skeleton in

In Cucumber, this check runs before every test, and can be set to make the test fail, or just be a warning. The output of cucumber might be:

Missing steps. Try adding:

@Given("there is a cuke")
public void there_is_a_cuke() {
    throws new PendingException();
}

We could do something similar in Spectrum. There could be a feature file scanner which might output:

Warning: missing test class for feature file "MyFeature.feature" with:

    Feature: My feature
    Scenario: My scenario

    Given a cuke
    When eat cukes
    Then no cukes

Try adding:

@RunWith(Spectrum.class)
public class MyFeature {{
    feature("My feature", () -> {
        scenario("My scenario", () -> {
            given("a cuke", () -> {
                pending();
            });
            when("eat", () -> {
                pending();
            });
            then("no cukes", () -> {
                pending();
            });
        });
    });
}}

Or maybe it could output diffs

Warning class "SomeFeatureTest" has extra:

    feature("some deleted feature" ...)

I think this would not be difficult to produce, and would make Spectrum a serious contender for a CucumberJVM replacement.

Consolidation refactoring - before/around/let - the grand unified theory of hooks

There are several constructs within Spectrum which are essentially hooks that run before/after or both before and after execution blocks. The let supplier is refreshed on a test by test basis. The beforeAll is run around a set of items. The beforeEach runs before individual specs.

All of these could be consolidated into a hook plugin architecture. We keep the user interface and just modify the internals so that there's a chain of hooks at each level of the hierarchy. Those hooks may or may not be connected with Supplier references that the test definition has kept. The hooks may or may not run at the level of a suite or down at the individual "test". For this to work, the concept of Atomic already incubating on #56 will be useful.

The hooks will also be exposed with functions hook and localHook or similar, for ANY external plugin to be possible. This means the JUnit rules implementation of #56 can become relegated to a plugin.

Hopefully a lot of code will go away in Suite with this change, as there'll only really be one implementation of lots of things suite does as a chain.

Paramaterized specs

As a follow up to the scenarioOutline idea, I'd like to make it simple and declarative to have parameterized tests in Spectrum.

Features:

  • any type of spec/suite should be easily parameterized without writing boilerplate
  • the explicit gherkin construct of scenarioOutline should be supported directly

Parameterization for spec-style tests?

#58 focuses specifically on parameterization for Gherkin-style tests (scenarioOutline). It would be nice to provide a cleaner parameterized test API for Spec-style tests too.

There is no obvious consensus among spec-style test runners as to the "canonical" way to do this. RSpec, Jasmine, and Mocha, for example, do not have any special support for parameterization out of the box. There are a few data points from the community, though:

We can use this thread to discuss the best approach for Spectrum, either taking ideas from above or going in a different direction.

Create official documentation page/site

To include:

  • More examples and detail than is relevant in README
  • IDE snippets for auto-completing describe, it, etc.
  • Suggested usage with tools like Mockito
  • Patterns for writing clean specs (helper functions, etc.)
  • Links to binaries, GitHub, repo, and contributor list

For simplicity, I'll probably set this up with GitHub Pages and some sort of static HTML generator with a pre-made template.

how to set expected exception

I would like to ask whether it is possible to define expected exceptions for it blocks. Currently I am using try catch to handle this. And I am wondering if this is possible with spectrum?

Support for categories/tagging of Specs

At the moment there is an Rspec like syntax for focusing or ignoring individual nodes within the tree. What would be useful is to enable CI builds that build sections of the tests, or for developers to be able to run WIP tests exclusively while stopping them from running in CI builds.

  • Make it easy to selectively run features and scenarios:
    • This probably means a bigger mechanism than the ignore/focus that's presently done within Spectrum
    • For non-trivial situations, this also probably means allowing for tagging of nodes in the tree
    • If there's tagging, then there needs to be a mechanism for selecting the tag
    • The selection mechanism needs to be available at development time and also at CI time
    • Duplicating Cucumber's tag feature is not a perfect solution, but the solution should be a cousin of that

Allow JUnit rules plugin to support individually annotated tests

In native JUnit, you can do something like this:

@Test
@DirtiesContext(methodMode = DirtiesContext.MethodMode.AFTER_METHOD)
public void willDoSomethingDestructiveToSpring() throws Exception {
    myBean.deleteAll();
}

The @DirtiesContext annotation is a clue to the JUnit runner or @Rule that the test method needs some variation in its treatment.

With #56 it is possible to use different mix-in objects to run JUnit rules differently in different parts of the spec. However, making a single it have different behaviour to another is not possible. This is largely because under the hood all tests appear to relate to a stub @Test public void stub() {} method.

Perhaps it would be possible to fix this by one of two ways:

// provide annotated test methods in the Mixin
public class Mixin {
   // rules stuff here

  @Test
  @InterestngAnnotation("1")
  public void interesting1() {
  }

  @Test
  @InterestngAnnotation("2")
  public void interesting2() {
  }
}

// and then in the spec refer to them

Supplier<Mixin> mixin = applyRules(Mixin.class);
describe("some suite", () -> {
  it("uses the mixin a particular way", with(methodSetup("interesting1"), () -> {
    // test something
  }));
});

An alternative is to provide the method selection when applying the rules:

Supplier<Mixin> mixin = applyRules(Mixin.class, "interesting1");
describe("some suite", () -> {
  it("uses the mixin a particular way", () -> {
    // test something
  });
});

More suggestions needed, I think.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.