givepenny / gherkinspec

True .NET Standard Gherkin test adapter

GherkinSpec

Overview

A lightweight, cross-platform .NET Standard test adapter that discovers Gherkin tests from feature files and executes them. Inspired by the giants of Cucumber, SpecFlow and BDDfy, filling a niche between them for microservice tests that are both lightweight (e.g. BDDfy) yet self-documenting in a natural language (e.g. SpecFlow/Cucumber).

Published package: https://www.nuget.org/packages/GivePenny.GherkinSpec.TestAdapter

Highlights

• Fully cross-platform (.NET Standard)
• No generated code files means cleaner pull requests (i.e. no *.Designer.cs)
• Use with any unit test framework or assertions library
• Supports dependency injection

Getting started

If you're upgrading to a new major version (e.g. 1.x to 2.0) please read the changelog

The best way to pick GherkinSpec up is to see the simple example repository, download the code and just try it out.

If you want to create a new test project from scratch ...

1. Create a new .NET Core project for your tests.
2. Add Nuget package references to GherkinSpec.TestAdapter and Microsoft.NET.Test.Sdk.
3. Add a plain-text file (ending in .feature) to your project, mark it as an Embedded Resource (either in the csproj file, or in Visual Studio's Properties pane)
4. Write your C# steps, tagging the class with a [Steps] attribute and your methods with [Given] or [When] or [Then]
5. Add a reference to a unit test framework of your choice if you would like to perform Assertions, for example MSTest.TestFramework

Steps 1, 2 and 5 can be speeded up by creating your test project using one of these templates in Visual Studio, either for MS Test, NUnit or xUnit. You will still need to add a reference to GherkinSpec.TestAdapter.

Full feature list

• Open source
• .NET CLI compatible (dotnet test)
• Visual Studio / Visual Studio Code compatible
• Azure DevOps compatible (test results reported)
• Supports async/await steps
• Efficiently runs tests in parallel (if they contain async steps) - great for eventually-consistent message-based microservices
• Fully cross-platform (.NET Standard)
• No generated code files means cleaner pull requests (e.g. no *.Designer.cs files)
• Minimal 3rd-party dependencies
• Use with any unit test framework or assertions library (MS Test, NUnit, xUnit)
• Supports dependency injection
• Support for the full Gherkin syntax, including tables and doc-strings
• Easy logging from within steps.
• Support for clear tests of eventually consistent services via [EventuallySucceeds] attribute
• Globalisation - supports specifications written in multiple languages

Gotchas

• Make sure you add the Test SDK package to your test project, again see the csproj for the simple example (e.g. <PackageReference Include="Microsoft.NET.Test.Sdk" Version="15.9.0" />). If you don't then tests won't get detected.
• When creating a new .feature file, make sure that it is added as an Embedded Resource (see the csproj of the simple example - in Visual Studio this can also be set in the Properties pane when the file is selected).

  <ItemGroup>
  	<None Remove="Features/**/*.feature" />
    <EmbeddedResource Include="Features/**/*.feature"/>
  </ItemGroup>

• When using steps from referenced assemblies, make sure that the assembly has been loaded (for example by referencing a type in that assembly: .AddAllStepsClassesAsScoped(typeof(ReferencedAssembly.StepsClass).Assembly)).
• If you keep getting .designer.cs files appear, uninstall (or disable) the SpecFlow Extension for Visual Studio or name your files .gherkin instead of .feature.
  • See the "useful links" section below for a replacement syntax highlighting extension (although the files must end in .feature )
  • After uninstalling the SpecFlow extension, feature/gherkin files can be created just by creating plain text files ending in .feature or .gherkin. Remember to mark them as an Embedded Resource if you haven't set the catch-all in the csproj.
• Steps classes and methods must be public. This is for two reasons: firstly, the classes are accessed from outside the steps assemblies by the Test Runner so it logically makes sense to advertise that. Secondly, code analysis tools can flag up private classes and methods as being unused and recommend that they are removed. Requiring them to be public avoids a requirement for these code analysis warnings to be suppressed.

Complete examples

• Simple example repository
• Feature-rich example repository

Reference guide / concepts

• Feature discovery
• Steps classes
• Before/after hooks
• Dependency injection
• Logging
• Eventually-consistent service support
• Localisation
• Configuration reference

Third-party references and useful links

• Credits and contributors
• Full Gherkin syntax
• Visual Studio Code Gherkin syntax highlighting and templating extensions:
  • Steve Purves' extension
  • Interesting review of 3 extensions
• Visual Studio Gherkin syntax highlighting:
  • https://marketplace.visualstudio.com/items?itemName=MadsKristensen.SyntaxHighlightingPack

gherkinspec's Issues

Please enhance for other hooks attributes

Would be good if other hooks attributes like BeforeFeature, AfterFeature, BeforeScenario are also there. can be used to extract the scenario level info/metadata before any scenario starts.
These data can be used in case of custom reporting

Visual Studio 2019 Test explorer show gerkin tests under <Empty Namespace>/<Empty Class> lines

Used:
Visual Studio 2019
.Net Core 3.1
xUnit
GivePenny.GherkinSpec.TestAdapter 2.1.7

Result:

Add support for non-english feature files

I tried using GherkinSpec in a project with Norwegian feature files, but I am unable to run the tests as I only get an exception saying:

Skipping assembly because an exception was thrown: GherkinSpec.Model.Parsing.InvalidGherkinSyntaxException: Expected the first line of a feature file to start with 'Feature:' or a tag.

This is an essential feature for my team as our acceptance criterias are all written in Norwegian and we do not wish things to possibly get "lost in translation" when being converted to tests.
You can find a set of translations from english to other languages here: https://github.com/cucumber/cucumber/blob/master/gherkin/gherkin-languages.json

Support clearer tests for eventually consistent services

@carl-hartshorn this is to capture and iterate on your idea before starting a PR.

Proposal:

1. Allow [Then] steps to be marked as eventually consistent, causing the individual step to be automatically retried if it fails. Do this by adding a new attribute: [EventuallySucceeds(TimeSpan within, TimeSpan retryInterval)]. Don't allow that attribute on anything other than a Then step.
2. Add a docs markdown file warning against using this capability to make non-deterministic tests pass rather than fixing the underlying cause of the determinism problem. Give an example of a Then checking that an event has been received and showing the EventuallySucceeds attribute.
3. Allow scenarios to be tagged @eventuallyConsistent(within=00:00:20,retryInterval=00:00:05). If any Then in the scenario fails, this causes all steps from (and including) the last encountered When step to be retried, not the whole scenario (Givens would generate new ids/data which is undesirable). The use case is of a When that performs a web API call, but a subsequent Then examines the response detail and finds that the service's data is not yet consistent and needs to wait then retry from the web API call on again.

@carl-hartshorn up for discussion - do we still think this is a good idea (it won't breed bad quality tests if misused?) and is this how we imagine we would want to use it?

Support full Gherkin spec

As per https://docs.cucumber.io/gherkin/reference/

GherkinSpec.Model does not exist in the namespace 'GherkinSpec'

Currently receive the following error when trying to reference GherkinSpec.Model.DataTable:
The type or namespace name 'Model' does not exist in the namespace 'GherkinSpec' (are you missing an assembly reference?)

Currently GherkinSpec.Model.dll is packaged to the build folder, does this need to be lib folder?

<_PackageFiles Include="$(OutputPath)\GherkinSpec.Model.dll">
      <BuildAction>None</BuildAction>
      <PackagePath>build/netstandard2.0</PackagePath>
    </_PackageFiles>

Visual Studio Text Explorer - Duplicate scenario names are not handled

When a scenario name is duplicated (either within or across feature files) the Visual Studio Test Explorer only picks up the first scenario it encounters.

Encountered in Visual Studio (2019 Community - Version 16.2.5).

The problem isn't manifested when running dotnet test - all scenarios with the same name run in this case.

Although scenarios with the same name may be confusing and not something we want to support, the present behaviour is also potentially confusing. This is particularly so when a scenario with a duplicated name fails. Running the tests using the VS Test Explorer appears to show that all tests are passing. Running via dotnet test or dotnet vstest (potentially on a build agent) shows the test failure.

The issue has been reproduced in this branch: https://github.com/GivePenny/GherkinSpec.SimpleExample/tree/duplicate-scenario-names

I'm happy to work on a PR to resolve this. @DaveAurionix what are your thoughts on a solution? We could fail when detecting duplicate scenario names, although the experience when this happens (for example when a feature file is not syntactically correct) isn't great. An alternative might be to append the duplicated scenario names with a counter of some sort.

IServicesCollection.AddAllStepsClassesAsScoped() extension method

… is needed for simpler test setup. Originally omitted due to not wanting to couple the TestModel assembly to a DI framework but we think adding a ServiceCollectionExtensions class to the TestAdapter will solve the problem (the TestAdapter is already coupled to the Microsoft.Extensions.DependencyInjection.Abstractions package).

Add support for translations with multiple possible phrases for keywords

Some translations have multiple possible phrases that could be used for some keywords. The localisation capability in GherkinSpec should be extended to support this.

One possible approach to retain the resx format is to store keyword translations as semi-colon-delimited lists.

Another approach might be to abandon resx and use a custom JSON format as Cucumber does.

Dynamic scenario discovery

Have you considered using source generators instead of using embedded resources for the feature files? This might make it so tests could be dynamically discovered since source generators run at design time as well as build.

Some other possible features that could be enabled by using source generators:

• The #line pragma could be used to make it so IDE's and debuggers know which file and line the generated code corresponds to. This would make it so that you shouldn't have to create custom IDE extensions.
• The generated test method could reference the actual step method so that the codelense reference counts and Find All References on the step definitions would just work.
• The #error and #warning pragmas could be used when a step definition could not be found to provide custom error/warning messages. This would provide design-time errors/warnings to without the need to build the project.

Need a Visual Studio Extension to allow navigating to C# step definitions from feature files

Very original Name xD

Dude, you really couldn't come up with more original name?
Single search would have saved the mankind. 🗡
That is just ridiculous. 🥇

My project: GherkinSpec
My nugets:
GherkinSpec.Core GherkinSpec.NUnit GherkinSpec.MsTest GherkinSpec.XUnit
Both initiatives (yours, mine) are dotnet...

I am not even mad. 😅
Just tell me how it happened.

Rider Test Explorer Support

At present, Rider's test explorer appears to display all scenarios as a flat list (irrespective of the test grouping selected).

Furthermore, double clicking a test does not navigate to the scenario in the feature file.

This issue proposes that we investigate whether we can make improvements to this Rider support (while maintaining the good level of integration with Visual Studio's test explorer).

Support Rule feature (introduced in Gherkin 6)

Cucumber docs have recently been updated to mention a new Rule keyword introduced in Gherkin 6. We should consider adding support for this even if just for spec-compatibility (not entirely convinced of the value of the feature yet).

Could improve matching based on number of parameters

Current matching doesn't take into account number of parameters, so in the following example function Test1 is a match for both gherkin When statements:

When I submit <value>
When I submit <value1> and <value2>

[When I submit (.*)]
public void Test1(string value)

[When I submit (.*) and (.*)]
public void Test2(string value1, string value2)

This is just a suggestion, as the workaround is to use less greedy regex and more specific step definitions.
But would it be worth matching based on number of parameters so that Test2 is preferred to Test1 for the second when statement ("When I submit and ")?

Failure to parse feature file in which all scenarios have tags

The following error is thrown when attempting to parse a feature file in which all scenarios have tags:

[18/12/2018 16:44:31 Warning] Skipping assembly because an exception was thrown: GherkinSpec.Model.Parsing.InvalidGherkinSyntaxException: Expected a step (Given, When, Then, And, But), a Scenario or a Scenario Outline, but found '@ignore'.
   at GherkinSpec.Model.Parsing.ScenarioParser.ParseScenario(LineReader reader, IEnumerable`1 tags)
   at GherkinSpec.Model.Parsing.Parser.Parse(LineReader reader)
   at GherkinSpec.Model.Parsing.Parser.Parse(String featureText)
   at GherkinSpec.TestAdapter.TestDiscoverer.DiscoverTests(String source, Assembly assembly, IMessageLogger logger)
   at GherkinSpec.TestAdapter.TestDiscoverer.DiscoverTests(String source, IMessageLogger logger)

A simple feature file with which this can be reproduced is as follows:

Feature: Feature file in which all scenarios have tags should be parsable

@ignore
Scenario: First scenario has a tag
	Given all scenarios in the feature file have tags
	When GherkinSpec parses the feature file
	Then parsing should succeed

@ignore
Scenario: Second scenario also has a tag
	Given all scenarios in the feature file have tags
	When GherkinSpec parses the feature file
	Then parsing should succeed

Support optional configuration as attribute parameters for `EventuallySucceeds` & `MustNotEventuallyFail`

This issue proposes supporting optional configuration of MaximumAttempts & DelayBetweenAttempts on EventuallySucceedsAttribute and MustNotEventuallyFailAttribute, e.g.:

[EventuallySucceeds(MaximumAttempts = 9, DelayBetweenAttemptsSeconds = 10)]
public void TheStepDefinition()
{
...

When these optional values are not provided, I propose that we use the existing TestRunContext.EventualSuccess values. This would continue to allow for project-wide defaults to be set, but for specific steps to override with values relevant to those steps if required.

Jetbrains Rider : Steps not detected

Hi,

Thanks for your great plugin.
I'm currently using Rider as development IDE, and seems that the test are not discovered by the runner (I test with MSTest and Xunit with default settings).

I don't know if is a bug or a lack of the IDE. So I'm just asking if it should work with Rider ?

AndAttribute and ButAttribute are not implemented

Hi Again,

On using the library in a more complexe Scenario, I do not found the AndAttribute and ButAttribute.

If we follow Gherkin guidelines in cucumber website, And and But should exist as attribute.

I tried to add the attribute with the following tests, but I do not found where the tests are localized. This is a quite simple changes for someone that already knows the project, so I prefer create this issue :)

Add Spanish, French and German translations

As per title. Needs multi-keyword support in the LineReader class.