lightbdd / lightbdd

BDD framework allowing to create easy to read and maintain tests.

LightBDD

The Lightweight Behavior Driven Development test framework

Category	Badge
Build
Chat (gitter)
LightBDD.NUnit3
LightBDD.XUnit2
LightBDD.MsTest3
LightBDD.Fixie3
LightBDD.Autofac
LightBDD.Extensions.DependencyInjection

See latest release details on What Is New wiki page!

Project description

LightBDD is a behaviour-driven development test framework offering ability to write tests that are easy to read, easy to track during execution and summarize in user friendly report, while allowing developers to use all of the standard development tools to maintain them.

Features

• Native support for refactoring, code analysis (like finding unused methods), test running and all features that Visual Studio / Intellisense / Resharper offer during code development,
• Easy to read scenario definitions,
• Scenario steps execution tracking and time measurement, usable during longer test execution,
• Support for parameterized steps with smart rules of inserting argument values to formatted step name,
• Support for advanced in-line and tabular verifiable parameters,
• Support for contextual scenario execution where steps shares dedicated context,
• Support for DI containers,
• Feature report generation in HTML, XML or Plain text format,
• In-code LightBDD configuration allowing to customize all LightBDD features,
• Productivity extensions for VisualStudio with Feature Class Templates, Project Templates and Code Snippets,
• Integration with NUnit, xUnit, MsTest.TestFramework and Fixie frameworks,
• Async scenario and steps execution support,
• Cross-platform support (.NET 5+ / .NET Framework / .NET Standard / .NET Core / UWP).

Tests structure and conventions

LightBDD runs on top of NUnit, xUnit, MsTest.TestFramework and Fixie, allowing to leverage the well known test frameworks and their features in writing the behavioral style scenarios, which makes it very easy to learn, adapt and use.

To learn more, please see LightBDD wiki page, or jump straight to:

• Quick Start and followup documentation pages,
• What Is New section with the newest features introduced in the library,
• Examples demonstrating features and integrations of LightBDD,
• Tutorials with complete sample projects using LightBDD features.

Example

[FeatureDescription(
@"In order to access personal data
As an user
I want to login into system")] //feature description
[Label("Story-1")]
public partial class Login_feature //feature name
{
   [Scenario]
   [Label("Ticket-1")]
   public void Successful_login() //scenario name
   {
      Runner.RunScenario(

         Given_the_user_is_about_to_login, //steps
         Given_the_user_entered_valid_login,
         Given_the_user_entered_valid_password,
         When_the_user_clicks_login_button,
         Then_the_login_operation_should_be_successful,
         Then_a_welcome_message_containing_user_name_should_be_returned);
   }
}

[FeatureDescription(
@"In order to pay for products
As a customer
I want to receive invoice for bought items")] //feature description
[Label("Story-2")]
public partial class Invoice_feature //feature name
{
   [Scenario]
   [Label("Ticket-2")]
   public void Receiving_invoice_for_products() //scenario name
   {
      Runner.RunScenario(

         _ => Given_product_is_available_in_product_storage("wooden desk"), //steps
         _ => Given_product_is_available_in_product_storage("wooden shelf"),
         _ => When_customer_buys_product("wooden desk"),
         _ => When_customer_buys_product("wooden shelf"),
         _ => Then_an_invoice_should_be_sent_to_the_customer(),
         _ => Then_the_invoice_should_contain_product_with_price_of_AMOUNT("wooden desk", 62),
         _ => Then_the_invoice_should_contain_product_with_price_of_AMOUNT("wooden shelf", 37));
   }
}

The above example shows feature partial classes containing scenario definitions only, which makes it easier to read.
The Login_feature class uses basic scenario format for defining scenario steps.
The Invoice_feature class uses extended scenario format allowing to use parameterized steps.
LightBDD offers multiple scenario formats and flavors suitable for various use cases - for details, please check Scenario Steps Definition page.

The implementation of step methods is located in other part of the class, in separate file and leverages standard features of the underlying test framwork (such as assert mechanisms) empowered by LightBDD features such as:

• Dependency Injection,
• Inline, Tabular, Tree-hierarchy parameters,
• Expectation Expressions,
• Step comments,
• Scenario and Step Decorators.

Example console output during tests execution:

What LightBDD offers for free is the scenario output provided in Visual Studio/Resharper test screens and/or Console window.
Where possible, the output is provided as the test executes, allowing to track the progress of the scenarios.

FEATURE: [Story-1] Login feature
  In order to access personal data
  As an user
  I want to login into system

SCENARIO: [Ticket-1] Successful login
  STEP 1/6: GIVEN the user is about to login...
  STEP 1/6: GIVEN the user is about to login (Passed after 2ms)
  STEP 2/6: AND the user entered valid login...
  STEP 2/6: AND the user entered valid login (Passed after <1ms)
  STEP 3/6: AND the user entered valid password...
  STEP 3/6: AND the user entered valid password (Passed after <1ms)
  STEP 4/6: WHEN the user clicks login button...
  STEP 4/6: WHEN the user clicks login button (Passed after 125ms)
  STEP 5/6: THEN the login operation should be successful...
  STEP 5/6: THEN the login operation should be successful (Passed after 4ms)
  STEP 6/6: AND a welcome message containing user name should be returned...
  STEP 6/6: AND a welcome message containing user name should be returned (Passed after 9ms)
  SCENARIO RESULT: Passed after 164ms


FEATURE: [Story-2] Invoice feature
  In order to pay for products
  As a customer
  I want to receive invoice for bought items

SCENARIO: [Ticket-4] Receiving invoice for products
  STEP 1/7: GIVEN product "wooden desk" is available in product storage...
  STEP 1/7: GIVEN product "wooden desk" is available in product storage (Passed after 2ms)
  STEP 2/7: AND product "wooden shelf" is available in product storage...
  STEP 2/7: AND product "wooden shelf" is available in product storage (Passed after <1ms)
  STEP 3/7: WHEN customer buys product "wooden desk"...
  STEP 3/7: WHEN customer buys product "wooden desk" (Passed after <1ms)
  STEP 4/7: AND customer buys product "wooden shelf"...
  STEP 4/7: AND customer buys product "wooden shelf" (Passed after <1ms)
  STEP 5/7: THEN an invoice should be sent to the customer...
  STEP 5/7: THEN an invoice should be sent to the customer (Passed after <1ms)
  STEP 6/7: AND the invoice should contain product "wooden desk" with price of "£62.00"...
  STEP 6/7: AND the invoice should contain product "wooden desk" with price of "£62.00" (Passed after <1ms)
  STEP 7/7: AND the invoice should contain product "wooden shelf" with price of "£37.00"...
  STEP 7/7: AND the invoice should contain product "wooden shelf" with price of "£37.00" (Passed after <1ms)
  SCENARIO RESULT: Passed after 30ms

For more information, please visit Test Progress Notification page.

Example HTML report generated after test execution:

When all tests are finished, LightBDD generates the HTML report file, providing the summary and the details of the executed scenarios - the fragment of the report is presented below.

To read more, please see Generating Reports wiki page.

Installation

The LightBDD is available on NuGet:

• PM> Install-Package LightBDD.NUnit3 for package using NUnit 3x
• PM> Install-Package LightBDD.XUnit2 for package using xUnit 2x
• PM> Install-Package LightBDD.MsTest3 for package using MsTest.TestFramework
• PM> Install-Package LightBDD.Fixie3 for package using Fixie 3x

Productivity Extensions for Visual Studio

LightBDD project offers also a VSIX extension for Visual Studio, containing:

• Project Templates for all integrations (nunit/xunit/mstest/fixie),
• Feature Class Templates for all integrations (nunit/xunit/mstest/fixie),
• Code Snippets for creating Scenarios and Composite Steps.

The VSIX extension can be downloaded from: Visual Studio Gallery (Visual Studio 2012 and newer versions supported).

Migrating LightBDD

The current LightBDD version series is 3.X. To find out how to migrate from previous versions, please visit Migrating LightBDD Versions wiki page.

Debugging LightBDD from NuGet packages

LightBDD provides debug symbols helping with diagnostics - please check Debugging LightBDD Scenarios for details.

More information about LightBDD

If you are interested about background of LightBDD creation or getting insight into what is driving it's evolution, please feel free to take a look at my blog posts.

Community Projects

• LightBdd.TabularAttributes - Designed to be used with LightBdd Tabular Parameters to allow for specifying input and output data for tests via attributes, in a truth table format.
• LightBDD.ScenarioSync - ScenarioSync for Azure DevOps, integrates the BDD process with Azure DevOps by connecting and synchronizing the LightBDD scenarios with Test Cases.

lightbdd's Issues

Extend runner to support fluent API for scenario definition

Progress

• Implement fluent interfaces for defining scenarios in basic, extended and contextual way
• Refactor code
• Update examples

Description

Allow to define scenarios in fluent way

await Runner.NewScenario()
    .AddSteps(
        _ => Given_XXX()
        _ => When_XXX()
        _ => Then_XXX(55))
    .AddAsyncSteps(
        _ => Then_YYY(param),
        _ => Then_ZZZ(param))
    .RunAsync();

Explore usability of LightBDD for UWP platform

Release LightBDD.MsTest with package MSTest.TestFramework >=1.1.11

Document various use cases for LightBDD

Description

Add wiki page with various use cases for LightBDD to present how common problems can be addressed with LightBDD.

Sample use cases:

• simple acceptance testing (given/when/then, labels, FeatureDescription),
• working with unfinished features (bypassing steps),
• working with common, repeatable steps (parmeterized steps, extension methods),
• working with algorithms (parameterized scenarios and steps),
• working with end to end tests (composite steps, extension methods),
  ... and more

Progress

• Feature is implemented,
• Ensured backward-compatibility,
• Ensured good debugging experience

How to support multiple parameterized tests

Description

I have a legacy system and I'm adding tests prior to some changes. One scenario requires testing an method with ten input parameters, which generate twenty properties which must be tested. There are 54 sets of input parameters to be tested. How would you approach this with your framework?

Add issue and PR templates

Add issue and PR templates to simplify development process and make it more structured

Update example projects with all features

Description

Revisit example projects and provide samples reflecting all LightBDD features and link them to documentation

Progress

• Feature is implemented,
• Ensured backward-compatibility,
• Ensured good debugging experience

Ability to add debug logs in test report

It would be great, if we will be able to print debug logs in HTML and XML report to investigate the problems in tests. E.g. using log4net I print information about request repsonse. It appears in console, but not in HTML report

Update VSIX templates for VS2017

• Update templates to support VS2017.
• Add item templates for new csproj format.
  Add project templates for new csproj format

How to use this with async?

If I'm testing something that is async, the step has to return a Task. There seems to be no overload for handling this?

I'm using Nunit

Update VSIX extension with project templates

Description

Add new csproj project templates for all integrations (nunit2, nunit3, xunit2, mstest2).
The project template should have minimal set of code needed to launch tests:

• dependency on LightBDD.XXX package,
• LightBddScope,
• sample feature with sample scenario

Progress

• Feature is implemented,
• Ensured backward-compatibility,
• Ensured good debugging experience

Extend support for .NET 45

Extend support for .NET 45 if possible

Introduce Analyzers

• Empty Scenario without Runner.RunXXX
• Framework specific ignore instead of IgnoreScenarioAttribute
• Step methods mixed with scenarios instead of being in separate file
• Step parameter not present in step name

Make parameter formatting to differentiate between null and empty values

Currently, if step/scenario method has a parameter with is either null or renders empty (ToString() returns ""), it is rendered in reports and progress in the same way.

The null values should render differently, and such behaviour should be applied after custom formatters, so no matter if formatted is specified or not, if final value is null, it should be correctly presented.

The collection formatters should be revisit to ensure that collections shows follows that patter as well.

NUnit3. Tests with ignore annotation doesn't appear in html report

Here is an example of test:

        [Ignore("false positive")]
        [Scenario]
        public void test(string version)
        {
            lat = Config.GetConfig ().testCities.msk.lat;
            lng = Config.GetConfig ().testCities.msk.lng;
            cinemaId = Config.GetConfig ().testCinemas.additionalgoodsCinema.id;
            
            Runner.RunScenario(
                _ => Given_api_version(version),
                _ => Given_lat_and_lng(lat, lng),
                _ => When_we_trying_post_order_to_3D_movie(cinemaId),
                _ => Then_we_get_statusCode_OK(response),
        }

When launch this test, in Nunit Console output this tests are marked as ignored
But in LightBDD output this tests doesn't appear at all

lightBDD.MsTest2

I try to run lightBdd unit tests using MSTestFramework and MSTest.TestAdapter in VS2013 and VS 2015, but at the end VS does not find these MSTest unit test. With Nunit3 all is working fine.
I have the impression that although I have not installed any Nunit packages the VS test environment is looking ofr NUnit runner
The same problem I have using vstes.console runner. It does not find any tests of MSTest v2.
One of the problems is that it seems VS is using Microsoft.VisualStudio.QualityTools.UnitTestFramework and with it the tests are discoverd, but lightBDD does not work with it, because it uses Microsoft.VisualStudio.TestPlatform.TestFramework andMicrosoft.VisualStudio.TestPlatform.TestFramework.Extensions, but these packages are not discovered than.
Any solution available to solve this problem
Best regards

thomas

Please publish a symbols package along .nuget

With NuGet, you can automatically build and publish a symbol package to accompany your .nuget file. This allows for a nicer debugging experience, so steping out of a feature step and into the next one would not show the dissassebly windows in Visual Studio.

Feature Request: Support for logging arbitrary text which will appear in the reports

Right now there isn't an easy way to get my test context output to the test reports. It would be nice if there was some option to output it easily. Ideally I would like to specify a class that represents my test context and LightBdd includes the representation after each test run.

Runner passing a test that should fail

This is my Feature code:

using LightBDD.Framework;
using LightBDD.Framework.Scenarios.Basic;
using LightBDD.NUnit3;
using NUnit.Framework;


namespace TSC.IK.Test.Behaviors
{
  [TestFixture]
  [FeatureDescription(
    @"  In order to start deploying Intranet Kickstart components
        As a developer
        I want to provision a demo site"
  )]
  [Label("DemoSite-Story")]
  public partial class DemoSite_Feature
  {
    public void Successful_login_to_O365_tenant()
    {
      Runner.RunScenario(
        Given_I_have_a_test_tenant,
        And_I_am_logged_in_as_a_SharePoint_Administrator,
        When_I_create_a_new_site_collection,
        Then_I_should_have_a_new_site_collection_created
      );
    }
  }
}

And this is the steps code:

public partial class DemoSite_Feature : FeatureFixture
  {


    public void Given_I_have_a_test_tenant()
    {
      throw new NotImplementedException();
    }
    public void And_I_am_logged_in_as_a_SharePoint_Administrator()
    {
      throw new NotImplementedException();
    }
    public void When_I_create_a_new_site_collection()
    {
      throw new NotImplementedException();
    }
    public void Then_I_should_have_a_new_site_collection_created()
    {
      throw new NotImplementedException();
    }
  }

However when the test is run, it reports as "Passed". It is using NUnit3. (See screenshot attached)

It should report as "failed" given that nothing has yet been implemented to satisfy the behavior.

Restore support for NUnit 2x to ease migration for LightBDD 2x

Introduce snippets for LightBDD scenarios and steps

Description

Explore an ability to add snippets with:

• VisualStudio itself,
• Resharper,
• VS Code
  and choose the best method to create them,

Explore an ability how to distribute snippets (VSIX, nuget packages).

Explore ability how to edit snippets

Provide snippets for:

• creating basic scenarios,
• creating extended scenarios,
• creating composite steps,
• adding basic step (typed text with spaces, to method name with underscores)
• add extended step (like basic but also with parameters)

Progress

• Feature is implemented,
• Ensured backward-compatibility,
• Ensured good debugging experience

LightBDD.NUnit3 - display immediate progress of executed tests

Currently LightBDD.NUnit3 uses TestContext.Out to render scenario progress, which effects with whole test output being displayed at the end of test case.

The integration should be changed to use:

• TestContext.Progress for reporting execution progress (it will be displayed immediately),
• ParallelProgressNotifierProvider instead of DefaultProgressNotifier to render progress in more readable manner for parallel execution.

---

Update after lack of nunit support in nunit/nunit#2177:

• TestContext.Progress will be used in conjunction with ParallelProgressNotifierProvider to display immediate progress,
• TestContext.Out will be still used in conjunction with DefaultProgressNotifier to display execution summary in test output windows (VS Test Explorer and Resharper) after they are finished,
• test progress notification configuration will be extended to allow change the default behavior in more manageable manner

Progress:

• Default progress notifiers are reconfigured to report immediate progress and summary,
• Progress notification configuration methods are extended to support easier customization,
• Example test scripts are updated to suggest the best configuration of test runners,
• Documentation is updated,
• LightBDD.NUnit3 notifiers are updated,
• LightBDD.XUnit2 notifiers are updated.

Configuring parameter default formatting

Progress

• Implement parameter formatting service,
• Implement default formatters,
• Implement formatter configuration,
• Ensure debugging experience is ok,
• Ensure backward compatibility

Idea

Currently, step method parameters formatting is controlled in following way:

1. custom formatting will be used if parameter is decorated with attribute deriving from ParameterFormatterAttribute,
2. ToString() will be used otherwise.

The problem is that types such as array like string[] does not produce meaningful text when its ToString() is used, so they always have to be decorated with [FormatCollection] attribute to format nicely.

The change would be to provide a default list of formattings, depending on parameter type, that would:

• be configurable on lightbdd startup, to allow user customizations,
• provide default formatting for types like collections.

With the change, the formatting method would be determined in following way:

1. custom formatting will be used if parameter is decorated with attribute deriving from ParameterFormatterAttribute
2. default formatting will be used if it is specified for given parameter type,
3. ToString() will be used otherwise.

Update nuget packages icon

Description

Update all nuget packages icon to higher resolution so that it would appear bigger on nuget.org.
Right now a small icon is visible: https://www.nuget.org/packages/LightBDD.NUnit3/

Ability to specify verifiable parameters

Progress

• Spike possible solution
• Implement Verfiable<T>
• Provide meaningful exception on expectation violation
• Update reports to visually represent expected/actual value comparison (add colors to HTML report)
• Handle custom formatting of arguments
• Ensure test coverage
• Implement Expect.To semantic
• Add support for custom expectation implementations
• Add LessThan/GreaterThan, wildcard/regex match expectations for values
• Add Contains, All, Any, Equivalent expectations for collection values,
• Add And, Or, Not, AllTrue, AnyTrue expectations for complex predicates,
• Rename Expected<T> to Verifiable<T>
• Modify MatchRegex*() to always include ^$
• Revisit all expectations and rename them to match pattern Expect.To.XXX
• Add ToVerifiable method

Problem

Lets assume we have a scenario:

Runner.RunScenario(
  _ => Given_an_user_with_id_name_surname_and_email(124, "Joe", "Johnson","[email protected]"),
  _ => When_I_request_user_details_for_id(124),
  _ => Then_I_should_receive_user_with_id_name_surname_and_email(124, "Joe", "Johnson","[email protected]")
);

Currently the implementation of Then_ step will have to look as follows:

void Then_I_should_receive_user_with_id_name_surname_and_email(int id, string name, string surname, string email)
{
   Assert.That(_receivedUser.Id, Is.EqualTo(id), "Wrong ID");
   Assert.That(_receivedUser.Name, Is.EqualTo(name), "Wrong name");
   Assert.That(_receivedUser.Surname, Is.EqualTo(surname), "Wrong surname");
   Assert.That(_receivedUser.Email, Is.EqualTo(email), "Wrong email");
}

The assertions are pretty verbose here and any failures requires user to read the scenario failure reason to figure out what field is wrong.

Initial Idea

The input parameters should be extended with an ability to specify expected value and either:

• actual value with comparison method,
• comparison method with failure reason (to be investigated if that makes sense).

The framework should be extended to perform parameter value validation after step finish and:

• render step name with inlined expected != actual values and different color (red) in html report,
• render step name with inlined expected = actual values and different color (green) in html report,
• failing step if any parameter has not matching values.

The implementation could be like:

void Then_I_should_receive_user_with_id_name_surname_and_email(ExpectedValue<int> id, ExpectedValue<string> name, ExpectedValue<string> surname, ExpectedValue<string> email)
{
   id.SetActual(_receivedUser.Id);
   name.SetActual(_receivedUser.Name);
   surname.SetActual(_receivedUser.Surname);
   email.SetActual(_receivedUser.Email);
}

where:

• there should be implicit conversion from T to ExpectedValue<T>,
• any formatting attributes applicable to T should be valid for ExpectedValue<T>,
• SetActual should accept T value
• passing constant on step call should still resolve to constant step parameter

An alternative could be:

StepResult Then_I_should_receive_user_with_id_name_surname_and_email(int id, string name, string surname, string email)
{
   return new StepResult()
      .AddParameter(nameof(id), id, _receivedUser.Id)
      .AddParameter(nameof(name), name, _receivedUser.Name)
      .AddParameter(nameof(surname), surname, _receivedUser.Surname)
      .AddParameter(nameof(email), email, _receivedUser.Email);
}

The solution should allow to add more verifiable classes such as ExpectedGreatherThan, ExpectedLessThan, ExpectedLike etc..

Future (epic)

It is worth to consider that in future there may be similar method allowing steps like:

void Then_I_should_receive_user_with_id_name_surname_and_email(Record<User> u)
{
   Assert.That( /* ... */ );
}

where Record<T> could work as T properties are specified explicitly, i.e. render properties separately in text...

void Then_I_should_receive_user_with_id_name_surname_and_email(ExpectedRecord<User> u)
{
   u.SetActual(_receivedUser);
}

where ExpectedRecord<T> could be a version of Record<T> with ability to validate each property separately...

Finally, it could be possible to specify a collection of Records that could be rendered in tabular way, but the idea how to exactly achieve it is open (alternative to parameter of table type, could be a collection of steps with rows/records that would be rendered together as a table with columns separating parameters from text)

Story definition

After spike, I decided to got with Verifiable<T> parameter approach, which is more explicit in use, so easier to understand.

Features:

• Verifiable<T> type that if used as step method parameter(s), LightBDD will ensure that parameter has been verified in the step, and fail the step if verification failed,
• Evaluation of all verifiable parameters to collect all failures for the step,
• Expect.To semantic allowing to define non-trivial expectations,
• Inline display of met / unmet expectations in the step name (colored for HTML report),
• Implicit conversion of value to Expect.To.Equal(value) to make easier usage for simple equality cases.

Example usage:

void Then_I_should_receive_user_with_id_name_surname_and_email(Verifiable<int> id, Verifiable<string> name, Verifiable<string> surname, Verifiable<string> email)
{
   id.SetActual(_receivedUser.Id);
   name.SetActual(_receivedUser.Name);
   surname.SetActual(_receivedUser.Surname);
   email.SetActual(_receivedUser.Email);
}

...
_ => Then_I_should_receive_user_with_id_name_surname_and_email(
            Expect.To.GreaterThan(0),
            "John",
            "Smith",
            Expect.To.MatchRegexIgnoreCase("[a-zA-Z0-9]+@mymail\\.com"))

Future complex verifiable parameters

While this story focus on verifiable parameter value (that either can be OK or not), there is a #70 story for using this concept for tabular parameters.
In future, the similar concept might be used to implement object-tree parameters allowing easier verification json responses etc..

Extend step methods with attributes allowing multiple step failures

Progress

• Implement MultiAssertAttribute
• Implement detailed test scenarios
• Ensure changes are backward-compatible
• Extend scenario/step execution extensions to allow configuring exception handling for sub-steps
• Update examples

Description

Currently the step execution finish on first failure.
It should be possible to mark scenario or composite step with attribute telling that all sub-steps should be executed irrespectively of their execution status and report any failures after all sub-step are executed.

Such feature would be useful if given scenario has multiple independent Then_ steps and it is desired to see the outcome of all of them.

Investigate possibility of running multiple lightbdd test assemblies from one process

Right now the LightBddScopeAttribute uses singleton to initialize the scope.
It means that if multiple test assemblies will be executed from one process, the other assembly will fail with the exception: FeatureCoordinator of XXX type is already installed

An investigation can be done to check if that could be changed to async local context and if it would be then possible to execute multiple assemblies at the same time.

This change may make lightbdd working better with NCrunch.

Explore autogenerating documentation of LightBDD features

Description

At this moment, the documentation and the description of all behaviours has to be manually written in wiki and kept in sync with code, that is getting more and more difficult task with growing project.

This ticket is to investigate a possibility to generate the documentation from code in conjunction with acceptance/spec tests describing behaviour of the feature.

Concepts:

• classes describing the feature could be annotated with attribute referring to the feature,
• feature acceptance/spec test class could have:
  • attributes or intellisense description describing the feature,
  • set of tests having a sample code and the expected behaviour,
• tool to generate the documentation out of the tests and metadata, that would be synced with wiki page on build or release

Progress

• Feature is implemented,
• Ensured backward-compatibility,
• Ensured good debugging experience

Extend fluent step builder to support repeated steps

Progress

• Implement RepeatStep with no context,
• Implement RepeatStep with context,
• Implement RepeatAsyncStep,
• Ensure debugging experience is ok,
• Ensure steps are evaluated upon execution (lazy)

Idea

Assuming collection of input values:

var expectations = new [] { 
    (x: 5, y: 7, sum: 12),
    (x: 2, y: 3, sum: 5),
    (x: -5, y: 2, sum: -3)
};

It should be possible to define a composite step with sub-step per entry:

return CompositeStep.DefineNew()
  .AddSteps(_ => Given_a_calculator())
  .RepeatStep(expectations, a => Then_sum_of_X_and_Y_should_be_VALUE(a.x, a.y, a.sum))
  .Build();

It should be possible to define a scenario with sub-step per entry:

Runner.NewScenario()
  .AddSteps(_ => Given_a_calculator())
  .RepeatStep(expectations, a => Then_sum_of_X_and_Y_should_be_VALUE(a.x, a.y, a.sum))
  .Run();

Note: there would be a complication of how to pass additional context. One option is to specify (ctx, arg) parameter, while other is to make a special parameter x with x.Context and x.Value

Benefits

1. Simplified usage
   syntactic sugar for foreach(var arg in expectations) builder.AddSteps(_ => Then_sum_of_X_and_Y_should_be_VALUE(arg.x, arg.y, arg.sum)
2. A potential, additional optimization can be made to ensure that arg accessors would be treated as constants

Really unclear how to update 1.x to 2.x

I'm trying to upgrade my project right now and am very confused. I've read this article on the wiki but it seems to offer little by way of explanation or examples.

My big confusion is what happened to the old Runner property. I note that we no longer use the FeatureFixture base class, but are now using partial classes but I see no mention of what the other "side" of the partial is supposed to be, nor any mention of this in the doc.

Also, there seems to not be a Scenario attribute...

Implement HTML progress notifier allowing to track execution progress with interactive html page

Idea:

1. progress being saved to html file in incremental manner
2. html page allowing to refresh itself to update progress
3. support for parallel test execution
4. Consider creation of the test run aggregation service offering trace of running tests, historical stats and reports.

Debugging experience

Ensure that LightBDD scenarios are easy for debug (debugger steps into the code that is meaningful for end-users)

NUnit 3.0

Hi i cannot use NUnit 3.0 Can you upgrade it?

Regards

Add DI container support for contextual runners

Progress

• Feature is implemented,
• Ensured backward-compatibility,
• Ensured good debugging experience.

Description

At this moment it is possible to specify Runner.WithContext<MyContext>() to use contextful runner with context having parameterless constructor. If context has dependencies, it has to be manually instantiated with code like Runner.WithContext(new MyContext(dependency)), and all dependencies has to be resolved manually in code. Finally, the contexts implementing IDisposable interfaces are not disposed after usage.

Initial Idea

• The ˋRunner.WithContext()ˋ should be extended to allow instantiating types with any constructor, using DI container to instantiate the context.
• The DI container should have sane defaults, such as:
  • working without any 3rd-party DI container integrations,
  • resolving dependencies to the widest constructor, without need of explicit registrations,
  • instantiate all types per test,
• The DI container should be configurable with LightBddConfiguration, allowing to:
  • choose a 3rd-party implementation if desired (like Autofac),
  • configure choosen implementation,
• The default DI container implementation should allow to:
  • customize registration of type (custom instantiation with ability to resolve dependency),
  • specify if instance is created per test or shared (optional requirement),
• The runner should be updated to control lifespan of the contexts (dispose if needed)

Story definition

After spike, I have decided to add DI container support to LightBDD with focus on supporting proper, mature DIs but also provide a very basic DI in LightBDD itself.

Features:

• LightBDDConfiguration.DependencyContainerConfiguration() configuration allowing to specify DI container type and configure it,
• LightBDD.Autofac project offering integration with Autofac,
• Basic DI container implementation (enabled by default), supporting:
  • container scoping,
  • instance registration,
  • constructor injections,
  • resource disposals,
• Modify WithContext<TContext>() extension methods to use DI containers and lifting where TContext: new() constraint,
• Add StepExecution.Current.GetScenarioDependencyResolver() extensions to allow retrieving DI resolver during scenario execution,
• Add ResourcePool<T> type for handling resource pools (DI friendly),
• Expose DI containers for scenario/step decorators.

Timings

I was thinking of adding an option to display timings for each step in the output. What do you think the best way to do this might be - an attribute, or is there already an options area somewhere?

Add LightBDD specific IgnoreScenarioAttribute

Tasks

• Extend LightBDD.Core IMetadataProvider to detect execution extensions for scenario and step methods applied with Step/ScenarioExecutionExtensionAttribute attribute
• Extend LightBDD.Core to call scenario and step execution extensions on annotated methods
• Add IgnoreScenarioAttribute and make it applicable on scenario and steps
• Ensure execution results are properly captured when IgnoreScenarioAttribute is applied on scenario
• Ensure backward compatibility

As described in #41, the test framework [Ignore] attributes prevents LightBDD reports from containing ignored scenarios. The only way to make scenario ignored is to use Assert.Ignore() from one of the executed steps, that however may be problematic if common steps are used.

It should be possible to introduce a IgnoreScenarioAttribute class that if applied on scenario method, it would allow runner to attempt executing the scenario but then immediately marking it as ignored. With that approach, the scenario would be included in execution reports.

Revisit documentation pages

Description

Revisit documentation pages to be more intuitive:

• check if Readme page can be make slimmer (and move of the some details to the wiki pages),
• make Quick Start really quick, providing a working sample as soon as possible and put the insights below,
• provide a wiki page with documentation for classes (autogenerated),
• revisit code intellisense docs to ensure they are not too intrusive during usage
• investigate an ability to configure help key (F1) pointing to class docs on wiki,
• document use cases (#63)

Progress

• Feature is implemented,
• Ensured backward-compatibility,
• Ensured good debugging experience

Update solution to VS 2017

1. Update projects
2. Update make scripts
3. Update packages

It is easy to omit await on RunScenarioActionsAsync()

VisualStudio / Resharper does not warn if Runner.RunScenarioActionsAsync() is used and not awaited.

The method implementation should be changed to async (i.e. use await instead of retun) that should trigger warning generation.

Ability to run multi assert steps in parallel

Description

Currently it is possible to mark scenario or composite step with [MultiAssert] attribute that instructs runner to execute all sub-steps regardless of their status and collect the outcome at the end of execution.

This change should allow to flag those sub-steps to run in parallel, assuming that they are independent.
The use case scenario would be a composite step containing sub-steps describing:

• a combination of input and expected output values for an algortihm,
• a list of (longer runnng) actions that can be executed against existing setup.

A spike is required to determine:

• if flag would be implemented as property of MultiAssertAttribute or separate attribute,
• if it makes sense to apply it to scenarios and composite steps or only composites

Progress

• Feature is implemented,
• Ensured backward-compatibility,
• Ensured good debugging experience

Feature request: autogrouping scenario steps

At present I am aware of 2 ways of running scenarios:

1

Runner.RunScenario(
    Given_a_customer,
    And_a_product,
    When_the_customer_selects_the_product,
    Then_foo,
    Then_bar);

• Good: most method names state whether they are arrange, act or assert making reuse unambiguous
• Bad: And_a_product() is ambiguous (could belong to G W or T)
• Bad: Then_foo and Then_bar will be printed in the report as "Then foo Then bar" instead of "Then foo And bar" which would read better

2

Runner.RunScenario(
    given => a_customer(),
    and => a_product(),
    when => the_customer_selects_the_product(),
    then => foo(),
    and => bar());

• Good: whole scenario prints in a nice to read way
• Bad: all methods become simple statements with no indication whether they are arrange act or assert making reuse hard (I've seen tests where the author mistakenly reused an arrange method as an assertion because of this)

Suggestion

Add an optional feature that formats steps in a way that if the first word of the current step equals the first word of the previous step, the first word of the current step will be replaced with And. This would result in a nicely readable output and unambiguous method names.

Example:

Runner.RunScenario(
    Given_a_customer,
    Given_a_product,
    When_the_customer_navigates_to_the_site,
    When_the_customer_selects_the_product,
    Then_foo,
    Then_bar);

would be formatted similarly to:

Given a customer
And a product
When the customer navigates to the site
And the customer selects the product
Then foo
And bar

Optional extras:

• configure case sensitivity for the comparison
• configure the number of leading words to compare and replace
• configure the replacement phrase ("And", "And also" etc.)

Tests not discoverable in VS 2017

Despite installing Nuget package xunit.visualstudio.runner, the test is still not discoverable. When I use [Fact], it finds the test. What is the difference here?

Ability to parameterize scenarios

In NUint there is TestCAse attribute, where you can setup parameters for test and it will be launch sereval times with different parameters. I'm Able to add it in LightBDD scenario, but these parameters don't appear in console output.

e.g.

	[TestCase("v5")]
	[TestCase("v4")]
	[Scenario]
        [Label("Ticket-1")]
        public void It_should_be_possible_to_get_movies_by_lat_and_lng_on_api_version(string version)
        {
			string lat = Config.getConfig().testCities.msk.lat;
			string lng = Config.getConfig().testCities.msk.lng;

			Runner.RunScenario(
				_ => Given_api_version(version),
				_ => Given_lat_and_lng(lat, lng),
				_ => When_trying_get_movies_by_lat_and_lng(),
				_ => Then_we_get_statusCode_OK(response),
				_ => Then_content_is_not_null(response));
        }

here is an output:

SCENARIO: [Ticket-1] It should be possible to get movies by lat and lng on api version
  STEP 1/5: GIVEN api version "v5"...
  STEP 1/5: GIVEN api version "v5" (Passed after 3ms)
  STEP 2/5: AND lat "55.7495307478992" and lng "37.6213073730469"...
  STEP 2/5: AND lat "55.7495307478992" and lng "37.6213073730469" (Passed after <1ms)
  STEP 3/5: WHEN trying get movies by lat and lng...
  STEP 3/5: WHEN trying get movies by lat and lng (Passed after 432ms)
  STEP 4/5: THEN we get statusCode OK [response: "RestSharp.RestResponse"]...
  STEP 4/5: THEN we get statusCode OK [response: "RestSharp.RestResponse"] (Passed after 7ms)
  STEP 5/5: AND content is not null [response: "RestSharp.RestResponse"]...
  STEP 5/5: AND content is not null [response: "RestSharp.RestResponse"] (Passed after 3ms)
  SCENARIO RESULT: Passed after 474ms

SCENARIO: [Ticket-1] It should be possible to get movies by lat and lng on api version
  STEP 1/5: GIVEN api version "v4"...
  STEP 1/5: GIVEN api version "v4" (Passed after <1ms)
  STEP 2/5: AND lat "55.7495307478992" and lng "37.6213073730469"...
  STEP 2/5: AND lat "55.7495307478992" and lng "37.6213073730469" (Passed after <1ms)
  STEP 3/5: WHEN trying get movies by lat and lng...
  STEP 3/5: WHEN trying get movies by lat and lng (Passed after 123ms)
  STEP 4/5: THEN we get statusCode OK [response: "RestSharp.RestResponse"]...
  STEP 4/5: THEN we get statusCode OK [response: "RestSharp.RestResponse"] (Passed after <1ms)
  STEP 5/5: AND content is not null [response: "RestSharp.RestResponse"]...
  STEP 5/5: AND content is not null [response: "RestSharp.RestResponse"] (Passed after <1ms)
  SCENARIO RESULT: Passed after 126ms

It would be great if scenario parameters will appear in output

Add R# code cleanup & code style settings to the project

Getting error "LightBddScopeAttribute is not defined in the project"

Following the quick start steps at https://github.com/LightBDD/LightBDD/wiki/Quick-Start but when I run the NUnit test runner I get the error

OneTimeSetUp: System.InvalidOperationException : LightBddScopeAttribute is not defined in the project. Please ensure that following attribute, or attribute extending it is defined at assembly level: [assembly:LightBddScopeAttribute]
  Exception doesn't have a stacktrace

Attaching the zip file of the VS Solution I was working with.
TheSoco.O365.Extensions.zip

Much thanks for any help solving this issue.

Push changes to nuget

Hi I think the nuget package is quite out of date. Could you perhaps update it?

Add integration with fixie framework

Description

• Check how mature is the framework,
• Conribute to it to solve any showstoppers for integrating it with LightBDD,
• Add integration package

Progress

• Feature is implemented,
• Ensured backward-compatibility,
• Ensured good debugging experience

Optimize LightBDD execution performance

Description

Introduce a project for measuring the performance of LightBDD features with an ability to run tests against different versions of LightBDD and capturing reports in order to compare them.

Based on the performance profiles, optimize the LightBDD tests performance to ensure that tests can execute with a small overhead for larger projects or more compex scenarios.

Progress

• Feature is implemented,
• Ensured backward-compatibility,
• Ensured good debugging experience

Composite steps

Progress:

• Extend Core to support hierarchical step execution
• Extend Framework to allow defining hierarchical steps
• Update Core step metadata to support hierarchical step index
• Extend Framework reports to render hierarchical steps
• Extend Framework progress notifiers to render hierarchical steps
• Ensure debugging experience is smooth
• Ensure that failure reasons and comments properly propagate to scenario result (and indicate which sub-step they belong to)
• Rework how LightBddConfguration is passed to scenario runners and step builders implemented in #23
• Implement context support in step group builder
• Document all new methods
• Update code to be backward compatible

An another idea would be an ability to group steps together, so on the report, steps could be represented in the hierarchical manner.

Such approach could be possibly useful in longer end-to-end / journey tests that covers multiple aspects of the system on execution.

The step group could look like as follows (or similar):

IStepGroup Authenticating_user()
{
  return Runner.DefineStepGroup(
    _ => Given_a_valid_user(),
    _ => When_user_logs_in(),
    _ => Then_user_should_be_successfully_logged_in()
    );
}

IStepGroup Adding_items_to_basket()
{
  return Runner.DefineStepGroup(
    _ => Given_an_authenticated_user(), // verify user is logged in
    _ => Given_item_with_price_is_available_in_shop("socks", 2.29),
    _ => When_user_adds_item_to_the_basket("socks"),
    _ => Then_item_should_be_in_basket("socks"),
    _ => Then_total_price_should_be(2.29),
    );
}

and the scenario could look like that:

void Shopping_journey()
{
  Runner.RunScenario(
    Authenticating_user,
    Adding_items_to_basket,
    Checking_out,
    Paying_with_credit_card,
    Retrieving_invoice
  );
}

Finally, in reports, the scenario could look like that:

• Shopping_journey
  • Authenticating_user
    • Given_a_valid_user
    • When_user_logs_in
    • Then_user_should_be_successfully_logged_in
  • Adding_items_to_basket
    • Given_an_authenticated_user
    • Given_item_with_price_is_available_in_shop
    • When_user_adds_item_to_the_basket
    • Then_item_should_be_in_basket
    • Then_total_price_should_be
  • Checking_out
    • ...
  • Paying_with_credit_card
    • ...
  • Retrieving_invoice
    • ...

Problems:
How to number steps in progress notification and in reports

Enhance rules for determining the step type in extended scenarios

Currently (ver 2.0) the step type would be determined from:

• lambda parameter name if it is not special name _, or
• step method name.

The special name _ rule could be extended by default to any single character name, which would give more flexibility to write scenarios and will prevent an obvious mistakes.

So with this change the following scenario would be properly interpreted:

Runner.RunScenario(
   x => Given_some_state(),
   x => When_some_action_happens(),
   x => Then_some_expectation_should_be_met()
);

The new behaviour should be enabled by default, but a new configuration should be available allowing to customize it:

configuration.StepTypeConfiguration()
   .UpdateStepTypeFromLambdaParameterDetection(name => name.Length == 1)