nunit / nunit-v2-framework-driver

Engine extension supporting execution of NUnit V2 tests

NUnit 4 Framework

NUnit is a unit-testing framework for all .NET languages. It can run on macOS, Linux and Windows operating systems. NUnit can be used for a wide range of testing, from unit testing with TDD to full fledged system and integration testing. It is a non-opinionated, broad and deep framework with multiple different ways to assert that your code behaves as expected. Many aspects of NUnit can be extended to suit your specific purposes.

The latest version, version 4, is an upgrade from the groundbreaking NUnit 3 framework. It is a modernized version, aimed at taking advantage of the latest .NET features and C# language constructs.

If you are upgrading from NUnit 3, be aware of the breaking changes. Please see the NUnit 4 Migration Guide and take care to prepare your NUnit 3 code before you do the upgrade.

Table of Contents

• Downloads
• Documentation
• Contributing
• License
• NUnit Projects

Downloads

The latest stable release of the NUnit Framework is available on NuGet or can be downloaded from GitHub. Pre-release builds are available on MyGet.

Documentation

Documentation for all NUnit projects can be found at the documentation site.

• NUnit Documentation
• Installation
• Release Notes
• Code Samples

Contributing

For more information on contributing to the NUnit project, please see CONTRIBUTING.md and the Developer Docs.

NUnit 3.0 was created by Charlie Poole, Rob Prouse, Simone Busoli, Neil Colvin and numerous community contributors. A complete list of contributors since NUnit migrated to GitHub can be found on GitHub.

Earlier versions of NUnit were developed by Charlie Poole, James W. Newkirk, Alexei A. Vorontsov, Michael C. Two and Philip A. Craig.

License

NUnit is Open Source software and NUnit 4 is released under the MIT license. Earlier releases used the NUnit license. Both of these licenses allow the use of NUnit in free and commercial applications and libraries without restrictions.

NUnit Projects

NUnit is made up of several projects. When reporting issues, please try to report issues in the correct project.

Core Projects

• NUnit Test Framework - The test framework used to write NUnit tests (this repository)
• NUnit Visual Studio Adapter - Visual Studio/Dotnet adapter for running NUnit 3 and 4 tests in Visual Studio or the dotnet command line.
• NUnit Console and Engine - Runs unit tests from the command line and provides the engine that is used by other test runners to run NUnit tests.

Visual Studio Extensions

• NUnit Visual Studio Adapter - Visual Studio adapter for running NUnit 3 and 4 tests in Visual Studio or the dotnet command line.
• NUnit Visual Studio Templates - Project templates and snippets for writing unit tests in Visual Studio. This repo is for reference only, as the templates have been donated to the Dotnet project and are maintained there, and also released as part of the dotnet releases.
• Visual Studio Test Generator - Generates NUnit tests in Visual Studio
• NUnit 2 Visual Studio Adapter - Visual Studio adapter for running older NUnit 2.x tests in Visual Studio and in VSTS/TFS builds

NUnit Engine Extensions

• NUnit 2 Driver - Allows the NUnit 3 engine to run NUnit 2 tests
• NUnit 2 Result Writer - Writes test results in the legacy NUnit 2 format
• Visual Studio Project Loader - Loads and parses Visual Studio projects and solutions
• NUnit Project Loader - Loads NUnit projects

nunit-v2-framework-driver's Issues

Set up standard labels and pipelines

Define labels and pipelines for this project to match other projects.

Publish 'dev' builds on MyGet

Whenever a new feature or fix is merged to master, we should publish pre-release packages using the 'dev' label on MyGet.

Explicitness of tests is ignored in the presence of a filter with adding multiple negated categories

When using a filter like the one mentioned below, tests marked with ExplicitAttribute (but without any of the mentioned categories) is selected. This is unexpected.
When using a filter with only a single negated category, no ExplicitAttribute marked tests is selected - as expected.

When using the nunit3 framework the filter works a expected.

--where "cat != Db and cat != UnitTest"

resulting in:

<filter>
  <and>
    <not>
      <cat>Db</cat>
    </not>
    <not>
      <cat>UnitTest</cat>
    </not>
  </and>
</filter>

There is a rather hackish workaround, which doesn't work with Nunit3: --where "cat != Db, UnitTest" - but I rather not use that.

Standardize build scripts for extensions

The extension build scripts have evolved independently so they are somewhat uneven.Although they work well, it would be convenient for maintenance of the extensions if they were reasonably parallel.

The current "ideal" CI script would use the following:

• Use of loaded cake files such that the same name is used for the same purpose in each project
• Use of typed Cake context in most tasks (BuildParameters)
• Use of GitVersion to determine build version
• Inclusion of package tests using multiple console versions
• Publication of dev packages to myget
• Publication of production packages to nuget and chocolatey
• Creation of draft and final releases on GitHub
• Ability to run publication and release steps independently in case a re-run is needed.

The v2 framework driver scripts will be reviewed to see which of the above items need to be added.

Can't build on Linux

The project for the extension has a post-build step that copies files to create a "pseudo-package" install so we can run integration tests. That step contains Windows paths that break the build on Linux. We'll have to find some other way to do it either by invoking a script that works on both environments or limiting the integration tests to the command-line builds that use Cake. For now, the Travis CI build is turned off.

Exploring tests with filter returns all tests

NUnit.Engine.ITestRunner.Explore(...) does not obey the filter that is passed as argument to it and returns a list of all the tests within the assembly.

This behaviour is inconsistent with calling NUnit.Engine.ITestRunner.Run(...), which obeys the filter as expected and only executes the tests which pass the filter.

Change default branch from master to main

Use NUnit V2 Core package rather than hard coded 2.6.4 binaries

With the NUnit 2.6.6 release, I plan to put out a package with the nunit.core and nunit.core.interfaces assemblies. It's aimed specifically at the v2 framework driver as well as any other code that wants to use V2 at this low level.

I propose that once the package is created we use it here updating to 2.6.6. This opens the possibility of fixing an occasional bug that affects the v2 driver in the V2 core itself.

Run integration tests on packages

Integration tests (V2 tests) are currently run in the bin directory immediately after unit testing. They would be more useful and effective if run after packaging, using the content of the packages themselves. This will also help with issue #1.

How does this work?

How do I use this? I've tried dropping the DLLs into my path but it made no difference. Do I need to configure something?

Event Listener and Report output incorrect or nunit2 test runs

When I'm running nunit2 tests using the nunit v2 framework driver - the console outputs the correct number of passes and failures, but the XML sent to both the Event Listener and the xml result output is incorrect.

For example:
I have 3 tests which output:
Test Run Summary
Overall result: Failed
Test Count: 3, Passed: 1, Failed: 2, Warnings: 0, Inconclusive: 0, Skipped: 0
Failed Tests - Failures: 1, Errors: 1, Invalid: 0

The TestResults.xml file reports:

<test-run id="2" testcasecount="3" result="Failed" total="3" passed="0" failed="0" inconclusive="0" skipped="0" asserts="0" engine-version="3.7.0.0" clr-version="4.0.30319.42000" start-time="2017-07-19 12:06:05Z" end-time="2017-07-19 12:09:15Z" duration="190.467029">

The Event Listener XML is the same:

<test-run id="2" testcasecount="3" result="Failed" total="3" passed="0" failed="0" inconclusive="0" skipped="0" asserts="0" engine-version="3.6.1.0" clr-version="4.0.30319.42000" start-time="2017-07-18 14:12:58Z" end-time="2017-07-18 14:16:08Z" duration="190.485957">

Interestingly, if I use the legacy nunit2 report format, the TestResults.xml file outputs correctly:

<test-results name="D:\MS\test-apps\Xamarin.NUnit2.Tests\bin\Debug\Xamarin.NUnit2.Tests.dll" total="3" errors="1" failures="1" not-run="0" inconclusive="0" ignored="0" skipped="0" invalid="0" date="2017-07-19" time="12:44:44">

Let me know if this issue is better placed in the main nunit3 repo - I'll have a dig about in the code and see if I can work out what's going on.

Create a level of support statement

Since NUnit V2 is legacy, we need a document that tells people what kind of issues are in scope and will be accepted on this project. Should probably go in the root.

Explicitness of tests is ignored in the presence of a negated filter

It seems, in a project referencing nunit.framework v2.6, that nunit3-console when given "--where=cat != SomeCategory" at the command line also runs tests marked Explicit. In the absence of any filter, tests marked Explicit are not run.

I'm filing the issue in the nunit-v2-framework-driver project rather than the nunit-console project as it seems the most likely place for the error. I would assume that others before me would have discovered this issue, as it runs contrary to the way category filters should work wrt. explicitness being selected in a NotFilter.

Integrate creation of chocolatey package into the build script

A Chocolatey package now exists for the extension, having been created using the binaries from the 3.6 release. See nunit/nunit-console#253 and nunit/nunit-project-loader#8.

V2 Driver does not support id filters

This doesn't impact the console runner or VS adapter, which don't use id filters. In a GUI, they are the most natural thing to use.

If the GUI knew when it was running V2 tests, it could pass a different filter. However, it doesn't know that and we wouldn't want each runner to have to deal with this separately.

The problem is exacerbated by the fact that the exception generated is swallowed whole by MasterTestRunner, which uses a when clause to only handle certain exceptions. Other exceptions are allowed to continue. That would be OK except that the method where this happens is run on a worker thread so the exception goes unhandled.

I'll fix this by first handling the exception and reporting it. Beyond that, we need to decide if the engine should do a filter conversion or the V2 driver should do it. Currently, the V2 driver converts other filters, but not the id filter.

It would also be possible to upgrade the legacy V2 code to directly handle id filters, in which case that feature could be used by a new release of the V2 driver.

I'm assigning this to myself for the purpose of at least reporting the exception. Some discussion on how to take it further is needed to take the next steps.

Are v2 addins supported?

Does the driver provide backwards compatibility for NUnit 2 addins?

I have a test assembly that has an ISuiteBuilder implementation. It would be great to be able to run this assembly using the NUnit 3 test runner, but it doesn't appear to support this at the moment.

Automate GitHub Release Process

Should be able to use same approach as for V2 Result Writer.

Automate the GitHub release process

Remove nunit.v2.driver.addins?

I don't understand why the NuGet package for this extension includes it's own .addins file. None of our other extensions do.

It wouldn't be picked up by any of the default location methods, as far as I can tell, and I'm not aware of anything which makes this extensions need different location logic to any of our others.

Was it a legacy way of installing? Should i be removed?

NUnit 3 console should produce xml events for ITestEventListener which contain unique id in the scope of all test agents for NUnit 2 tests

See new issue nunit/teamcity-event-listener#28
See issue nunit/nunit-console#37
See PR nunit/nunit-console#38

Is it possible to use this driver along with NUnit.ConsoleRunner?

I was considering making use of the NUnit.ConsoleRunner nuget in order to make each project's console runner configurable per project in our custom build system but if I go this route it doesn't look like there is any obvious way to integrate this V2 driver to run on projects that contain NUnit 2.x tests. Am I right? Is there a corresponding console runner package for NUnit 2 that I should be using for these projects instead?

Change API reference to released version

The nuget reference to NUnit.Engine.API refers to a dev release on myget because there is as yet no production release of the API. Once the 3.5 engine is released, this should be changed for each extension.

Name and Fullname are the same in V3 XML format

Hi,

I noticed that when running V3 tests with the V3 console against an assembly, the output XML's first test-suite element (with type="Assembly") has a name attribute, which is only the assembly name (like Foo.dll) and a fullname (like C:\Bar\Foo.dll).

Running V2 tests with the V3 runner and the nunit-v2-framework-driver seems to have C:\Bar\Foo.dll in both the name and fullname attribute.

I'd expect name to be the assembly file name with V2 tests as well.

Best regards,
Stefan Ilic

V2 Framework Driver not picked up

Hey there,

I have a .NET 6.0 console project which references NUnit.Engine 3.14.0 and NUnit.Extension.NUnitV2Driver 3.9.0. When I try to run NUnit2 tests from test assembly (targeting .NET Framework 4.8, referencing NUnit 2.6.4), I get the following result:

Either assembly contains no tests or proper test driver has not been found.

According to the documentation, V2 Framework Driver should be used automatically if installed. Am I missing something?

Please find attached solution which can be used to reproduce the issue.
NUnit3_Repro.zip