nunit / nunit-v2-framework-driver Goto Github PK
View Code? Open in Web Editor NEWEngine extension supporting execution of NUnit V2 tests
License: MIT License
Engine extension supporting execution of NUnit V2 tests
License: MIT License
Whenever a new feature or fix is merged to master, we should publish pre-release packages using the 'dev' label on MyGet.
With the NUnit 2.6.6 release, I plan to put out a package with the nunit.core and nunit.core.interfaces assemblies. It's aimed specifically at the v2 framework driver as well as any other code that wants to use V2 at this low level.
I propose that once the package is created we use it here updating to 2.6.6. This opens the possibility of fixing an occasional bug that affects the v2 driver in the V2 core itself.
How do I use this? I've tried dropping the DLLs into my path but it made no difference. Do I need to configure something?
The project for the extension has a post-build step that copies files to create a "pseudo-package" install so we can run integration tests. That step contains Windows paths that break the build on Linux. We'll have to find some other way to do it either by invoking a script that works on both environments or limiting the integration tests to the command-line builds that use Cake. For now, the Travis CI build is turned off.
This doesn't impact the console runner or VS adapter, which don't use id filters. In a GUI, they are the most natural thing to use.
If the GUI knew when it was running V2 tests, it could pass a different filter. However, it doesn't know that and we wouldn't want each runner to have to deal with this separately.
The problem is exacerbated by the fact that the exception generated is swallowed whole by MasterTestRunner
, which uses a when
clause to only handle certain exceptions. Other exceptions are allowed to continue. That would be OK except that the method where this happens is run on a worker thread so the exception goes unhandled.
I'll fix this by first handling the exception and reporting it. Beyond that, we need to decide if the engine should do a filter conversion or the V2 driver should do it. Currently, the V2 driver converts other filters, but not the id filter.
It would also be possible to upgrade the legacy V2 code to directly handle id filters, in which case that feature could be used by a new release of the V2 driver.
I'm assigning this to myself for the purpose of at least reporting the exception. Some discussion on how to take it further is needed to take the next steps.
NUnit.Engine.ITestRunner.Explore(...)
does not obey the filter that is passed as argument to it and returns a list of all the tests within the assembly.
This behaviour is inconsistent with calling NUnit.Engine.ITestRunner.Run(...)
, which obeys the filter as expected and only executes the tests which pass the filter.
See new issue nunit/teamcity-event-listener#28
See issue nunit/nunit-console#37
See PR nunit/nunit-console#38
A Chocolatey package now exists for the extension, having been created using the binaries from the 3.6 release. See nunit/nunit-console#253 and nunit/nunit-project-loader#8.
When I'm running nunit2 tests using the nunit v2 framework driver - the console outputs the correct number of passes and failures, but the XML sent to both the Event Listener and the xml result output is incorrect.
For example:
I have 3 tests which output:
Test Run Summary
Overall result: Failed
Test Count: 3, Passed: 1, Failed: 2, Warnings: 0, Inconclusive: 0, Skipped: 0
Failed Tests - Failures: 1, Errors: 1, Invalid: 0
The TestResults.xml file reports:
<test-run id="2" testcasecount="3" result="Failed" total="3" passed="0" failed="0" inconclusive="0" skipped="0" asserts="0" engine-version="3.7.0.0" clr-version="4.0.30319.42000" start-time="2017-07-19 12:06:05Z" end-time="2017-07-19 12:09:15Z" duration="190.467029">
The Event Listener XML is the same:
<test-run id="2" testcasecount="3" result="Failed" total="3" passed="0" failed="0" inconclusive="0" skipped="0" asserts="0" engine-version="3.6.1.0" clr-version="4.0.30319.42000" start-time="2017-07-18 14:12:58Z" end-time="2017-07-18 14:16:08Z" duration="190.485957">
Interestingly, if I use the legacy nunit2 report format, the TestResults.xml file outputs correctly:
<test-results name="D:\MS\test-apps\Xamarin.NUnit2.Tests\bin\Debug\Xamarin.NUnit2.Tests.dll" total="3" errors="1" failures="1" not-run="0" inconclusive="0" ignored="0" skipped="0" invalid="0" date="2017-07-19" time="12:44:44">
Let me know if this issue is better placed in the main nunit3 repo - I'll have a dig about in the code and see if I can work out what's going on.
I was considering making use of the NUnit.ConsoleRunner nuget in order to make each project's console runner configurable per project in our custom build system but if I go this route it doesn't look like there is any obvious way to integrate this V2 driver to run on projects that contain NUnit 2.x tests. Am I right? Is there a corresponding console runner package for NUnit 2 that I should be using for these projects instead?
The extension build scripts have evolved independently so they are somewhat uneven.Although they work well, it would be convenient for maintenance of the extensions if they were reasonably parallel.
The current "ideal" CI script would use the following:
The v2 framework driver scripts will be reviewed to see which of the above items need to be added.
Hi,
I noticed that when running V3 tests with the V3 console against an assembly, the output XML's first test-suite element (with type="Assembly") has a name
attribute, which is only the assembly name (like Foo.dll
) and a fullname
(like C:\Bar\Foo.dll
).
Running V2 tests with the V3 runner and the nunit-v2-framework-driver seems to have C:\Bar\Foo.dll
in both the name
and fullname
attribute.
I'd expect name
to be the assembly file name with V2 tests as well.
Best regards,
Stefan Ilic
Does the driver provide backwards compatibility for NUnit 2 addins?
I have a test assembly that has an ISuiteBuilder implementation. It would be great to be able to run this assembly using the NUnit 3 test runner, but it doesn't appear to support this at the moment.
It seems, in a project referencing nunit.framework v2.6, that nunit3-console when given "--where=cat != SomeCategory" at the command line also runs tests marked Explicit. In the absence of any filter, tests marked Explicit are not run.
I'm filing the issue in the nunit-v2-framework-driver project rather than the nunit-console project as it seems the most likely place for the error. I would assume that others before me would have discovered this issue, as it runs contrary to the way category filters should work wrt. explicitness being selected in a NotFilter.
Should be able to use same approach as for V2 Result Writer.
When using a filter like the one mentioned below, tests marked with ExplicitAttribute (but without any of the mentioned categories) is selected. This is unexpected.
When using a filter with only a single negated category, no ExplicitAttribute marked tests is selected - as expected.
When using the nunit3 framework the filter works a expected.
--where "cat != Db and cat != UnitTest"
resulting in:
<filter>
<and>
<not>
<cat>Db</cat>
</not>
<not>
<cat>UnitTest</cat>
</not>
</and>
</filter>
There is a rather hackish workaround, which doesn't work with Nunit3: --where "cat != Db, UnitTest"
- but I rather not use that.
Since NUnit V2 is legacy, we need a document that tells people what kind of issues are in scope and will be accepted on this project. Should probably go in the root.
Define labels and pipelines for this project to match other projects.
I don't understand why the NuGet package for this extension includes it's own .addins
file. None of our other extensions do.
It wouldn't be picked up by any of the default location methods, as far as I can tell, and I'm not aware of anything which makes this extensions need different location logic to any of our others.
Was it a legacy way of installing? Should i be removed?
The nuget reference to NUnit.Engine.API refers to a dev release on myget because there is as yet no production release of the API. Once the 3.5 engine is released, this should be changed for each extension.
Hey there,
I have a .NET 6.0 console project which references NUnit.Engine 3.14.0 and NUnit.Extension.NUnitV2Driver 3.9.0. When I try to run NUnit2 tests from test assembly (targeting .NET Framework 4.8, referencing NUnit 2.6.4), I get the following result:
Either assembly contains no tests or proper test driver has not been found.
According to the documentation, V2 Framework Driver should be used automatically if installed. Am I missing something?
Please find attached solution which can be used to reproduce the issue.
NUnit3_Repro.zip
Integration tests (V2 tests) are currently run in the bin directory immediately after unit testing. They would be more useful and effective if run after packaging, using the content of the packages themselves. This will also help with issue #1.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.