macro187 / testrunner

A console MSTest runner

TestRunner

A console MSTest runner

Features

Lightweight, with no external dependencies.

Cross-platform, tested on .NET Framework (Windows), .NET Core (Windows and Linux), and Mono (Linux).

Process isolation runs test assemblies in separate processes.

Reflection-based test discovery supports test assemblies built against any variant or version of the MSTest dll.

Test output captured from Console.Out, Console.Error, and System.Diagnostics.Trace.

Timings measured for all test, initialize, and cleanup methods.

Exception details provided for expected exceptions in tests, and unexpected failures in test, initialize, and cleanup methods.

Test attributes supported include [TestClass], [TestMethod], [TestInitialize], [TestCleanup], [ClassInitialize], [ClassCleanup], [AssemblyInitialize], [AssemblyCleanup], [ExpectedException], and [Ignore].

TestContext members supported include CurrentTestOutcome, FullyQualifiedTestClassName, and TestName.

Assembly .config files are supported.

Limitations

Some test attributes are unsupported and are ignored and have no effect.

Some TestContext members are unsupported and return null at runtime.

Assembly .config files don't work on Mono. Issue #17.

<bindingRedirect>s in assembly .config files have no effect.

Requirements

.NET Framework 4.6.1 (or newer)

.NET Core 2.0 (or newer)

Mono 5.0.0 (or newer)

NuGet Package

https://www.nuget.org/packages/TestRunner/

Includes net461 binaries for .NET Framework and Mono, and netcoreapp2.0 binaries for .NET Core.

Usage

SYNOPSIS

    testrunner.exe [options] <testfile>...
    testrunner.exe --help

DESCRIPTION

    Run tests in <testfile>(s)

OPTIONS

    --outputformat <outputformat>
        Set the output format

        human
            Human-readable text format (default)

        machine
            Machine-readable JSON-based format (experimental)

    --class <namespace>.<class>
    --class <class>
        Run the specified test class.

        If <namespace> is omitted, run all test classes with the specified
        name.

        If not specified, run all test classes.

        Can be specified multiple times.

        Case-sensitive.

        Does not override [Ignore] attributes.

    --method <namespace>.<class>.<method>
    --method <method>
        Run the specified test method.

        If <namespace> and <class> are omitted, run all test methods with
        the specified name (constrained by --class).

        If not specified, run all test methods (constrained by --class).

        Can be specified multiple times.

        Case-sensitive.

        Does not override [Ignore] attributes.

    --help
        Show usage information

EXIT STATUS

    0 if all specified test files succeed, non-zero otherwise.

    Test files succeed if all test, initialization, and cleanup methods run
    successfully.

    Test files succeed if they contain no tests.

    Test files succeed if they are not .NET assemblies.

    Test files fail if any test, initialization, or cleanup methods fail.

    Test files fail if they do not exist.

EXAMPLES

    .NET Framework

        testrunner.exe C:\Path\To\TestAssembly.dll C:\Path\To\AnotherTestAssembly.dll

    .NET Core

        dotnet testrunner.dll C:\Path\To\TestAssembly.dll C:\Path\To\AnotherTestAssembly.dll

    Mono

        mono --debug testrunner.exe /path/to/TestAssembly.dll /path/to/AnotherTestAssembly.dll

History

Forked from Bernd Rickenberg's https://testrunner.codeplex.com/ revision 87713 on September 24th, 2016.

testrunner's Issues

Add --class option

To run only particular test class(es).

Original request:

Could you add some arguments to the command line to limit the run to any/some testclass or any/some testmethod

ex:

testrunner.exe C:\Path\To\TestAssembly.dll C:\Path\To\AnotherTestAssembly.dll -c "xxxns.yyyyns.MyTestClass1,xxxns.zzzns.MyTestClass2" -m "xxxns.zzzns.MyTestClass3.TestingMethod1,......"

Research correct behaviour on exceptions in initialize and cleanup methods

Find out what MS runner behaviour is, add tests for it, and adjust TestRunner's behaviour if necessary.

TestContext support

Hi,

I don't have the TestContext available in my Tests (e.g. in a [AssemblyInitialize] method) if I run the test with the testrunner.exe.

Is it possible to get the TestContext in another way or do you know how to fix the problem?

Many thanks!

Multiple output formats support

Refactor TestRunner to support different output formats, with an initial "default" format producing the current format.

Not compatible with System.Reactive.Interfaces, Version=3.1.1

Test Runner gives exception when used with System.Reactive.Interfaces, Version=3.1.1. The exception stack trace gives information that it is looking for 'System.Reactive.Interfaces, Version=3.0.0.0".

xUnit test support

Mono automated builds and tests

Add --method option

To run only particular test method(s).

Support running multiple test assemblies in a single TestRunner invocation

Hi,

do you plan to support testing multiple assemblies at once (either by wildcard or multiple parameters)?

Thanks.

Multiprocess not working on .NET Core

This is probably because TestRunner needs to be taught how to re-invoke itself using the dotnet command on .NET Core.

=================================
testrunner v1.0.0
Copyright (c) 2012-2018
Ron MacNeil, Rickenberg, JDeltorp
=================================

Unhandled Exception: System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

.trx output file format support

Requires #3 #20 #21

Auto-discover tests in solution

If no test DLL's specified on command line, auto-discover them in some reasonable way

TestRunDirectory support

Figure out how this normally works in MSTest, then implement it and surface appropriate information in TestContext.

cc @eczimmermann

NUnit test support

.config file redirection doesn't work on Mono

Originally thought this was impossible due to...
mono/mono#10584

...but there may be hope...
https://stackoverflow.com/a/39394998/2167919

TestContext property getter(s) not working on .NET Core

For some reason, .NET Core will not accept the dynamically generated TestContext overrides. .NET Framework seems to accept them.

=============================================================================================
Assembly: testrunner.Tests.MSTest/bin/Debug/netcoreapp2.0/publish/testrunner.Tests.MSTest.dll
=============================================================================================

Test Assembly:
C:\p\testrunner\testrunner.Tests.MSTest\bin\Debug\netcoreapp2.0\publish\testrunner.Tests.MSTest.dll

[AssemblyInitialize] AssemblyInitialize()

An internal error occurred in testrunner.dll:
Method 'get_Properties' in type 'TestContextProxy' from assembly 'TestContextProxyAssembly, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' does not have an implementation.
Type: System.TypeLoadException
Source: System.Private.CoreLib
StackTrace:
  at System.Reflection.Emit.TypeBuilder.TermCreateClass(RuntimeModule module, Int32 tk, ObjectHandleOnStack type)
  at System.Reflection.Emit.TypeBuilder.CreateTypeNoLock()
  at System.Reflection.Emit.TypeBuilder.CreateTypeInfo()
  at TestRunner.Domain.TestContextProxy.GetTypeFromTypeBuilder(TypeBuilder typeBuilder)
    in C:\p\testrunner\testrunner\Domain\TestContextProxy.cs:line 189
  at TestRunner.Domain.TestContextProxy.BuildProxy()
    in C:\p\testrunner\testrunner\Domain\TestContextProxy.cs:line 59
  at TestRunner.Domain.TestContextProxy.get_Proxy()
    in C:\p\testrunner\testrunner\Domain\TestContextProxy.cs:line 26
  at TestRunner.Program.Program.RunMethod(MethodInfo method, Object instance, Boolean takesTestContext, Type expectedException, Boolean expectedExceptionAllowDerived, String prefix)
    in C:\p\testrunner\testrunner\Program\Program.cs:line 509
  at TestRunner.Program.Program.RunTestAssembly(String assemblyPath)
    in C:\p\testrunner\testrunner\Program\Program.cs:line 198
  at TestRunner.Program.Program.Main2(String[] args)
    in C:\p\testrunner\testrunner\Program\Program.cs:line 59

Pure reflection test discovery

Don't depend on test framework assemblies (UnitTestFramework.dll, nunit.dll, etc.)

ResultAccumulatingEventHandler

For building event handlers that need to remember all the results from a test run.

Research whether it's possible for assembly redirects in switched config files to work

If not, can it be improved?

Remove assembly reference(s) from test suites to TestRunner

All tests currently fail with the following exception, likely due to how per-test-assembly child processes are being run. Does not occur on Windows.

Could not resolve type with token 01000008
  Type: System.TypeLoadException
  StackTrace:
    at (wrapper managed-to-native) System.Reflection.MonoMethod:InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&)
    at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x00038]
      in <f712f98eb8e445c8918edaf595bbe465>:0

Move non-essential logic out of runners into event handlers

Runners are still doing too much, e.g. results tabulation, "success" decision logic, and maybe more.

Reify method, test, class, and assembly result information as types in a new Results namespace.

Teach *End events to carry results using those types.

Shift logic that aggregates finer-grained event(s) into *End results into a new ResultBuilderEventHandler or similar.

???

Fold `Results` into `Events`

Is there any reason for the Results classes to exist separately from their respective End events, or can they just be folded into them?

Add TestRunBeginEvent, TestRunEndEvent, and TestRunResults

For use by output format event handlers.