Support running multiple test assemblies in a single TestRunner invocation about testrunner

[macro187]

Okay, that's a really good point. So would TestRunner be better in your situation if you could pass multiple .dlls to it on the command line, like:

TestRunner.exe /path/to/sometests.dll /path/to/somemoretests.dll /path/to/evenmoretests.dll

I don't personally think this is any better than using for or find, but if supporting it makes TestRunner a better drop-in replacement for mstest, then I think you've convinced me it's a good idea.

[macro187]

Hi Pascal,

Apologies for taking so long to respond - I've been on vacation.

My (and Einstein's) general philosophy is to keep things as simple as they can be, but no simpler. So I'd assumed you'd just loop and run TestRunner for each assembly.

It's also nice from a technical perspective because each assembly runs in a pristine process, guaranteed to be free of leftover threads, assembly configuration, memory consumption, etc.

That said, I would consider the feature if there was a compelling use case. Do you have one?

Regards,
Ron

[picrap]

Hi Ron,

vacations are always a priority, so I understand 😉
Yes, I have a case where it would be useful. I use Gitlab CI, and the most simple option here (as Einstein would have loved) is to test all assemblies from one single command, since Gitlab CI will use the error code to stop. The other option for me is to create a batch, detect generated assemblies, use the errorlevel to exit if there is an error.
However, I was ready to contribute and submit the pull request for this.

Regards.
Pascal.

[macro187]

Hi Pascal,

Won't something like this work as your CI test step:

for %%f in (C:\path\to\tests\*.dll) do (
    path\to\TestRunner.exe "%%f"
    if not %errorlevel% == 0 exit /b %errorlevel%
)

This will work even if it picks up .dll files that contain no tests or aren't even .NET assemblies, since TestRunner returns exit code 0 in those cases.

I'm really hesitant to add file-finding functionality to TestRunner when better tools for the job already exist (in this case Windows for and Unix find).

I'm still open to being convinced otherwise, if you feel strongly about it and have a good argument.

Regards,
Ron

[macro187]

I've also updated the readme to document exit codes in more detail as discussed above. 6e4ea30

[picrap]

Yes it would work, but currently the original mstest application does the job better, so I'll probably stay with it.

[macro187]

For the record, the mstest program is no better than TestRunner in this regard: It accepts a path to a single test dll via its /testcontainer switch.

According to its documentation, it also appears to be able to work off of .orderedtest and .vsmdi files, but it's not clear to me what those are or how useful that functionality is.

https://msdn.microsoft.com/en-us/library/ms182489.aspx#Anchor_0

[picrap]

... But it accepts the /testcontainer multiple times, so it can test multiple assemblies in a single call 😉

[macro187]

Done for v1.5

[picrap]

Thank you, I'll take a look at it.