akkadotnet / akka.multinodetestrunner Goto Github PK

Currently each NodeRunner runs it's node in separate process. This provides good isolation, but does not allow to debug specs with break points, etc. Only debugging via test output is possible.

That would be cool to be able to run specs from VS, and set breakpoints for debugging. This is possible if running nodes in same test process, but different AppDomains - this is basically what xunit runner does for parallel test runs.

Should give it a try.

This issue is ported from akkadotnet/akka.net#3615.

What is expected

1. New Nuget package should be implemented, i.e. Akka.MultiNodeTestRunner.VisualStudio
2. In project with tests. we should reference required testing frameworks packages (i.e. xunit or maybe only Akka.Remote.TestKit), and test runner Akka.MultiNodeTestRunner.VisualStudio instead of usual xunit.runner.visualstudio. Kind of:

<ItemGroup>
	<PackageReference Include="Microsoft.NET.Test.Sdk" />
	<PackageReference Include="Akka.Remote.TestKit" />
	<PackageReference Include="Akka.MultiNodeTestRunner.VisualStudio" />
</ItemGroup>

3. After that, we should have VS discovering tests, and running them, like:
   

Also, tests will be able to run with simple dotnet test, just like with xunit runner.

Some more details can be found in the original issue.

As defined in this section at the bottom, we can attach log output and other attachments to each test result. Need to use this feature to report failure description, aggregated output logs and logs per each node (related to #10)

Dependabot couldn't reach https://www.myget.org/F/xunit/api/v3/index.json as it timed out.

Is https://www.myget.org/F/xunit/api/v3/index.json accessible over the internet? If it is then this may be a transitory issue and can be ignored - Dependabot will close it on its next successful update run.

View the update logs.

Currently all messages are published to SinkCoordinator or just printed in console. Need to take these messages (possibly not all of them, but important ones) and pass to IFrameworkHandle.SendMessage method. This will write them in dotnet test output.

After #24 will be merged, the we will be able to reference Akka.MultiNode.TestAdapter nuget package and run dotnet test on the test projects.

But - after referencing package, Akka.MultiNode.NodeRunner.runtimeconfig.json is added to a project and can not be excluded. This is because of <None> item in package's custom msbuild target file.

So while this helps to copy runtime config to output directory, we need to fix target to not include this config file into consuming project.

As described in this section of TestAdapter how-to it is possible to show several results per one spec. Example is how xunit shows results for [Theory] marked tests.

Might be a good idea to display each node's result in such way, and global spec result in parent node.

Currently first test in spec is checked for Skip property, and if it is set the whole spec is skipped.

Need to skip only the marked test, and iterate all other tests in spec.

Need to have nuget package that can be included to any project to enable MNTR discovery

Version Information
Version of Akka.NET? v1.4.27, Akka.MultiNodeTestRunner 1.0
Which Akka.NET Modules? Akka.Cluster.TestKit

Describe the bug
When attempting to run any of the built-in Akka.NET multinode tests, i.e. https://github.com/akkadotnet/akka.net/tree/dev/src/core/Akka.Cluster.Tests.MultiNode - which target multiple .NET versions, dotnet test will hang indefinitely without printing any output.

To Reproduce
Execute the the following command after cloning and checking out to https://github.com/akkadotnet/akka.net/tree/dev/src/core/Akka.Remote.Tests.MultiNode

PS> dotnet test

I see the following output, and that's it:

  Akka.Remote.Tests.MultiNode -> D:\Repositories\olympus\akka.net\src\core\Akka.Remote.Tests.MultiNode\bin\Debug\net5.0\Akka.Remote.Tests.MultiNode.dll
Test run for D:\Repositories\olympus\akka.net\src\core\Akka.Remote.Tests.MultiNode\bin\Debug\net5.0\Akka.Remote.Tests.MultiNode.dll (.NETCoreApp,Version=v5.0)

Microsoft (R) Test Execution Command Line Tool Version 16.10.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.

Nothing will happen. The test will just sit there.

This also doesn't work if I try to specify a target framework.

dotnet test -c Release --no-build --logger:trx --framework net5.0

Note: the tests run fine if I launch them via FAKE.

Expected behavior
I should immediately see some output from the dotnet test process.

Actual behavior
I don't see anything at all - it looks like the process is hanging.

Screenshots

Environment
Are you running on Linux? Windows? Docker? Which version of .NET?

Windows 10, using .NET 5 on both PowerShell and on the traditional DOS prompt.

Need to add some end-to-end integration tests which use Akka.Cluster, the Akka.Cluster.TestKit NuGet packages, etc...

Hey guys,
I'm getting the following error at running MSBuild @ vs2019

1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2182,5): error MSB4018: The "ResolveAssemblyReference" task failed unexpectedly.
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2182,5): error MSB4018: Microsoft.Build.Shared.InternalErrorException: MSB0001: Internal MSBuild Error: Unexpected CopyLocal flag.
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2182,5): error MSB4018:    at Microsoft.Build.Tasks.CopyLocalStateUtility.IsCopyLocal(CopyLocalState state)
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2182,5): error MSB4018:    at Microsoft.Build.Tasks.ResolveAssemblyReference.LogResults(ReferenceTable dependencyTable, List`1 idealAssemblyRemappings, List`1 idealAssemblyRemappingsIdentities, List`1 generalResolutionExceptions)
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2182,5): error MSB4018:    at Microsoft.Build.Tasks.ResolveAssemblyReference.Execute(FileExists fileExists, DirectoryExists directoryExists, GetDirectories getDirectories, GetAssemblyName getAssemblyName, GetAssemblyMetadata getAssemblyMetadata, GetRegistrySubKeyNames getRegistrySubKeyNames, GetRegistrySubKeyDefaultValue getRegistrySubKeyDefaultValue, GetLastWriteTime getLastWriteTime, GetAssemblyRuntimeVersion getRuntimeVersion, OpenBaseKey openBaseKey, GetAssemblyPathInGac getAssemblyPathInGac, IsWinMDFile isWinMDFile, ReadMachineTypeFromPEHeader readMachineTypeFromPEHeader)
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2182,5): error MSB4018:    at Microsoft.Build.Tasks.ResolveAssemblyReference.Execute()
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2182,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute()
1>C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Current\Bin\Microsoft.Common.CurrentVersion.targets(2182,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.<ExecuteInstantiatedTask>d__26.MoveNext()

Packages referenced via NuGet

    <PackageReference Include="Akka.Cluster.TestKit" Version="1.4.18" />
    <PackageReference Include="Akka.MultiNodeTestRunner" Version="1.4.18"/>
    <PackageReference Include="Akka.TestKit.Xunit2" Version="1.4.18" />

and I was able to workaround the problem via adding IncludeAssets and PrivateAssets in .csproj

    <PackageReference Include="Akka.Cluster.TestKit" Version="1.4.18" />
    <PackageReference Include="Akka.MultiNodeTestRunner" Version="1.4.18">
	    <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
	    <PrivateAssets>all</PrivateAssets>
    </PackageReference>    
    <PackageReference Include="Akka.TestKit.Xunit2" Version="1.4.18" />

Might need to port the custom code we use for packing up the MNTR exes from the Akka.NET solution:

https://github.com/akkadotnet/akka.net/blob/35171031e9c8bc0e9107f99af930748d27a00cbb/build.fsx#L412-L470

That will depend on how we choose to ship the new MNTR going forward.

TestExecutor has Cancel method used by dotnet test to perform cancellation. VS will use it to stop tests too.

It is no-op now, but should cancel all running nodes (I guess by killing processes) and stop MNTR.

Right now there are lots of blocking things. Need to rewrite it to use async/await.

As stated in TestAdapter How-To, and also documented on MSDN, the only way to pass parameters from dotnet test to our custom adapter is to use `.runsettings xml file and pass it to the runner (as shown in this SO answer).

So need to add settings parsing.

There are some packages like Microsoft.Extensions.DependencyModel that are referenced from multiple projects with explicit version specified. Need to store this versions in one place, common.props file.