stryker-mutator / stryker-net Goto Github PK

Right now there are 2 kinds of custom exceptions, both inherrit from StrykerException. I initially thought there would be much more kinds of exceptions, but it turned out this is not the case.

I would like to see one kind of custom exception, for example UserException or InputException. It should contain a message that clearly explains to the user what they did wrong and how they can invoke Stryker.NET right next time. Everything that the user can fix themselves should be shown to them using this kind of exception.

• Initital testrun failed
• Initial build failed
• Arguments wrong
• 2 project references found

I would like to see that the user sees no stacktrace in this case, only a clear message about what they did wrong and how they can improve. A stacktrace could give wrong impressions to users, as they think Stryker.NET is broken.

I suggest the following:

• The Core project will handle all exceptions by logging them (except the custom exceptions)
• The CLI project will handle the custom exceptions without a stacktrace

It would be helpful if we could set a threshold for the Mutation score that if not reached would result in the process returning non-zero exit code.

Also failures (like the one described in #120) causes the process to exit with 0.

For CI purposes it is fundamental to return a proper exit code.

The BinaryExpressionMutator is currently called the BinairyExpressionMutator.

In stryker js there is no argument called logFile. the closest argument is logLevelFile. I would like to see that implementation instead of logFile

It would be great if an initial version could be published to NuGet to allow for easier use.

Use the following lanes:

• Inbox
• Committed
• In progress
• Needs review
• Done

Azure DevOps pipelines are now free for open source projects. We could use them to easily build, test and release our application(s). Integrating with SonarCloud would also be easier from Azure DevOps.

How will we create the changelog for new releases?

For Stryker itself, we use conventional commit messages. With those we can generate beautiful changelogs. It also helps to clean up our commit messages. A win-win situation.

An example would be: feat(parser): add ability to parse arrays.. This translates to:

""

Features

Parser: add ability to parse arrays.
""

For C# i would suggest taking a look at clog: https://github.com/clog-tool

@richardwerkman if you agree: i would suggest starting squashing PR into single commits while using these commit messages.

At the moment, mutations are being placed inside if statements like this

if(Environment.GetEnvironmentVariable("ActiveMutation") == "<id>") {
    // mutation
} else {
    // original
}

Not all mutations can be inserted like this. Some mutations can only be inserted using ternary statements like this:

var localVariable = if(Environment.GetEnvironmentVariable("ActiveMutation") == "<id>") ? // mutation : //original

These kind of mutations should also be rollbacked different. My suggestion is an interface for both type of mutant placements. Something like:

public interface IMutantPlacement {
    SyntaxNode InsertMutant(SyntaxNode, Mutant);
    SyntaxNode RemoveMutant(SyntaxNode, Mutant);
}

It would have two implementations. One for if statements and one for ternary mutations.

After a mutationtestrun the mutated assembly stays in place. It is located in the test project bin folder. When code coverage is analysed with the mutated assembly in place this influcences the code coverage result.

This is due to the added if statements inside the mutated assembly. They are not hit (of course) during the code coverage analysis.

This could be resolved by removing the mutated assembly after Stryker is done. Stryker could replace it with the original assembly file to leave all files as they were before the run.

Where would we like to document our technical references? Things like:

• How the mutation algorithm works
• How Mutators can be added
• How the different packages work together

I see some options but would like to share thoughts.

1. GitHub wiki
2. Markdown files in a sub-folder
3. External website

Especially diagrams are hard to maintain together. Maybe some sequence/activity diagrams could be useful. I've seen some tools that can generate diagrams from markdown. Which are best?

Right now the mutation score color thresholds are fixed at 60 and 80.

It would be nice to let users pass their own preferred thresholds. Also an extra threshold could be added for returning a non zero exit code.

Param suggestion: --thresholds 60 80 90 for:

• < 60 return exit code 1
• > 60 mutation score is red
• > 80 mutation score is yellow
• > 90 mutation score is green

Currently stryker supports reading its settings from a json file. Which I fine I guess. But most other dotnet CLI tools and Visual Studio plugins read their settings from the csproj file.

Not sure what led up to the decision to go with a JSON file, just curious if you thought about putting the settings inside the csproj file.

Stryker is reporting different results on different OSes.

TL;DR

While testing Stryker I stumbled upon an intriguing issue, Stryker was reporting some non-killed mutants which I didn't think twice about as it seemed it was working as expected. Then I uploaded the project to a private GitLab server and looked at CI results, one mutant was killed (which didn't happen on my Windows 10 PC).

The issue

When running on Windows 10 (log-20180928-windows-10.txt) or Ubuntu under Windows 10 (log-20180928-windows-10-linux.txt) all running .Net Core SDK 2.1.402 following mutant is marked as Survived but it shouldn't survive given the test case (see MutationTesting.zip).

Binary expression mutation on line 6: 'temp < 1000' ==> 'temp > 1000'

While running on Docker (image: microsoft/dotnet:2.1.402-sdk) it is killed as expected (log-20180928-linux-docker.txt).

I was trying to replicate on public GitLab server, but I got this error:

[13:49:31 ERR] An error occurred during the mutationtest run 
System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values.
Parameter name: MaxDegreeOfParallelism
   at System.Threading.Tasks.ParallelOptions.set_MaxDegreeOfParallelism(Int32 value)
   at Stryker.Core.MutationTest.MutationTestProcess.Test()
   at Stryker.Core.StrykerRunner.RunMutationTest(StrykerOptions options)

During my testing I also got both mutants killed on Windows 10, which happened after I deleted my native language folder from SDK installation so I could send a proper (English speaking) report. Cleaning output folders did fix this issue and results got back to "normal".

Assignment statements can be mutated.

A mutator should be added that mutates these statements.

Right now the only status reporter is the dot reporter. It will print dots and characters after a mutant has been tested.

It would be nice to show users a progress bar to show the progress of the mutationtestrun. A estimation could also be displayed next to the progress bar.

Right now only one testrun occurs at a time. This is not optimal since .Net is excellent in managing threads and parallel processes.

A better situation would be if more testrunners could run in parallel. The number of testrunners should be configurable. When not configured, the number of cores of the machine could be used as default.

This feature would speed up the mutationtest process by a large margin.

When backslash (\) is used as folder separator in csproj (which is inserted by Visual Studio) when referencing another project in a solution, FileNotFoundException is thrown.

csproj snippet:

  <ItemGroup>
    <ProjectReference Include="..\..\src\Lib\Lib.csproj" />
  </ItemGroup>

Output:

Analyzing project
[12:21:49 INF] Using /builds/mutation-testing/tests/Lib.Tests/Lib.Tests.csproj as project file
[12:21:49 ERR] An error occurred during the mutationtest run 
System.IO.FileNotFoundException: Could not find file '/builds/mutation-testing/tests/Lib.Tests/..\..\src\Lib\Lib.csproj'.
File name: '/builds/mutation-testing/tests/Lib.Tests/..\..\src\Lib\Lib.csproj'
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at System.IO.StreamReader..ctor(String path, Encoding encoding, Boolean detectEncodingFromByteOrderMarks, Int32 bufferSize)
   at System.IO.File.OpenText(String path)
   at Stryker.Core.Initialisation.InputFileResolver.FindProjectUnderTestAssemblyName(String projectFilePath)
   at Stryker.Core.Initialisation.InputFileResolver.ResolveInput(String currentDirectory, String projectName)
   at Stryker.Core.Initialisation.InitialisationProcess.Initialize(StrykerOptions options)
   at Stryker.Core.StrykerRunner.RunMutationTest(StrykerOptions options)
[12:21:49 INF] Time Elapsed 00:00:00.1874362

Workaround

Converting to slashes by hand fixes the issue.

We need to configure a online build server. Any thougths on which platform to use?

I know of AppVeyor and Travis

At the moment you can give the CLI invalid loglevels which result in warning-loglevel. I think we might want to throw an error here.

On large projects, it might occur that not all compile errors are returned by Roslyn after the first compile try. If this happens, not all "broken" mutations will be rollbacked.

For now this problem has not occurred yet. Stryker should be tested on a large project to discover if this is an existing bug or not.

Where is the readme.md?

I cloned this repo and tried to run it. Didn't work.
When is the next release? @richardwerkman

Stryker is currently working on a generic html dashboard for mutationtest results. To integrate with this dashboard we will have to generate a *.json file with the testrun results in a specific structure.

In this issue the json schema will be defined.

For now Stryker.NET can only generate the needed json. When the complete dashboard is done we can integrate it with the dashboard.

What needs to be done to add support for .NET Framework next to .NET core support
For now:

• Discover Referenced Assemblies and Types
• Build test project and referenced projects
• VSTest adapter testrunner (https://www.nuget.org/packages/Microsoft.TestPlatform.Portable)
• Dotnet Global Tool

On .NET Core 2.1 projects the build/dotnet-stryker.targets seems to be ignored. This file is being copied by msbuild in the other .NET Core versions.

An awesome feature is added in 2.1 called DotnetTools. This can be used to create global tools like with npm npm i -g

To support DotnetTool a new package has to be created with the type DotnetTool. This could also be added within the current CLI package.

For now the CLI seems to be broken on 2.1. To fix this users can copy the task by hand:

  <Target Name="PrintReferences" DependsOnTargets="ResolveProjectReferences;ResolveAssemblyReferences">
    <Message Text="@(_ResolveAssemblyReferenceResolvedFiles)" Importance="high" />
  </Target>

Into their test .csproj file

Right now it seems like every interface in in a separate file. I personally find it easier to find an interface and the implementation if the interface is in the same file as the implementation (as long as there is one implementation).

Right now, all *.cs files will be mutated by Stryker. Just like with code coverage, users would like to exclude some files.

I see two ways of excluding a file:

• Using an array of filenames as userinput
• By using Attributes inside the code

My preference goes to the attributes. Mainly because this option has more future posibilities like

• excluding parts of files
• excluding only sertain mutators using different attributes.

This could look like:

namespace Stryker.Core.Testing
{
    [ExcludeFromStryker]
    public class ProcessExecutor : IProcessExecutor
    {

namespace Stryker.Core.Testing
{
    [ExcludeFromStryker("BinaryMutator")]
    public class ProcessExecutor : IProcessExecutor
    {

In your solution you call the reports rapport. This is not english, it should be report. There is also at least one class with this mistake in it.

int a = checked(b * c);

Should be mutated to:

int a = unchecked(b * c);

To see if a test is present for System.OverflowException was thrown. And maybe also the other way around. See dotnet csharplang for more info on the checked operator.

Right now all code is checked by every mutator. Some code is not fit for mutating so will always return 0 mutations. Therefore it is not necessary to check this code. Example:

using Stryker.Core.Initialisation.ProjectComponent;
using Stryker.Core.Mutants;
using Stryker.Core.Testing;
using System;
using System.Globalization;
using System.IO;
using System.Linq;

namespace Stryker.Core.Reporters
{
    /// <summary>
    /// The default reporter, prints a simple progress and end result.
    /// </summary>
    public class ConsoleRapportReporter : IReporter
    {

Cannot be mutated by any mutator. So why call the mutators onto these nodes?

Suggestion: only mutate method bodies.

Atm you must have a config file if you dont give any arguments to the CLI. I would like to be able to not give a config file and still dont have a config file.

It would be cool if we check whether there is a config file and if it does not exist we take the default values.

We should be able to configure stryker via a config file. Right now, all config settings are provided using the command line.

This can be in the .csproj file or via a custom stryker configuration file.

It would be cool if this logic can be reused from stryker for dotnet core as well as .NET framework.

The user should be able to define the number of parallel testruns. The default right now is the number of logical cores / 2.

dotnet stryker --parallel 2 

{
  "stryker-config": {
    "parallel": 2
  }
}

We should also explain clearly in the documentation what this setting does.

The number should be validated to be > 0 and be <= the number of logical cores.

When running against a test project using <TargetFrameworks> instead of <TargetFramework> causes a System.NullReferenceException.

[14:39:28 DBG] Determining project under test with name filter null
[14:39:28 ERR] An error occurred during the mutationtest run
System.NullReferenceException: Object reference not set to an instance of an object.
   at Stryker.Core.Initialisation.ProjectFileReader.FindTargetFrameworkReference(XDocument document)
   at Stryker.Core.Initialisation.ProjectFileReader.ReadProjectFile(XDocument projectFileContents, String projectUnderTestNameFilter)
   at Stryker.Core.Initialisation.InputFileResolver.ReadProjectFile(String projectFilePath, String projectName)
   at Stryker.Core.Initialisation.InputFileResolver.ResolveInput(String currentDirectory, String projectName)
   at Stryker.Core.Initialisation.InitialisationProcess.Initialize(StrykerOptions options)
   at Stryker.Core.StrykerRunner.RunMutationTest(StrykerOptions options)
[14:39:28 INF] Time Elapsed 00:00:00.2799214

The only indication that the issue is the TargetFramework is that the stack trace fails on FindTargetFrameworkReference. This could do with a better messaging if you only support targeting one framework, or better support when TargetFrameworks is used.

Stryker has some problems with compiling a few projects as shown in #128.

If users could pass a list/array of files and folders to compile, Stryker would also be able to compile these projects.

{
    "stryker-config":
    {
        "files-to-compile": [
            "C:/some/location",
            "C:/another/location/specific-file.cs",
         ],
    }
}

We will have to test these values to be files of the right extension and scan the given folders for *.cs files. The content of the found files should be parsed to AST's and be fed to Roslyn before the first compilation.

@nicojs what do you think about this solution and the configuration name?

A mutator should be added that mutates strings into other strings.

The mutator should look for StringLiteralExpression nodes

Unary statements can be mutated.

PrefixUnaryStatements

PostfixUnaryStatements

A mutator should be added that mutates these statements.

With nuget packages, you can include compilable source files: https://blog.nuget.org/20160126/nuget-contentFiles-demystified.html

These files end up compiled into the obj\Debug\netcoreapp2.1\NuGet\F3F32FCC102FAF927C28583AC554AF5CC6C67A63\MyLibrary\0.1.2 folder of the project that consumes them. They are effectively linked-in source files.

When Stryker mutates these files, it fails to mutate them back again (presumably because nuget automatically resets them?)

[12:26:48 ERR] Unable to rollback mutation for node EmptyQuery with diagnosticmessage The name 'EmptyQuery' does not exist in the current context
[12:26:48 ERR] An error occurred during the mutationtest run
System.ArgumentException: Node to track is not a descendant of the root.
   at Microsoft.CodeAnalysis.SyntaxNodeExtensions.TrackNodes[TRoot](TRoot root, IEnumerable`1 nodes) in /_/src/Compilers/Core/Portable/Syntax/SyntaxNodeExtensions_Tracking.cs:line 43
   at Stryker.Core.Compiling.RollbackProcess.RemoveMutantIfStatements(SyntaxTree originalTree, ICollection`1 diagnosticInfo)
   at Stryker.Core.Compiling.RollbackProcess.Start(CSharpCompilation compiler, ImmutableArray`1 diagnostics)
   at Stryker.Core.Compiling.CompilingProcess.Compile(IEnumerable`1 syntaxTrees, MemoryStream ms)
   at Stryker.Core.MutationTest.MutationTestProcess.Mutate()
   at Stryker.Core.StrykerRunner.RunMutationTest(StrykerOptions options)

In the above message, EmptyQuery refers to a method defined in a base class inside the imported source contentFile from the nuget package.

It would be great if either compiled contentFiles from obj were ignored from mutation, and/or the ability to ignore files was added. The error looks similar to the error in #125

In the handbook are the mutators described that are being used within the Stryker eco system. Now there is a mismatch of the terminology being used in documentation/code of Stryker.NET.

Binary mutator -> Binary Operators
Boolean mutator -> Boolean Substitutions
PrefixUnaryStatements -> Update operators or Boolean Substitutions
PostfixUnaryStatements -> Update operators

I don't know how to contribute to this project because it lacks a contributions.md.
Plz fix, k thx.

Change the argument names to use dashes instead of camel case. For instance: timeoutMS will change to timeout-ms

Sooooooo when are you going to use AppVeyor to display the build status? @richardwerkman

It would be cool if we use stryker-net on stryker-net itself in the build. That way we can check our own score.

On a small personal project to implement an immutable priority queue, Stryker.NET crashes with the following error:

[07:59:19 ERR] Unable to rollback mutation for node head with diagnosticmessage The name 'head' does not exist in the current context
[07:59:19 ERR] An error occurred during the mutationtest run
System.ArgumentException: Node to track is not a descendant of the root.
   at Microsoft.CodeAnalysis.SyntaxNodeExtensions.TrackNodes[TRoot](TRoot root, IEnumerable`1 nodes) in /_/src/Compilers/Core/Portable/Syntax/SyntaxNodeExtensions_Tracking.cs:line 43
   at Stryker.Core.Compiling.RollbackProcess.RemoveMutantIfStatements(SyntaxTree originalTree, ICollection`1 diagnosticInfo)
   at Stryker.Core.Compiling.RollbackProcess.Start(CSharpCompilation compiler, ImmutableArray`1 diagnostics)
   at Stryker.Core.Compiling.CompilingProcess.Compile(IEnumerable`1 syntaxTrees, MemoryStream ms)
   at Stryker.Core.MutationTest.MutationTestProcess.Mutate()
   at Stryker.Core.StrykerRunner.RunMutationTest(StrykerOptions options)
[07:59:19 INF] Time Elapsed 00:00:10.8997693

Using the --logConsole trace flag I was able to isolate the cause to this piece of code:

var (head, toEnqueue) = _comparer(Head, value) <= 0
    ? (Head, value)
    : (value, Head);

BTW: thank you for including good diagnostic support!

As a workaround, I modified the code by removing use of tuples and the ternary operator:

T head;
T toEnqueue;
if (_comparer(Head, value) <= 0)
{
    head = Head;
    toEnqueue = value;
}
else
{
    head = value;
    toEnqueue = Head;
}

With this change, Stryker.NET doesn't crash on this piece of code ... though it does crash on two other very similar examples. Once those were similarly amended, all the mutants were tested - and bugs were identified. 😄

A simple ternary operator seems to be fine, and tuple deconstruction that doesn't require mutation also seems to be fine, so I suspect the problem occurs only with both language features combined.

When running stryker twice, the second time the initial build will fail becouse of a still running process. This still needs investigating but it looks like this happens after a timeout process kill.