Mutator Ignore Comment about humbug

[arnovr]

A bump, i wonder how to deal with this:

    public static function createRandom(): string
    {
        return sha1(uniqid("", true));
    }

1) \Humbug\Mutator\Boolean\TrueValue
--- Original
+++ New
@@ @@
     {
-        return uniqid("", true);
+        return uniqid("", false);
     }

I do realize you could extract the uniqid to a public and measure length. However that creates bloat in code because of the test suite.

[stof]

What is the goal of this ? Forbidding to apply mutations here means you are forbidding to check whether your testsuite prevents mistakes in this code

[padraic]

I see no upside to such a change. The point of Mutation Testing is to generate a log of mutations that should be investigated to see whether improvements to unit tests are feasible. It is taken for granted that a Human will review the logs now and again, and have some understanding of when errors, false positives, etc. may be ignored. Having a facility to disable testing of specific source code line ranges, within the source code itself no less, would undermine this.

[jg-development]

But with always some errors in your result with false positives, the investigation time increases.
Every new member or from time to time you need to check. In PHPUnit I can set a test to ignore or similar. In the case above, it will always results in an error, but the test covered it completly.

[padraic]

Main thing to note is that errors are a positive outcome - it's a good result! We allow errors to be logged because Humbug may also produce them internally and we want to know when that happens.

I think a better approach would be to log everything, and from the log extract a "false positive" list that could flag future similar false positives to be added in subsequent logs under a "Known False Positive" header, e.g. a simple md5 hash ID of mutator, file, line and diff. This is fairly complicated for the end result, so I can't make it a priority at this time when I need to be focusing on testing and getting an initial version out the door.

[jg-development]

Hi,
I tested a little bit and this code in the "mutates" function seems to work. Further testing is needed, but I think this is a good aproach.

        $humbugMutatorIgnoreEndIndex = 0;
        $humbugMutatorIgnoreStartIndex = 0;
        foreach ($tokens as $key => $token) {
            if (is_array($token)) {
                $token = $token[1];
            }
            $humbugMutatorIgnoreStart = strpos($token, '@humbugMutatorIgnoreStart');
            if ($humbugMutatorIgnoreStart !== false) {
                $humbugMutatorIgnoreStartIndex = $key;
            }

            $humbugMutatorIgnoreEnd = strpos($token, '@humbugMutatorIgnoreEnd');
            if ($humbugMutatorIgnoreEnd !== false) {
                $humbugMutatorIgnoreEndIndex = $key;
            }

            if ($humbugMutatorIgnoreEndIndex < $humbugMutatorIgnoreStartIndex &&
                $index > $humbugMutatorIgnoreStartIndex) {
                return false;
            }
            if ($key > $index) {
                break;
            }
        }

[jg-development]

If the actual index/token is after a @humbugMutatorIgnoreStart the mutator will be ignored.
(And the @humbugMutatorIgnoreEnd is not before the index, too)

[padraic]

Closing this. I really cannot figure out the point of such a change other than removal of false positives and there should be better ways of doing that during reporting.

[robinkanters]

@arnovr that's exactly what I'm running into... I now have to live with 1 mutation always surviving... sad.

[wromijn]

This prevents us from using Humbug as part of our continuous delivery process. A simple ignore annotation would do. The fact that some issues were ignored could even be part of the report.