Weird escaped mutations about humbug

[stof]

If you want to reproduce this, you can use https://github.com/stof/Mink/tree/humbug to have the humbug config file I used (it is not yet in the main repo)

[stof]

Out of the 19 mutations reported as escaped, here is my own report:

• 4 of them are harmless
• 3 are actual missing testcases
• 8 are catched by tests when I apply them manually and run the testsuite => bug in Humbug
• 4 are generated infinite loops when I apply them manually => should be catched as timeout by Humbug

[padraic]

ok 2 - Behat\Mink\Tests\Selector\PartialNamedSelectorTest::testInvalidLocator

Oddly enough, passing when I get the result output on that specific mutation. So the mutation is somehow not getting loaded via the StreamWrapper, or it's getting kicked in before the StreamWrapper is triggered. Will check into tomorrow when it's not 3am here ;). Thanks for trying it out - and kudos, you are barely within the definition of my target audience.

[padraic]

The job script that is destined for an adapter was suppressing notices. That will get you up to 85% on the specific example given. A few of the timeouts appear to creating fatal errors before the Humbug timeout horizon is reached.

[stof]

@padraic fatal errors should be considered as errors, not as escaped, right (we already have a few of them reported properly) ?

[padraic]

There's a certain ambiguity to fatal errors since Humbug sits in the loop and it can also error out (esp at this stage in its development). It's fairly unlikely but we're literally just over the hill from that territory where runkit was a mess. At present, errors are indeed counted as kills in the final scores but we still report them separately just in case Humbug is at fault.

I should probably emphasise that in the readme and switch to something less alarming that a big bright yellow E though!

[stof]

@padraic sure they are reported spearately. This is OK IMO. But the 19 mutations I reported above (which are now only 18 thanks to your latest fix) don't include the ones reported as errors.

[padraic]

I know :). I'm going through them now one by one to locate cause.

[padraic]

Just to confirm the issue cause, apparently there is a bug in registering intercepts with the StreamWrapper. Mink has 119 mutants generated, and 109 intercepts are registered to the StreamWrapper, so there would be 10 mutations untested.

Looks like phpunit is not loading the bootstrap at all, but only in those specific cases.

[padraic]

Busy yesterday, but taking a look this morning the problem is that these particular mutations use data providers exclusively so they are being skipped over by Humbug (there's a messy mapping between mutation/code coverage/junit logs - the junit part is complex and sometimes unreliable). Tt\he JUnit stuff is down for replacement with a custom listener (exact same thing as I implemented for phpspec time logging), so I'll do that today.

The new logging should also make it possible to push performance higher by tackling test grouping - at present, we are limited to test class filtering which means a lot of unnecessary tests are being run, though this performance boost is entirely reliant on tests being genuinely insulated from each other hence it will be optional.

[padraic]

@stof Give it a whirl now after composer update. I implemented the basics of the phpunit-extensions package and replaced all the funky junit to code coverage mapping. This should provide a more reliable and accurate run (assuming there be no bugs in the filtering!).

[stof]

not working. It reports all mutations as uncovered

[padraic]

Darn it. I ran it on Mink here and it's running fine with a 95% score. I am using an older branch however. I'll check again later using the parent repo's master when I have time.

[padraic]

This is likely a separate bug. Confirmed as a fix for others so we're getting there.

[stof]

why is it marked as duplicate ? Where is the duplicated report ?

anyway, in case it helps you, here are the versions of the dependencies being used by my humbug setup in case it is related to them (this is the output of composer show -i:

doctrine/instantiator        1.0.4              A small, lightweight utilit...
mockery/mockery              0.9.3              Mockery is a simple yet fle...
padraic/phpunit-accelerator  v1.0.2             PHPUnit Accelerator: Speeds...
padraic/phpunit-extensions   dev-master 8526850 Collection of Listeners to ...
phpunit/php-code-coverage    2.0.15             Library that provides colle...
phpunit/php-file-iterator    1.3.4              FilterIterator implementati...
phpunit/php-text-template    1.2.0              Simple template engine.
phpunit/php-timer            1.0.5              Utility class for timing
phpunit/php-token-stream     1.4.0              Wrapper around PHP's tokeni...
phpunit/phpunit              4.4.4              The PHP Unit Testing framew...
phpunit/phpunit-mock-objects 2.3.0              Mock Object library for PHP...
sebastian/comparator         1.1.0              Provides the functionality ...
sebastian/diff               1.2.0              Diff implementation
sebastian/environment        1.2.1              Provides functionality to h...
sebastian/exporter           1.1.0              Provides the functionality ...
sebastian/global-state       1.0.0              Snapshotting of global state
sebastian/recursion-context  1.0.0              Provides functionality to r...
sebastian/version            1.0.4              Library that helps with man...
symfony/console              v2.6.3             Symfony Console Component
symfony/finder               v2.6.3             Symfony Finder Component
symfony/process              v2.6.3             Symfony Process Component
symfony/yaml                 v2.6.3             Symfony Yaml Component

[padraic]

@stof #49

It's technically a duplicate (re escaping mutants that shouldn't escape), but the fix not working indicates something else above the original #49 so I'm tracking that here.

[padraic]

@stof Checked out Mink master, and it's running fine. Dependencies at same version as above (tried under both phpunit 4.4.4 and 4.4.5 that was released earlier).

padraic@padraic:~/projects/Mink$ ../humbug/bin/humbug

 _  _            _
| || |_  _ _ __ | |__ _  _ __ _
| __ | || | '  \| '_ \ || / _` |
|_||_|\_,_|_|_|_|_.__/\_,_\__, |
                          |___/ 
Humbug version 1.0-dev

Humbug running test suite to generate logs and code coverage data...

  357 [==========================================================] 15 secs

Humbug has completed the initial test run successfully.

Humbug is analysing source files...

Mutation Testing is commencing on 27 files...
(.: killed, M: escaped, S: uncovered, E: fatal error, T: timed out)

...........M...............................................T |   60 ( 8/27)
..TT............MTM..EE.......EE.............M.MM.........M

119 mutations were generated:
     104 mutants were killed
       0 mutants were not covered by tests
       7 covered mutants were not detected
       4 fatal errors were encountered
       4 time outs were encountered

Out of 119 test covered mutations, 94% were detected.
Out of 119 total mutations, 94% were detected.
Out of 119 total mutations, 100% were covered by tests.

Remember that some mutants will inevitably be harmless (i.e. false positives).

Humbug results are being logged as JSON to: build/humbuglog.json
Humbug results are being logged as TEXT to: build/humbuglog.txt

[stof]

@padraic my mistake. I forgot I had disabled XDebug locally for another task. The new warning you added in Humbug told me about it and it is indeed running fine now.

[padraic]

@stof No problem :)