Sneeky Mutants escaping when they shouldn't about humbug

[padraic]

Any chance of a reproducible example?

I'd need to look into it, and I can't think of a reason why Humbug would be interfering, but that doesn't mean something isn't going on. The only items Humbug should be manipulating are the source code (never the test code), and it does use an edited copy of your phpunit.xml to run test classes in a fastest-first order.

Could you try running tests using the --tap option using phpunit - just to make sure it still reports those failures (by implanting the mutations) correctly? The TAP does need testing for more scenarios so it's one possible cause.

Also noting to self that I do need to dump junit, and there be naming shenanigans at work when iterating data providers into the result log.

[carnage]

The --tap option shows failures properly, I'll try and construct a test case at the weekend.

[padraic]

Thanks @carnage, would appreciate that :).

[marcaube]

I got the same kind of result with tests using a data provider. Simply moving the data to an array inside the test case and looping on this data removed the false positives.

[padraic]

It is the data providers. The problem is that one of the older bits of pre-Humbug nameless library was the reliance on Junit which is fairly complex for our needs. There's a bug in there around the testsuite nesting. I'd fix it, but honestly I'd prefer to spend the time moving onwards to our own time logging solution which is JSON, simpler, and provides the option of finer grained test aggregation (at present, Humbug is limited to test CLASS aggregation due to PHPUnit limitations which means we run a lot of pointless tests - shall be optional since test insulation is not guaranteed, but a welcome perf boost for those who can swing it).

[padraic]

@carnage @marcaube Guys, give the new master a trial run - do a composer update first. Hopefully this resolves any data provider issues.

[marcaube]

@padraic It works fine for me now. Thanks!

[carnage]

@padraic Fixed it for me as well.

[padraic]

Excellent! :P