Write Many Unit Tests about humbug

[mbj]

and then running humbug on itself

Because I like seeing mutation testing getting traction in other languages I'd like to share my story on this.

I faced a similar challenge with mutant to run the mutation testing engine for ruby, on itself. The moment the engine gets targeted on itself (or any of its dependencies) the results for killed mutations cannot be trusted anymore. Because you cannot easily decide if:

• A mutation was killed by the test suite
• A mutation was killed by a behavior change before the tests actually ran

Also some mutations led the engine to crash, before the test results could be collected. So instead to fiddle around with this rather fragile situation (it can be mitigated somewhat), to get a decent self hosting I created the "zombie mode":

This mode creates a dynamic in-memory self copy of mutant (and its dependencies) inside another namespace. I do this via tracing the requires, expanding the require against the load path, modifing the AST in-transit to duplicate the require into two semantic effects:

• The normal regular one, as if require would not be hijacked, into the regular ::Mutant namespace.
• A namespaced one to build up a 2nd instance of the mutation testing engine, the so called "zombie", into the ::Zombie::Mutant namespace. A dependency like ::Foo would move to ::Zombie::Foo.

Than after the engine is loaded I use the API from ::Zombie::Mutant to kill the mutations in ::Mutant. As the mutation testing engines code exist 2 times now (including a subset of the dependency tree that works nice with this approach), one side can safely mutate the other side, without the shortcomings of a self hosting mutation testing engine that introduces doubt to the result.

Also I'm now able to test dependencies, like AST transform libraries, unparsers, and other building blocks without that much doubt.

I have no idea if this strategy translates well to this project, but it might help to know about.

This post is naturally only 95% of the story, I omitted some ruby specific details, so feel free to ping me on any question. Happy mutating.

References:

• CLI handling for zombie mode
• Mutant self zombification entry point
• Main Zombification operations
• Small utility to handle details for a specific hijacked require

[stof]

and then running humbug on itself 😄

[padraic]

Of course ;). Your 85% that is probably over 90% is taunting me :P.

[padraic]

@mbj In very broad strokes, the approach would work in PHP also. The main facet is to basically have two copies of Humbug assigned to different namespaces where one does not impact the other once both are loading into memory. Achieving it in PHP would likely be a bit more ugly owing to how loading classes works in practice, and the difficulties in actually manipulating that dynamically (it boils down to one undocumented file access flag in PHP file streams of all things), but it's certainly doable.

Thanks for dropping by to suggest it!