Even though I have commented out some specific asserts from my tests, the Mutation Coverage still indicates that the expected lines that should survive (indicate that they are not covered by unit tests) are killed.
I know there's a chance that maybe I didn't comment out all asserts that are checking those lines, but I really did, and I've spent the last 3 hours investigating why this happens and I have no clue why this is happening.
One thing I've noticed, there is some extra logging going on as you can see below
6:04:15 PM PIT >> INFO : Verbose logging is disabled. If you encounter an problem please enable it before reporting an issue.
6:04:16 PM PIT >> INFO : MINION : objc[935]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/bin/java (0x107de74c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x107eaf4e0
6:04:16 PM PIT >> INFO : MINION : ). One of the two will be used. Which one is undefined.
6:04:16 PM PIT >> INFO : Sending 1 test classes to minion
6:04:16 PM PIT >> INFO : Sent tests to minion
6:04:16 PM PIT >> INFO : MINION : 6:04:16 PM PIT >> INFO : Checking environment
6:04:16 PM PIT >> INFO : MINION : 6:04:16 PM PIT >> INFO : Found 16 tests
6:04:16 PM PIT >> INFO : MINION : 6:04:16 PM PIT >> INFO : Dependency analysis reduced number of potential tests by 0
6:04:16 PM PIT >> INFO : MINION : 6:04:16 PM PIT >> INFO : 16 tests received
6:04:20 PM PIT >> INFO : MINION : com.android.volley.AuthFailureError$$EnhancerByMockitoWithCGLIB$$1424bfbf
6:04:20 PM PIT >> INFO : MINION : Caused by: com.android.volley.AuthFailureError
6:04:20 PM PIT >> INFO : MINION : at com.mycompany.access.webservice.helpers.WSManagerTest.createSpyAuthFailureError(WSManagerTest.java:359)
at com.mycompany.access.webservice.helpers.WSManagerTest.testAddRequest_errorThrown(WSManagerTest.java:143)
6:04:20 PM PIT >> INFO : MINION : at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
6:04:20 PM PIT >> INFO : MINION : at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
6:04:20 PM PIT >> INFO : MINION : at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.robolectric.RobolectricTestRunner$HelperTestRunner$1.evaluate(RobolectricTe
6:04:20 PM PIT >> INFO : MINION : stRunner.java:488)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.robolectric.internal.SandboxTestRunner$2.evaluate(SandboxTestRunner.java:209)
at org.robolectric.internal.SandboxTestRunner.runChild(SandboxTestRu
6:04:20 PM PIT >> INFO : MINION : nner.java:109)
at org.robolectric.internal.SandboxTestRunner.runChild(SandboxTestRunner.java:36)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
6:04:20 PM PIT >> INFO : MINION :
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
6:04:20 PM PIT >> INFO : MINION :
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
6:04:20 PM PIT >> INFO : MINION : at org.robolectric.internal.SandboxTestRunner$1.evaluate(SandboxTestRunner.java:63)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
6:04:20 PM PIT >> INFO : MINION : at org.pitest.junit.adapter.CustomRunnerExecutor.run(CustomRunnerExecutor.java:42)
at org.pitest.junit.adapter.AdaptedJUnitTestUnit.execute(AdaptedJUnitTestUnit.java:85)
6:04:20 PM PIT >> INFO : MINION :
at org.pitest.coverage.execute.CoverageDecorator.execute(CoverageDecorator.java:50)
at org.pitest.testapi.execute.containers.UnContainer.execute(UnContainer.java:32)
at org.pitest.testapi.execute.Pitest.executeTests(Pitest.java:59)
6:04:20 PM PIT >> INFO : MINION : at org.pitest.testapi.execute.Pitest.run(Pitest.java:50)
at org.pitest.coverage.execute.CoverageWorker.run(CoverageWorker.java:52)
at org.pitest.coverage.execute.CoverageMinion.main(CoverageMinion.java:85)
6:04:20 PM PIT >> INFO : MINION : com.android.volley.AuthFailureError$$EnhancerByMockitoWithCGLIB$$1424bfbf
6:04:20 PM PIT >> INFO : MINION : Caused by: com.android.volley.AuthFailureError
at com.mycompany.access.webservice.helpers.WSManagerTest.createSpyAuthFailureError(WSManagerTest.java:359)
6:04:20 PM PIT >> INFO : MINION : at com.mycompany.access.webservice.helpers.WSManagerTest.testAddRequest_errorThrown(WSManagerTest.java:143)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcces
6:04:20 PM PIT >> INFO : MINION : sorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org
6:04:20 PM PIT >> INFO : MINION : .junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
a
6:04:20 PM PIT >> INFO : MINION : t org.robolectric.RobolectricTestRunner$HelperTestRunner$1.evaluate(RobolectricTestRunner.java:488)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.robolectric.internal.SandboxTestRunner$2.evaluate(SandboxTestRunne
6:04:20 PM PIT >> INFO : MINION : r.java:209)
at org.robolectric.internal.SandboxTestRunner.runChild(SandboxTestRunner.java:109)
at org.robolectric.internal.SandboxTestRunner.runChild(SandboxTestRunner.java:36)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit
6:04:20 PM PIT >> INFO : MINION : .runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java
6:04:20 PM PIT >> INFO : MINION : :268)
at org.robolectric.internal.SandboxTestRunner$1.evaluate(SandboxTestRunner.java:63)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.pitest.junit.adapter.CustomRunnerExecutor.run(CustomRunnerExecutor.java:42)
at org.pitest.juni
6:04:20 PM PIT >> INFO : MINION : t.adapter.AdaptedJUnitTestUnit.execute(AdaptedJUnitTestUnit.java:85)
at org.pitest.coverage.execute.CoverageDecorator.execute(CoverageDecorator.java:50)
at org.pitest.testapi.execute.containers.UnContainer.execute(UnContainer.java:32)
at org.pitest.test
6:04:20 PM PIT >> INFO : MINION : api.execute.Pitest.executeTests(Pitest.java:59)
at org.pitest.testapi.execute.Pitest.run(Pitest.java:50)
at org.pitest.coverage.execute.CoverageWorker.run(CoverageWorker.java:52)
at org.pitest.coverage.execute.CoverageMinion.main(CoverageMinion.java:85)
6:04:20 PM PIT >> INFO : MINION :
6:04:20 PM PIT >> INFO : MINION : 6:04:20 PM PIT >> WARNING : More threads at end of test (18) testAddRequest_errorThrown(com.mycompany.access.webservice.helpers.WSManagerTest) than start. (4)
6:04:20 PM PIT >> INFO : MINION : 6:04:20 PM PIT >> WARNING : More threads at end of test (30) testAddRequest_shouldNotAddForwardHttpHeader(com.mycompany.access.webservice.helpers.WSManagerTest) than start. (18)
6:04:20 PM PIT >> INFO : MINION : 6:04:20 PM PIT >> WARNING : More threads at end of test (42) testAddSerialRequest_shouldAddForwardHttpHeader(com.mycompany.access.webservice.helpers.WSManagerTest) than start. (30)
6:04:20 PM PIT >> INFO : MINION : 6:04:20 PM PIT >> WARNING : More threads at end of test (54) testExecuteRequestOnHold_emptyRequestsOnHoldQueue(com.mycompany.access.webservice.helpers.WSManagerTest) than start. (42)
6:04:20 PM PIT >> INFO : MINION : com.android.volley.AuthFailureError$$EnhancerByMockitoWithCGLIB$$1424bfbf
6:04:20 PM PIT >> INFO : MINION : Caused by: com.android.volley.AuthFailureError
6:04:20 PM PIT >> INFO : MINION : at com.mycompany.access.webservice.helpers.WSManagerTest.createSpyAuthFailureError(WSManagerTest.java:359)
at com.mycompany.access.webservice.helpers.WSManagerTest.testExecuteRequestOnHold_errorThrown(WSManagerTest.jav
6:04:20 PM PIT >> INFO : MINION : a:202)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang
6:04:20 PM PIT >> INFO : MINION : .reflect.Method.invoke(Method.java:497)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMe
6:04:20 PM PIT >> INFO : MINION : thod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.robolectric.RobolectricTestRunner$HelperTestRunner$1.evaluate(RobolectricTestRunner.java:488)
at org.junit.intern
6:04:20 PM PIT >> INFO : MINION : al.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.robolectric.internal.SandboxTestRunner$2.evaluate(SandboxTestRunner.java:209)
at org.robolectric.internal.SandboxTestRunner.runChild(SandboxTestRunner.java:109)
at org.robolectric.inte
6:04:20 PM PIT >> INFO : MINION : rnal.SandboxTestRunner.runChild(SandboxTestRunner.java:36)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:
6:04:20 PM PIT >> INFO : MINION : 288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.robolectric.internal.SandboxTestRunner$1.evaluate(SandboxTestRunner.java:63)
at org.junit.runners.ParentR
6:04:20 PM PIT >> INFO : MINION : unner.run(ParentRunner.java:363)
at org.pitest.junit.adapter.CustomRunnerExecutor.run(CustomRunnerExecutor.java:42)
at org.pitest.junit.adapter.AdaptedJUnitTestUnit.execute(AdaptedJUnitTestUnit.java:85)
at org.pitest.coverage.execute.CoverageDecorator.e
6:04:20 PM PIT >> INFO : MINION : xecute(CoverageDecorator.java:50)
at org.pitest.testapi.execute.containers.UnContainer.execute(UnContainer.java:32)
at org.pitest.testapi.execute.Pitest.executeTests(Pitest.java:59)
at org.pitest.testapi.execute.Pitest.run(Pitest.java:50)
at org.pitest
6:04:20 PM PIT >> INFO : MINION : .coverage.execute.CoverageWorker.run(CoverageWorker.java:52)
at org.pitest.coverage.execute.CoverageMinion.main(CoverageMinion.java:85)
6:04:20 PM PIT >> INFO : MINION : 6:04:20 PM PIT >> WARNING : More threads at end of test (66) testExecuteRequestOnHold_errorThrown(com.mycompany.access.webservice.helpers.WSManagerTest) than start. (54)
6:04:20 PM PIT >> INFO : MINION : 6:04:20 PM PIT >> WARNING : More threads at end of test (78) testGetSerialRequestQueue_requestQueueNotNull(com.mycompany.access.webservice.helpers.WSManagerTest) than start. (66)
6:04:20 PM PIT >> INFO : MINION : 6:04:20 PM PIT >> WARNING : More threads at end of test (90) testHandleTokenAbsoluteLifeTimeExpiration(com.mycompany.access.webservice.helpers.WSManagerTest) than start. (78)
6:04:20 PM PIT >> INFO : MINION : 6:04:20 PM PIT >> WARNING : More threads at end of test (102) testHandleTokenNotAuthenticated(com.mycompany.access.webservice.helpers.WSManagerTest) than start. (90)
6:04:20 PM PIT >> INFO : MINION : 6:04:20 PM PIT >> WARNING : More threads at end of test (114) testRemoveListenerForVolleyRequest_nullCompletionHandler(com.mycompany.access.webservice.helpers.WSManagerTest) than start. (102)
6:04:20 PM PIT >> INFO : MINION : 6:04:20 PM PIT >> WARNING : More threads at end of test (126) testRemoveListenerForVolleyRequest_volleyDataRequest(com.mycompany.access.webservice.helpers.WSManagerTest) than start. (114)
6:04:20 PM PIT >> INFO : MINION : 6:04:20 PM PIT >> WARNING : More threads at end of test (138) testRemoveListenerForVolleyRequest_volleyStringRequest(com.mycompany.access.webservice.helpers.WSManagerTest) than start. (126)
|6:04:20 PM PIT >> INFO : Calculated coverage in 4 seconds. /
6:04:20 PM PIT >> INFO : Created 1 mutation test units
stderr : objc[938]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home/jre/bin/java (0x108e344c0) and /Library/Java/JavaVirtualMachi6:04:26 PM PIT >> INFO : Completed in 10 secondstrument.dylib (0x108efc4e0stderr : ). One of the two will be used. Which one is undefined. /
- Timings
================================================================================
scan classpath : < 1 second
coverage and dependency analysis : 4 seconds
build mutation tests : < 1 second
run mutation analysis : 5 seconds
Total : 10 seconds
================================================================================
- Statistics
================================================================================
Generated 27 mutations Killed 26 (96%)
Ran 27 tests (1 tests per mutation)
================================================================================
- Mutators
================================================================================
org.pitest.mutationtest.engine.gregor.mutators.VoidMethodCallMutator
Generated 14 Killed 13 (93%)
KILLED 13 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0
MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
NO_COVERAGE 0
org.pitest.mutationtest.engine.gregor.mutators.ReturnValsMutator
Generated 5 Killed 5 (100%)
KILLED 5 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
NO_COVERAGE 0
org.pitest.mutationtest.engine.gregor.mutators.NegateConditionalsMutator
Generated 8 Killed 8 (100%)
KILLED 8 SURVIVED 0 TIMED_OUT 0 NON_VIABLE 0
MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0
NO_COVERAGE 0
BUILD SUCCESSFUL
Total time: 35.859 secs
Initially I was using Robolectric 2.4 but I've also updated to 3.3.2 and the issue is the same (although different logs). The log above is from 3.3.2