Code Coverage Options With Combined Coverage Results about unity-test-runner

[nicholas-maltbie]

Alright, I debugged the changes and finishing up all the additions for these requirements of code coverage. Here is a link to the PR, I included some tests and docs to make sure it works as expected. I'll describe the scenarios that I tried to include (I think I have all the ones you outlined form this issue). If you see any issues or have any suggestions, let me know @jhornsb2 :)

#182

[jhornsb2]

@nicholas-maltbie I just got the chance to look through all of the changes. Thank you for implementing it. It looks great and should cover exactly what I need.

[nicholas-maltbie]

I've been looking at the repo and it looks like it could probably be added as a flag to the end of this file, https://github.com/game-ci/unity-test-runner/blob/main/dist/steps/run_tests.sh

Maybe add a new flag like "-enableCodeCoverage"

And at if both "$PLAY_MODE" = "true" and "$EDIT_MODE" = "true" and the code coverage flag is enabled (as a separate clause added at the end), we could add a clause to add custom flags based on if code coverage is included.

I might fork this repo tomorrow to see if I can get it working on my own project for validation. I'll post an update if I come up with anything.

[nicholas-maltbie]

I'm working on drafting a PR for this addition right now, my first time using node in a while so code might be a bit off.

I don't think it will work yet (still need to test it) but working on just setting out the boiler plate code with three parameters

• enableCodeCoverage (optional flag, default of disabled)
• assemblyFilters (optional, requires enableCodeCoverage)
• coverageResultsPath (optional, requires enableCodeCoverage)

https://github.com/nicholas-maltbie/unity-test-runner/tree/nicholas-maltbie/codecoverage-support

I'll work on testing it, but have to get on a flight soon so will probably have some update in the next few days! hopefully can test it with my project soon.

[jhornsb2]

Oh, I was planning on working on the solution for this myself. I was just checking to see if this was a desired change since there have been other issues and pull requests related to code coverage.

[spencer-cheng]

@nicholas-maltbie, @davidmfinol Thanks for your work on the code coverage. I tried to run game-ci/unity-test-runner@v2 in a self-hosted windows runner. I did not provide coverageOptions parameter so the default value is used. I can see the tests are run successfully and the test artifact result is expected. Also, code coverage folder is generated with content. However, I got 0% coverage for all files. When I run the test from the cmd with the same set of parameters locally I get the expected code coverage result which is not 0% for some files. Do you guys have any idea why this is happening? Thanks.

The image below is the command list captured from game-ci/unity-test-runner@v2 log which is the same as the one I used locally except for the path difference.

[nicholas-maltbie]

Hi @spencer-cheng , i ran into this issue as well. I found that explicitly including the assemblies I want to generate coverage options for such as 'assemblyFilters:+my.assembly.*' may help. Are there coverage files generated in the output folder that or is the result just empty?

[spencer-cheng]

Hi, @nicholas-maltbie Thanks for your reply. I tried to explicit include them. But it does not help.
I did more tests in the past two days. It works when i run it on "ubuntu-latest".
However the same code does not when i choose "windows-2019". Similarly, if i run it using a self-hosted windows 2019 runner, the action does not work. If i change to use "run unity command line directly" in step instead of using unity-test-runner action, the coverage data is generated correctly as expected.
I guess there might be a problem with code coverage's windows implementation. Thanks.

[nicholas-maltbie]

@spencer-cheng , there are a few differences between unity when it runs on windows versus Linux in CLI mode. Are you able to produce coverage files properly when running on your local windows machine via cli directly? It might be good to isolate if it's a windows problem when running with CLI or a problem with the way the test runner is configured.

If you're able, could you share the configuration you're using to generate test coverage? Having this workaround of using the run unity command line directly may prove useful to others if this issue takes a while to resolve for windows containers.

[spencer-cheng]

@nicholas-maltbie I am able to produce coverage files properly when running on my local windows machine via CLI directly.

My workaround is like:

1. install Unity on the Windows self-hosted runner and activate the license
2. instead of using unity-test-runner action, i use a step like

 - name: run tests directly from unity
    shell: pwsh
    run: |
      cd dir/path/to/run_test.bat
      ./run_test.bat ${{ env.unity_exe_path}} ${{env.TARGET_TEST_DIR}} ${{env.TARGET_COVERAGE_DIR}}
     echo "::set-output name=artifactsPath::${{ env.TARGET_TEST_DIR}}"
     echo "::set-output name=coveragePath::${{ env.TARGET_COVERAGE_DIR}}"

run_test.bat is like (depends on where the bat file is, the relative path needs to be updated):
"%1" -batchmode -nographics -logFile "..\..\..\%2\playmode.log" -projectPath "..\..\..\3DMonitoringSystem" -runTests -testPlatform playmode -testResults "..\%2\playmode-results.xml" -debugCodeOptimization -enableCodeCoverage -coverageResultsPath "..\%3" -coverageHistoryPath "..\%2" -coverageOptions "generateAdditionalMetrics;generateHtmlReport;generateHtmlReportHistory;generateBadgeReport;assemblyFilters:-Assembly-CSharp-Editor,-My.Assembly;pathFilters:+*/Assets/_Project/*,+*/Assets/InputSettings/*"