The test runner gets stuck in the "Testing in playmode" phase about unity-test-runner

[Andri-gunawan-susanto]

just found solution in my case, just changing uses: game-ci/unity-test-runner@v2 to uses: game-ci/[email protected] maybe there some run test code in new game-ci version that I can't used in game-ci test runner

[markus-seidl]

I guess this could be interesting: https://docs.unity3d.com/2022.3/Documentation/Manual/CLIBatchmodeCoroutines.html
TLDR: Not all Coroutines are supported in batch mode, most notably: WaitForEndOfFrame . Which might be part of this issue for some.

[ugnelis]

Probably related https://answers.unity.com/questions/1888121/unity-gets-stuck-on-applicationreload-when-enterin.html

[ugnelis]

Looks like one of the releases caused this. It would be extremely helpful if you could help us find out which one.

Do the versions v2.0.1 and v2.0.2 work for you too?

I have tested v2.0.1 and v2.0.2. Both of these versions get stuck in the "Testing in playmode" phase

[ugnelis]

No problem. I am not sure what is wrong. The maintainer should answer this question. FIY, I have tested v2.0.0, v2.0.1, and v2.0.2 but only v2.0.0 works as intended

[asyasmi]

@ugnelis
Yeah, I'm also using game-ci for a while and I'm super happy with it.

@asyasmi what output do you get?

Creating "/github/workspace/ubuntu-latest_2021.3.21f1_StandaloneLinux64_artifacts" if it does not exist.

###########################
#    Project directory    #
###########################

total 56K
drwxr-xr-x 11 1001  123 4.0K May 19 10:54 .
drwxr-xr-x  4 root root 4.0K May 19 10:54 ..
drwxr-xr-x  2 1001  123 4.0K May 19 10:52 .ci
drwxr-xr-x  8 1001  123 4.0K May 19 10:52 .git
-rw-r--r--  1 1001  123  210 May 19 10:52 .gitattributes
drwxr-xr-x  3 1001  123 4.0K May 19 10:52 .github
-rw-r--r--  1 1001  123 1.2K May 19 10:52 .gitignore
drwxr-xr-x  4 1001  123 4.0K May 19 10:52 Assets
drwxr-xr-x  2 1001  123 4.0K May 19 10:52 Packages
drwxr-xr-x  2 1001  123 4.0K May 19 10:52 ProjectSettings
-rw-r--r--  1 1001  123   51 May 19 10:52 README.md
drwxr-xr-x  2 root root 4.0K May 19 10:54 _activate-license
drwxr-xr-x  2 root root 4.0K May 19 10:54 ubuntu-latest_2021.3.21f1_StandaloneLinux64_artifacts

###########################
#   Testing in editmode  #
###########################

that's all before the cancellation of the job :(

Regarding timeout-minutes - we had no timeout at all, it still never for out of this state.
But I think this time I got it.

UPD: Eventually that were our tests.
The issue was with tests, one of them eventually was broken and just hung a process and logs were not shown before they were done and tests were never done, so... The job ended in this limbo.

So I'd really be waiting for game-ci/cli#12 to be realized.

[spaskhalov]

I have messed with same behaviour... Test runner just freezing on "Testing in playmode"

[Peaj]

Same problem here. Locally the tests take around 7s to complete.
I canceled the test action after around 20min.
Is there any way to get more information on whats going on after "Testing in Playmode" is logged?

[kcarlson]

We're getting the same problem with both edit and playmode tests.

[Andri-gunawan-susanto]

Any update with this problem? I've got the same issue after update unity 2020.3.38f1

[webbertakken]

Any update with this problem?

Updates would be posted in this thread. It doesn't look like a complete failure for projects and all versions, since there's only 5 reports here in 2 to 3 months.

What would help is if everyone added their unity versions, and ideally also some stats on memory usage (you could use atop or other measuring tools).

Either way it may be hard to debug the cause here because the Unity Editor isn't giving us much to work with. Perhaps some kind of debug output would also help. Not sure if anyone managed to get more output than nothing at all?

[kcarlson]

We experienced failures using unity 2019.4.40f1
For now a workaround, we are circumventing the test-runner action and executing something like the following, which works:

- name: Run Tests
  id: tests
  env:
    UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
  run: |
    docker run --gpus all --rm \
        --env UNITY_LICENSE \
        --volume="${PWD}:/opt/project" \
        unityci/editor:ubuntu-2019.4.40f1-linux-il2cpp-1 bash -c /opt/project/ci.sh

ci.sh

#!/bin/bash

FILE_PATH=UnityLicenseFile.ulf
echo "$UNITY_LICENSE" | tr -d '\r' >$FILE_PATH
unity-editor \
    -batchmode \
    -nographics \
    -logFile /dev/stdout \
    -quit \
    -manualLicenseFile $FILE_PATH

platforms=(editmode playmode)

COVERAGE_OPTIONS="pathFilters:+/**/Assets/Scripts/**;assemblyFilters:+assets;generateAdditionalMetrics;generateHtmlReport;generateBadgeReport"

for platform in "${platforms[@]}"; do
    unity-editor \
        -runTests \
        -projectPath /opt/project \
        -testResults /opt/project/Logs/${platform}_test_results.xml \
        -logfile /dev/stdout \
        -debugCodeOptimization \
        -enableCodeCoverage \
        -testPlatform $platform \
        -coverageResultsPath /opt/project/CodeCoverage \
        -coverageOptions $COVERAGE_OPTIONS
done

[kcarlson]

this solves our issue

[ugnelis]

uses: game-ci/[email protected]

That worked, thanks!

[webbertakken]

Looks like one of the releases caused this. It would be extremely helpful if you could help us find out which one.

Do the versions v2.0.1 and v2.0.2 work for you too?

[Peaj]

I am using Unity version 2019.4.28f1.

The project does not have any playmode tests only editmode tests.

[email protected] it hangs at "Testing in Editmode"
test-runner@v2 it hangs at "Testing in Playmode"
[email protected] it hangs at "Testing in Playmode"

test-runner-log.txt

Trying to run the tests on window-latests also failed:

Run game-ci/unity-test-runner@v2
D:\a\_actions\game-ci\unity-test-runner\v2\dist\main.js:1
index.js
^

ReferenceError: index is not defined
    at Object.<anonymous> (D:\a\_actions\game-ci\unity-test-runner\v2\dist\main.js:1:1)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:17:47

Looks like the simlink is broken?

[dev-cmurphy]

I am using Unity version 2019.4.28f1.

The project does not have any playmode tests only editmode tests.

[email protected] it hangs at "Testing in Editmode" test-runner@v2 it hangs at "Testing in Playmode" [email protected] it hangs at "Testing in Playmode"

test-runner-log.txt

Trying to run the tests on window-latests also failed:

Run game-ci/unity-test-runner@v2
D:\a\_actions\game-ci\unity-test-runner\v2\dist\main.js:1
index.js
^

ReferenceError: index is not defined
    at Object.<anonymous> (D:\a\_actions\game-ci\unity-test-runner\v2\dist\main.js:1:1)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:17:47

Looks like the simlink is broken?

The windows issue is related to this issue it seems.

[oferei]

I also have this problem with play-mode tests. I tried the "2.0.0" solution but it doesn't work for me.

It would also be helpful to flush log messages as they happen instead of dumping them all at the end. Even before this bug the entire bunch of tests log messages would be dumped at one time point--after all the tests were complete--and all having almost the same time stamp.

[webbertakken]

Grouping and flushing output is tracked in game-ci/cli#12

Contributions are welcomed :)

[Peaj]

I just tried running the test on 2021.3.14f1 and instead of getting stuck on "Testing in Playmode" it fails when trying to fetch private git packages. I wonder if that might always be the reason it fails and just does not handle it properly in earlier versions.
@ugnelis it seems like you are using private git packages too as there is a "gitPrivateToken" in your config?

[oferei]

I just tried running the test on 2021.3.14f1 and instead of getting stuck on "Testing in Playmode" it fails when trying to fetch private git packages. I wonder if that might always be the reason it fails and just does not handle it properly in earlier versions. @ugnelis it seems like you are using private git packages too as there is a "gitPrivateToken" in your config?

I still have the bug with Unity 2021.3.14f1.

It sounds to me that your fetch failing should not have anything to do with the Unity version.

[ugnelis]

I just tried running the test on 2021.3.14f1 and instead of getting stuck on "Testing in Playmode" it fails when trying to fetch private git packages. I wonder if that might always be the reason it fails and just does not handle it properly in earlier versions. @ugnelis it seems like you are using private git packages too as there is a "gitPrivateToken" in your config?

Probably, you need this step for running test (important line gitPrivateToken):

# Test
- name: Run Tests
  uses: game-ci/unity-test-runner@v2
  id: tests
  env:
    UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
  with:
    gitPrivateToken: ${{ secrets.GIT_PRIVATE_TOKEN }}
    githubToken: ${{ secrets.GITHUB_TOKEN }}

or:

# Build
- uses: game-ci/unity-builder@v2
  env:
    UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
  with:
    gitPrivateToken: ${{ secrets.GIT_PRIVATE_TOKEN }}
    targetPlatform: ${{ matrix.targetPlatform }}
    buildsPath: 'Builds'
    buildName: ${{ needs.setup.outputs.completeName }}

Of course, you need to add Github secret token which has access of private repos (https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)

[ugnelis]

I just tried running the test on 2021.3.14f1 and instead of getting stuck on "Testing in Playmode" it fails when trying to fetch private git packages. I wonder if that might always be the reason it fails and just does not handle it properly in earlier versions. @ugnelis it seems like you are using private git packages too as there is a "gitPrivateToken" in your config?

I still have the bug with Unity 2021.3.14f1.

It sounds to me that your fetch failing should not have anything to do with the Unity version.

Tested yesterday game-ci/unity-builder@v2 and game-ci/[email protected], everything works on the Unity version 2020.3.19f1.

[Peaj]

Probably, you need this step for running test (important line gitPrivateToken):

I do have the gitPrivateToken. I am using the same config that works for the builder:

# Test
      - name: Test 🔬
        uses: game-ci/unity-test-runner@v2
        env:
          UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
          UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
          UNITY_SERIAL: ${{ secrets.UNITY_SERIAL }}
        with:
          githubToken: ${{ secrets.GITHUB_TOKEN }}
          gitPrivateToken: ${{ secrets.PRIVATE_TOKEN }}
          customParameters: -username='${{ secrets.USERNAME }}' -password='${{ secrets.PASSWORD }}'

[ugnelis]

Probably, you need this step for running test (important line gitPrivateToken):

I do have the gitPrivateToken. I am using the same config that works for the builder:

# Test
      - name: Test 🔬
        uses: game-ci/unity-test-runner@v2
        env:
          UNITY_EMAIL: ${{ secrets.UNITY_EMAIL }}
          UNITY_PASSWORD: ${{ secrets.UNITY_PASSWORD }}
          UNITY_SERIAL: ${{ secrets.UNITY_SERIAL }}
        with:
          githubToken: ${{ secrets.GITHUB_TOKEN }}
          gitPrivateToken: ${{ secrets.PRIVATE_TOKEN }}
          customParameters: -username='${{ secrets.USERNAME }}' -password='${{ secrets.PASSWORD }}'

If possible, please send here your log file. Also, be sure that you gave enough access for you generated Github access token

[Peaj]

The log just says:
Error when executing git command. remote: Invalid username or password
For my 3 git hosted packages. But its the same token and user data that work for the build step.

[ugnelis]

Not sure how you are fetching git projects but in our case, we do through UPM as far as I remember. If you are fetching through git directly, this might be useful https://stackoverflow.com/a/73229045/2122457

Just in case I put my current CI configuration that might be useful:

# See: https://game.ci/docs/github/getting-started
name: CI

on:
  workflow_dispatch: { }
  push:
    branches: [ master ]
  pull_request:
    types: [review_requested, ready_for_review]
    branches: [ master ]

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  test:
    name: Unity Tests Workflow
    runs-on: ubuntu-latest
    timeout-minutes: 60

    strategy:
      matrix:
        unityVersion:
          - 2020.3.19f1
        testMode:
#          - playmode
          - editmode

    steps:

      # Checkout (without LFS)
      - name: Checkout Repository
        uses: actions/checkout@v2

      # Git LFS
      - name: Create LFS file list
        run: git lfs ls-files -l | cut -d' ' -f1 | sort > .lfs-assets-id

      - name: Restore LFS cache
        uses: actions/cache@v2
        id: lfs-cache
        with:
          path: .git/lfs
          key: ${{ runner.os }}-lfs-${{ hashFiles('.lfs-assets-id') }}

      - name: Git LFS Pull
        run: |
          git lfs pull
          git add .
          git reset --hard

      # Cache
      - uses: actions/cache@v2
        with:
          path: Library
          key: Library-${{ hashFiles('Assets/**', 'Packages/**', 'ProjectSettings/**') }}
          restore-keys: |
            Library-

      # Test
      - name: Run Tests
        uses: game-ci/[email protected]
        id: tests
        env:
          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
        with:
          gitPrivateToken: ${{ secrets.GIT_PRIVATE_TOKEN }}
          githubToken: ${{ secrets.GITHUB_TOKEN }}

      # Test Results
      - name: Upload Test Results
        uses: actions/upload-artifact@v2
        if: always()
        with:
          name: Test Restuls
          path: ${{ steps.tests.outputs.artifactsPath }}

If you still experiencing some problems, If I were you, I would try to run Github Actions locally (for example, https://github.com/nektos/act) and try to debug your DevOps scenario on a local machine.

[ugnelis]

I talked with my friend, so in our project, we are not using other private git repos anymore, we have embedded them into Unity project. We have done that as not all team members use Git or have access to all private repos.

In your case, you need to follow game-ci docs on how to set github private repos fetching.

[Peaj]

I talked with my friend, so in our project, we are not using other private git repos anymore, we have embedded them into Unity project. We have done that as not all team members use Git or have access to all private repos.

In your case, you need to follow game-ci docs on how to set github private repos fetching.

Thanks for your help. As mentioned the same settings work for the Build step to get the private packages.
Running the actions locally is a good idea. I will try that out. Also to figure out the original issue.

[oferei]

I'd like to return to the error in the OP - the test runner gets stuck.
Any info I can add to help with handling this issue?

[ugnelis]

I'd like to return to the error in the OP - the test runner gets stuck. Any info I can add to help with handling this issue?

Have you tried uses: game-ci/[email protected] (putting v2.0.0 instead of v2)?

[oferei]

Yes. Unfortunately it didn't help in my case. (I wrote this above)

[kenorb]

I've got the same hang with game-ci/unity-test-runner@v2. See: https://github.com/game-ci/unity-actions/issues/168
I was able to reproduce it on the empty project with a public repository right after cache was restored. And I'm using Linux runner.

Build: https://github.com/Ecilos/ecilos-unity-assets-sandbox/actions/runs/4288532708/jobs/7470635266#step:4:141

Workflow file: https://github.com/Ecilos/ecilos-unity-assets-sandbox/actions/runs/4288532708/workflow

[ugnelis]

@kenorb try:

      - name: Set up Unity
        uses: game-ci/[email protected] # <--
        env:
          UNITY_LICENSE: ${{ secrets.UNITY_LICENSE }}
        with:
          projectPath: .
          githubToken: ${{ secrets.GITHUB_TOKEN }}

[kenorb]

      - name: Set up Unity
        uses: game-ci/[email protected] # <--

Thanks. Seems like it worked using v2.0.0 as per build 7529977372. I'm going to check for other repositories to make sure. Update: 2nd repo with more asset files worked fine as well (before it didn't).

Since v2.0.0 is older, do you know what's the (known) issue with the newer version?

[sjdharme]

I am also facing this issue with playmode tests. I am using unity version 2020.3.37f1. I tried using v2.0.0 but it still gets stuck in the "Testing in playmode" phase.

[davidmfinol]

Any chance your tests involve a coroutine that has yield return null; in it? I've seen that cause it to hang sometimes.

[sjdharme]

Yes, there are multiple coroutines which has yield return null; in it. Since I am new to the unity world, I don't know what will come in place of yeild return null;.
I tried adding -nographics and -quit as a customParameters and the build passed successfully BUT it did not run any tests.
✅ PlayMode Test Results - 0/0 - Passed in 0.000s

[asyasmi]

Hey,
Suddenly started to get this

###########################
#   Testing in editmode  #
###########################

again, tested v2, v2.0.0, v2.0.1, till v2.1.1 but with no luck :(
I use 2020.3.36f1, 2021.3.21f1, 2022.2.12f1.

Somehow fixed it the last time, but here it happens again.

[ugnelis]

@asyasmi what output do you get?

[ugnelis]

To this day I am using v2.0.0 works like a charm.

Some advice from experience using this action:

• Check Github Actions timeout-minutes are enough for your project. In my case, we have set timeout-minutes: 60
• Check on your computer if a clean build doesn't take too long. One of the problems might be related to shader variances.

[timcassell]

I'm not sure if the progress logs get printed to the Unity log, but I do this in my tests to see what test caused a hang (works in my Core tests).

public static class TestHelper
{
    private static Stopwatch _stopwatch;

    public static void Setup()
    {
        if (_stopwatch == null)
        {
            // Other global setup
            _stopwatch = Stopwatch.StartNew();
        }
        // Other per-test setup
        TestContext.Progress.WriteLine("Begin time: " + _stopwatch.Elapsed.ToString() + ", test: " + TestContext.CurrentContext.Test.FullName);
    }

    public static void Cleanup()
    {
        // Other per-test cleanup
        TestContext.Progress.WriteLine("Success time: " + _stopwatch.Elapsed.ToString() + ", test: " + TestContext.CurrentContext.Test.FullName);
    }
}

Then in all of my test classes, I put this:

[SetUp]
public void Setup()
{
    TestHelper.Setup();
}

[TearDown]
public void Teardown()
{
    TestHelper.Cleanup();
}

[asyasmi]

Thanks, @timcassell !
I'll think of implementing something similar on our side.