Cypress tests not reporting correctly in CI Testing Visibility with `cypress-fail-fast` about dd-trace-js

[MaximilianoAdaro]

Currently, we have a NX project configuration. Just adding the --headed flag it runs it in that way, resulting in cypress run --headed

[MaximilianoAdaro]

Thanks Juan for your support.
As you still couldn't replicate that issue, I'll be working on a side-project in order to help you to reproduce it.

[juan-fernandez]

Juan, firstly, I want to extend my apologies for the delay in getting back to you. After dedicating some time to thorough debugging, I've successfully identified the root cause of our issue.

It appears that for the integration with the Datadog CI Visibility Test service via dd-trace, it's essential not only to avoid overriding the after:run event but also the after:spec event.

Current Challenges:

1. Our setup includes the cypress-split plugin (found here: cypress-split plugin), which utilizes the after:spec event in Cypress. This has resulted in a conflict with the dd-trace implementation.
2. Additionally, we have implemented custom logic that overrides this particular event within Cypress, further compounding the issue.

To aid in the diagnosis and facilitate a more straightforward debugging process, I've prepared a replicable side-project, which you can find here: Replicable Side-Project. This project is designed to mimic the error in a controlled environment.

Proposed Solution:

Given how effectively you've managed to expose the after:run event—resulting in seamless functionality—might I suggest applying a similar approach to the after:spec event?

Thank you for your time and assistance thus far!

thank you so much for the context and the reproducible case! I think we can indeed do something similar for the after:spec event. Let me investigate a bit and get back you

[MaximilianoAdaro]

Hey @juan-fernandez,

I just tested the configuration change you suggested, and it worked! By using version dd-trace v4.30.0 and adjusting what you advised, we're now seeing the test failure statuses reported correctly in Datadog, even when running tests in parallel.

This has been a perplexing issue. I deeply appreciate the guidance and support you've provided throughout this troubleshooting process.

Thank you for sticking with us on this and for the timely and effective solutions. Your assistance has been invaluable.

Best!

[juan-fernandez]

thank you for your patience and help, @MaximilianoAdaro ! This has been very useful to us, to learn how to make our plugin better.

This is not ideal and we need to find a better way to handle this but in the meantime it should fix your issue.

I'll keep you updated with the changes we do to improve this 😄

[juan-fernandez]

hey @MaximilianoAdaro ! Thanks for the report.

Sadly I can't reproduce the issue. With the following setup:

I get the test results just fine (tests pass until the first one fails, then they're all skipped):

Is there any other plugin that could be causing issues? A small reproducible case would be extra useful 😄

The command I execute is simply cypress run:

"cypress:run": "cypress run",

[MaximilianoAdaro]

Hey Juan, thanks for the quick support.

Could you please try Cypress Open instead?

[juan-fernandez]

Hey Juan, thanks for the quick support.

Could you please try Cypress Open instead?

Cypress open is not supported, sadly: https://docs.datadoghq.com/tests/setup/javascript/?tab=cloudciprovideragentless#cypress-interactive-mode.

We target not breaking the user tests and attempt to report data, but the way events are fired is not reliable - there's no way (that I see) to make it work reliably.

Do you use cypress open in your CI?

[MaximilianoAdaro]

Yes, we use cypress open in our CI.

We were forced to use cypress open mode in those apps using Auth0 for the following reasons:

• To bypass Auth0 “authorize” step, we need to avoid using localhost domains
• To use a custom domain in a local environment (in this case the CI runner), we need to:
  • Add the mapping in etc/hosts file
  • Auth0 library will complain about custom domains without https protocol. To prevent that we need Cypress to initialize Chrome with --unsafely-treat-insecure-origin-as-secure security flag
• Seems like cypress run won’t work when providing these custom security flags configuration, so we are forced to use cypress open to make it work

[juan-fernandez]

Yes, we use cypress open in our CI.

We were forced to use cypress open mode in those apps using Auth0 for the following reasons:

• To bypass Auth0 “authorize” step, we need to avoid using localhost domains

• To use a custom domain in a local environment (in this case the CI runner), we need to:

  • Add the mapping in etc/hosts file
  • Auth0 library will complain about custom domains without https protocol. To prevent that we need Cypress to initialize Chrome with --unsafely-treat-insecure-origin-as-secure security flag

• Seems like cypress run won’t work when providing these custom security flags configuration, so we are forced to use cypress open to make it work

that's an interesting use case, thanks for the detail. I haven't been able to use cypress open without actual human intervention. How do you set up cypress to run automatically with cypress open?

[juan-fernandez]

Running without dd-trace

Cypress config is

module.exports = defineConfig({
  e2e: {
    setupNodeEvents: (on, config) => {
      require("cypress-fail-fast/plugin")(on, config);
      return config
    },
    supportFile: 'cypress/support/index.js',
    specPattern: 'cypress/integration/**/*.spec.js',
    baseUrl: "http://localhost:3000",
    experimentalInteractiveRunEvents: true
  }
})

and support file:

import "cypress-fail-fast";

If I run:

yarn run cypress run --headed

I see the tests in headed mode and I get these results:

       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✖  barcelona.spec.js                        00:03        3        1        1        -        1 │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  colon.spec.js                            00:04        4        4        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  madrid.spec.js                           00:04        4        4        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  sevilla.spec.js                          00:04        4        4        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  1 of 4 failed (25%)                      00:17       15       13        1        -        1 

Running with dd-trace

Cypress config is

module.exports = defineConfig({
  e2e: {
    setupNodeEvents: (on, config) => {
      require('cypress-fail-fast/plugin')(on, config);
      require('dd-trace/ci/cypress/plugin')(on, config)
      on('after:run', (details) => {
        return require('dd-trace/ci/cypress/after-run')(details)
      })
      return config
    },
    supportFile: 'cypress/support/index.js',
    specPattern: 'cypress/integration/**/*.spec.js',
    baseUrl: "http://localhost:3000",
    experimentalInteractiveRunEvents: true
  }
})

and support file:

import "cypress-fail-fast";
require('dd-trace/ci/cypress/support')

If I run:

DD_SERVICE=testing-suite-cypress yarn run cypress run --headed

I see the tests in headed mode and I get these results:

       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✖  barcelona.spec.js                        00:02        3        1        1        -        1 │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  colon.spec.js                            00:04        4        4        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  madrid.spec.js                           00:04        4        4        -        -        - │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✔  sevilla.spec.js                          00:04        4        4        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  1 of 4 failed (25%)                      00:17       15       13        1        -        1  

I can also see the tests reported to datadog:

It looks to me like the plugin is behaving correctly. What am I missing?

[MaximilianoAdaro]

Juan, firstly, I want to extend my apologies for the delay in getting back to you. After dedicating some time to thorough debugging, I've successfully identified the root cause of our issue.

It appears that for the integration with the Datadog CI Visibility Test service via dd-trace, it's essential not only to avoid overriding the after:run event but also the after:spec event.

Current Challenges:

1. Our setup includes the cypress-split plugin (found here: cypress-split plugin), which utilizes the after:spec event in Cypress. This has resulted in a conflict with the dd-trace implementation.
2. Additionally, we have implemented custom logic that overrides this particular event within Cypress, further compounding the issue.

To aid in the diagnosis and facilitate a more straightforward debugging process, I've prepared a replicable side-project, which you can find here: Replicable Side-Project. This project is designed to mimic the error in a controlled environment.

Proposed Solution:

Given how effectively you've managed to expose the after:run event—resulting in seamless functionality—might I suggest applying a similar approach to the after:spec event?

Thank you for your time and assistance thus far!

[juan-fernandez]

@MaximilianoAdaro

I think the problem is indeed solved by exposing after:spec. I've done just that in #4127.

I've tested it with your reproducible case by installing the PR's branch: MaximilianoAdaro/cypress-config#1. I think the problem you're seeing is solved:

Test runs before:

Test runs after:

Test suites before:

Test suites after:

Give it a try and let me know if this works for you.

[MaximilianoAdaro]

Hey @juan-fernandez,

Thank you for the quick turnaround and the effort to address the issue with the after:spec exposure – it's much appreciated.

Your fix worked well in the replicable environment and did report the passed tests, which is a significant step forward. However, in our actual setup where we run tests in parallel, we're still not seeing the failures reported as expected. They seem to be marked as skipped instead. Datadog Results

I'm planning to incorporate our custom parallelization setup into the replicable environment to see if we can mimic the behavior observed in our production setup. This might help us pinpoint why the failures aren't being reported correctly when running in parallel.

Our setup also runs Cypress in headed mode with the fail-fast option enabled, which necessitates some customization on our part. I'll share more details as I progress with replicating this parallel testing scenario.

Thanks again for your support. I'll get back to you with any findings or further questions.

Best regards,
Maximiliano

[juan-fernandez]

Hey @juan-fernandez,

Thank you for the quick turnaround and the effort to address the issue with the after:spec exposure – it's much appreciated.

Your fix worked well in the replicable environment and did report the passed tests, which is a significant step forward. However, in our actual setup where we run tests in parallel, we're still not seeing the failures reported as expected. They seem to be marked as skipped instead. Datadog Results

I'm planning to incorporate our custom parallelization setup into the replicable environment to see if we can mimic the behavior observed in our production setup. This might help us pinpoint why the failures aren't being reported correctly when running in parallel.

Our setup also runs Cypress in headed mode with the fail-fast option enabled, which necessitates some customization on our part. I'll share more details as I progress with replicating this parallel testing scenario.

Thanks again for your support. I'll get back to you with any findings or further questions.

Best regards, Maximiliano

hey, it's great to hear that it's getting better. Could you get the Test suite logic.spec.ts? (you can change the test level in the dropdown left to the seach bar). Does the test suite report the error? If a test suite fails in a hook, there might not be any test runs after the error (even if they show up as errored in cypress, they still didn't run, hence the "skipped" status)

[MaximilianoAdaro]

@juan-fernandez

Indeed, the Test Suite is shown as a failure

[MaximilianoAdaro]

In the GitHub Actions run, a test titled “Login automatically redirects…” failed, but this wasn't reflected in the Datadog dashboard. Instead, it was marked as skipped.

I've enabled additional logging and inspected the Cypress results sent to the dd-trace after-spec command. The logs indicate a failure in that test:

portal-e2e: CONSOLE: Cypress after spec ***
portal-e2e:   error: null,
portal-e2e:   reporter: 'spec',
portal-e2e:   reporterStats: ***
portal-e2e:     suites: 1,
portal-e2e:     tests: 6,
portal-e2e:     passes: 5,
portal-e2e:     pending: 0,
portal-e2e:     failures: 1,
portal-e2e:     start: '2024-03-04T19:10:18.244Z',
portal-e2e:     end: '2024-03-04T19:11:10.226Z',
portal-e2e:     duration: 51982
portal-e2e:   ***,
portal-e2e:   screenshots: [...],
portal-e2e:   spec: ***
portal-e2e:     absolute: '/runner/_work/client-monorepo/client-monorepo/apps/portal-e2e/src/integration/standard/login.spec.ts',
portal-e2e:     fileExtension: '.ts',
portal-e2e:     fileName: 'login',
portal-e2e:     name: 'login.spec.ts',
portal-e2e:     relative: 'src/integration/standard/login.spec.ts'
portal-e2e:   ***,
portal-e2e:   stats: ***
portal-e2e:     duration: 51984,
portal-e2e:     endedAt: '2024-03-04T19:11:10.215Z',
portal-e2e:     failures: 1,
portal-e2e:     passes: 5,
portal-e2e:     pending: 0,
portal-e2e:     skipped: 1,
portal-e2e:     startedAt: '2024-03-04T19:10:18.231Z',
portal-e2e:     suites: 1,
portal-e2e:     tests: 7
portal-e2e:   ***,
portal-e2e:   tests: [
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 12572,
portal-e2e:       state: 'passed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 6480,
portal-e2e:       state: 'passed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 4744,
portal-e2e:       state: 'passed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 7363,
portal-e2e:       state: 'passed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 4015,
portal-e2e:       state: 'passed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: 'TypeError: Cannot set property message of [object DOMException] which has only a getter\n' +
portal-e2e:         '\n' +
portal-e2e:         'Because this error occurred during a `after each` hook we are skipping all of the remaining tests.\n' +
portal-e2e:         '    at modifyErrMsg (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:75090:15)\n' +
portal-e2e:         '    at Object.appendErrMsg (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:75095:10)\n' +
portal-e2e:         '    at Runner.<anonymous> (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:162674:68)\n' +
portal-e2e:         '    at Runner.emit (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:146414:7)\n' +
portal-e2e:         '    at Runner.fail (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:155613:8)\n' +
portal-e2e:         '    at Runner.fail (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:145675:25)\n' +
portal-e2e:         '    at Runner.failHook (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:155651:8)\n' +
portal-e2e:         '    at Hook.<anonymous> (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:155725:14)\n' +
portal-e2e:         '    at next (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:163064:24)\n' +
portal-e2e:         '    at http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:163091:13\n' +
portal-e2e:         '    at tryCatcher (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1807:23)\n' +
portal-e2e:         '    at Promise._settlePromiseFromHandler (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1519:31)\n' +
portal-e2e:         '    at Promise._settlePromise (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1576:18)\n' +
portal-e2e:         '    at Promise._settlePromise0 (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1621:10)\n' +
portal-e2e:         '    at Promise._settlePromises (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1701:18)\n' +
portal-e2e:         '    at Promise._fulfill (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1645:18)\n' +
portal-e2e:         '    at Promise._resolveCallback (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1439:57)\n' +
portal-e2e:         '    at Promise._settlePromiseFromHandler (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1531:17)\n' +
portal-e2e:         '    at Promise._settlePromise (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1576:18)\n' +
portal-e2e:         '    at Promise._settlePromise0 (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1621:10)\n' +
portal-e2e:         '    at Promise._settlePromises (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1701:18)\n' +
portal-e2e:         '    at Promise._fulfill (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1645:18)',
portal-e2e:       duration: 16381,
portal-e2e:       state: 'failed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 0,
portal-e2e:       state: 'skipped',
portal-e2e:       title: [Array]
portal-e2e:     ***
portal-e2e:   ],
portal-e2e:   video: null

[juan-fernandez]

In the GitHub Actions run, a test titled “Login automatically redirects…” failed, but this wasn't reflected in the Datadog dashboard. Instead, it was marked as skipped.

I've enabled additional logging and inspected the Cypress results sent to the dd-trace after-spec command. The logs indicate a failure in that test:

portal-e2e: CONSOLE: Cypress after spec ***
portal-e2e:   error: null,
portal-e2e:   reporter: 'spec',
portal-e2e:   reporterStats: ***
portal-e2e:     suites: 1,
portal-e2e:     tests: 6,
portal-e2e:     passes: 5,
portal-e2e:     pending: 0,
portal-e2e:     failures: 1,
portal-e2e:     start: '2024-03-04T19:10:18.244Z',
portal-e2e:     end: '2024-03-04T19:11:10.226Z',
portal-e2e:     duration: 51982
portal-e2e:   ***,
portal-e2e:   screenshots: [...],
portal-e2e:   spec: ***
portal-e2e:     absolute: '/runner/_work/client-monorepo/client-monorepo/apps/portal-e2e/src/integration/standard/login.spec.ts',
portal-e2e:     fileExtension: '.ts',
portal-e2e:     fileName: 'login',
portal-e2e:     name: 'login.spec.ts',
portal-e2e:     relative: 'src/integration/standard/login.spec.ts'
portal-e2e:   ***,
portal-e2e:   stats: ***
portal-e2e:     duration: 51984,
portal-e2e:     endedAt: '2024-03-04T19:11:10.215Z',
portal-e2e:     failures: 1,
portal-e2e:     passes: 5,
portal-e2e:     pending: 0,
portal-e2e:     skipped: 1,
portal-e2e:     startedAt: '2024-03-04T19:10:18.231Z',
portal-e2e:     suites: 1,
portal-e2e:     tests: 7
portal-e2e:   ***,
portal-e2e:   tests: [
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 12572,
portal-e2e:       state: 'passed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 6480,
portal-e2e:       state: 'passed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 4744,
portal-e2e:       state: 'passed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 7363,
portal-e2e:       state: 'passed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 4015,
portal-e2e:       state: 'passed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: 'TypeError: Cannot set property message of [object DOMException] which has only a getter\n' +
portal-e2e:         '\n' +
portal-e2e:         'Because this error occurred during a `after each` hook we are skipping all of the remaining tests.\n' +
portal-e2e:         '    at modifyErrMsg (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:75090:15)\n' +
portal-e2e:         '    at Object.appendErrMsg (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:75095:10)\n' +
portal-e2e:         '    at Runner.<anonymous> (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:162674:68)\n' +
portal-e2e:         '    at Runner.emit (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:146414:7)\n' +
portal-e2e:         '    at Runner.fail (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:155613:8)\n' +
portal-e2e:         '    at Runner.fail (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:145675:25)\n' +
portal-e2e:         '    at Runner.failHook (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:155651:8)\n' +
portal-e2e:         '    at Hook.<anonymous> (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:155725:14)\n' +
portal-e2e:         '    at next (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:163064:24)\n' +
portal-e2e:         '    at http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:163091:13\n' +
portal-e2e:         '    at tryCatcher (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1807:23)\n' +
portal-e2e:         '    at Promise._settlePromiseFromHandler (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1519:31)\n' +
portal-e2e:         '    at Promise._settlePromise (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1576:18)\n' +
portal-e2e:         '    at Promise._settlePromise0 (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1621:10)\n' +
portal-e2e:         '    at Promise._settlePromises (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1701:18)\n' +
portal-e2e:         '    at Promise._fulfill (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1645:18)\n' +
portal-e2e:         '    at Promise._resolveCallback (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1439:57)\n' +
portal-e2e:         '    at Promise._settlePromiseFromHandler (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1531:17)\n' +
portal-e2e:         '    at Promise._settlePromise (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1576:18)\n' +
portal-e2e:         '    at Promise._settlePromise0 (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1621:10)\n' +
portal-e2e:         '    at Promise._settlePromises (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1701:18)\n' +
portal-e2e:         '    at Promise._fulfill (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1645:18)',
portal-e2e:       duration: 16381,
portal-e2e:       state: 'failed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 0,
portal-e2e:       state: 'skipped',
portal-e2e:       title: [Array]
portal-e2e:     ***
portal-e2e:   ],
portal-e2e:   video: null

thanks for the detail! I'll review the code around error handling

[juan-fernandez]

I'm planning to incorporate our custom parallelization setup into the replicable environment to see if we can mimic the behavior observed in our production setup. This might help us pinpoint why the failures aren't being reported correctly when running in parallel.

that'd be very useful 😄

[juan-fernandez]

I've pushed a branch to add an after each error to your reproducible case:
MaximilianoAdaro/cypress-config@master...juan-fernandez:cypress-config:juan-fernandez/fix-cypress-config

I'm using the dev release now:

yarn add --dev dd-trace@dev

When I run the tests

yarn cypress:run:headed

I get:

       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✖  examples.spec.ts                         00:01        3        -        1        -        2 │
  ├────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ ✖  test.spec.ts                             00:01        3        1        1        -        1 │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  2 of 2 failed (100%)                     00:02        6        1        2        -        3 

and the results look correct in datadog.

so there must be something else 😅

[MaximilianoAdaro]

Hey @juan-fernandez,

Thanks for the update and for pushing the new branch to add an after each error to the reproducible case.

I've gone ahead and updated the reproducible project with your changes and tested it using the dd-trace@dev channel. Interestingly, I'm not seeing the same results as in my main project. In the main project setup, despite the Cypress after-spec sending a failure status, these are appearing as skipped in the Datadog dashboard, which is quite puzzling.

It seems there's a disconnect somewhere between what Cypress reports and what Datadog ultimately logs. Since the issue doesn't replicate in the controlled environment, I'm inclined to think there might be another variable at play in our full setup that's not present in the reproducible case.

I'm still working to mimic the full environment as closely as possible in the reproducible case (MaximilianoAdaro/cypress-config), but as of now, I haven't been able to replicate the discrepancy in test reporting.

Could you possibly take another look from your end to see if there's something we might be overlooking? Any additional help or suggestions you could offer would be immensely helpful as we continue to troubleshoot this issue.

[MaximilianoAdaro]

portal-e2e:     (Attempt 1 of 3) automatically redirects to Google SSO auth if query param is passed
portal-e2e:     ✓ pending invitation is displayed correctly (6964ms)
portal-e2e:     (Attempt 2 of 3) automatically redirects to Google SSO auth if query param is passed
portal-e2e:     ✓ invitation awaiting verification is displayed correctly (7788ms)
portal-e2e:     1) "after each" hook for "automatically redirects to Google SSO auth if query param is passed"
portal-e2e:   5 passing (50s)
portal-e2e:   1 failing
portal-e2e:   1) Login
portal-e2e:        "after each" hook for "automatically redirects to Google SSO auth if query param is passed":
portal-e2e:      TypeError: Cannot set property message of [object DOMException] which has only a getter
portal-e2e: Because this error occurred during a `after each` hook we are skipping all of the remaining tests.
portal-e2e:       at modifyErrMsg (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:75090:15)
portal-e2e:       at Object.appendErrMsg (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:75095:10)
portal-e2e:       at Runner.<anonymous> (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:162674:68)
portal-e2e:       at Runner.emit (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:146414:7)
portal-e2e:       at Runner.fail (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:155613:8)
portal-e2e:       at Runner.fail (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:145675:25)
portal-e2e:       at Runner.failHook (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:155651:8)
portal-e2e:       at Hook.<anonymous> (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:155725:14)
portal-e2e:       at next (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:163064:24)
portal-e2e:       at http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:163091:13
portal-e2e:       at tryCatcher (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1807:23)
portal-e2e:       at Promise._settlePromiseFromHandler (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1519:31)
portal-e2e:       at Promise._settlePromise (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1576:18)
portal-e2e:       at Promise._settlePromise0 (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1621:10)
portal-e2e:       at Promise._settlePromises (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1701:18)
portal-e2e:       at Promise._fulfill (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1645:18)
portal-e2e:       at Promise._resolveCallback (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1439:57)
portal-e2e:       at Promise._settlePromiseFromHandler (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1531:17)
portal-e2e:       at Promise._settlePromise (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1576:18)
portal-e2e:       at Promise._settlePromise0 (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1621:10)
portal-e2e:       at Promise._settlePromises (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1701:18)
portal-e2e:       at Promise._fulfill (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1645:18)
portal-e2e:
portal-e2e: CONSOLE: Cypress after spec ***
portal-e2e:   error: null,
portal-e2e:   reporter: 'spec',
portal-e2e:   reporterStats: ***
portal-e2e:     suites: 1,
portal-e2e:     tests: 6,
portal-e2e:     passes: 5,
portal-e2e:     pending: 0,
portal-e2e:     failures: 1,
portal-e2e:     start: '2024-03-06T00:41:34.924Z',
portal-e2e:     end: '2024-03-06T00:42:24.755Z',
portal-e2e:     duration: 49831
portal-e2e:   ***,
portal-e2e:   screenshots: [],
portal-e2e:   spec: ***
portal-e2e:     absolute: '/runner/_work/client-monorepo/client-monorepo/apps/portal-e2e/src/integration/standard/login.spec.ts',
portal-e2e:     fileExtension: '.ts',
portal-e2e:     fileName: 'login',
portal-e2e:     name: 'login.spec.ts',
portal-e2e:     relative: 'src/integration/standard/login.spec.ts'
portal-e2e:   ***,
portal-e2e:   stats: ***
portal-e2e:     duration: 49839,
portal-e2e:     endedAt: '2024-03-06T00:42:24.748Z',
portal-e2e:     failures: 1,
portal-e2e:     passes: 5,
portal-e2e:     pending: 0,
portal-e2e:     skipped: 1,
portal-e2e:     startedAt: '2024-03-06T00:41:34.909Z',
portal-e2e:     suites: 1,
portal-e2e:     tests: 7
portal-e2e:   ***,
portal-e2e:   tests: [
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 12817,
portal-e2e:       state: 'passed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 7159,
portal-e2e:       state: 'passed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 5216,
portal-e2e:       state: 'passed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 7257,
portal-e2e:       state: 'passed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 2742,
portal-e2e:       state: 'passed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: 'TypeError: Cannot set property message of [object DOMException] which has only a getter\n' +
portal-e2e:         '\n' +
portal-e2e:         'Because this error occurred during a `after each` hook we are skipping all of the remaining tests.\n' +
portal-e2e:         '    at modifyErrMsg (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:75090:15)\n' +
portal-e2e:         '    at Object.appendErrMsg (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:75095:10)\n' +
portal-e2e:         '    at Runner.<anonymous> (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:162674:68)\n' +
portal-e2e:         '    at Runner.emit (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:146414:7)\n' +
portal-e2e:         '    at Runner.fail (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:155613:8)\n' +
portal-e2e:         '    at Runner.fail (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:145675:25)\n' +
portal-e2e:         '    at Runner.failHook (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:155651:8)\n' +
portal-e2e:         '    at Hook.<anonymous> (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:155725:14)\n' +
portal-e2e:         '    at next (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:163064:24)\n' +
portal-e2e:         '    at http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:163091:13\n' +
portal-e2e:         '    at tryCatcher (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1807:23)\n' +
portal-e2e:         '    at Promise._settlePromiseFromHandler (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1519:31)\n' +
portal-e2e:         '    at Promise._settlePromise (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1576:18)\n' +
portal-e2e:         '    at Promise._settlePromise0 (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1621:10)\n' +
portal-e2e:         '    at Promise._settlePromises (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1701:18)\n' +
portal-e2e:         '    at Promise._fulfill (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1645:18)\n' +
portal-e2e:         '    at Promise._resolveCallback (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1439:57)\n' +
portal-e2e:         '    at Promise._settlePromiseFromHandler (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1531:17)\n' +
portal-e2e:         '    at Promise._settlePromise (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1576:18)\n' +
portal-e2e:         '    at Promise._settlePromise0 (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1621:10)\n' +
portal-e2e:         '    at Promise._settlePromises (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1701:18)\n' +
portal-e2e:         '    at Promise._fulfill (http://dev.capchase.com:4202/__cypress/runner/cypress_runner.js:1645:18)',
portal-e2e:       duration: 14267,
portal-e2e:       state: 'failed',
portal-e2e:       title: [Array]
portal-e2e:     ***,
portal-e2e:     ***
portal-e2e:       attempts: [Array],
portal-e2e:       displayError: null,
portal-e2e:       duration: 0,
portal-e2e:       state: 'skipped',
portal-e2e:       title: [Array]
portal-e2e:     ***
portal-e2e:   ],
portal-e2e:   video: null
portal-e2e: ***
portal-e2e:   (Results)
portal-e2e:   ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
portal-e2e:   │ Tests:        7                                                                                │
portal-e2e:   │ Passing:      5                                                                                │
portal-e2e:   │ Failing:      1                                                                                │
portal-e2e:   │ Pending:      0                                                                                │
portal-e2e:   │ Skipped:      1                                                                                │
portal-e2e:   │ Screenshots:  3                                                                                │
portal-e2e:   │ Video:        false                                                                            │
portal-e2e:   │ Duration:     49 seconds                                                                       │
portal-e2e:   │ Spec Ran:     login.spec.ts                                                                    │
portal-e2e:   └────────────────────────────────────────────────────────────────────────────────────────────────┘
portal-e2e:   (Screenshots)
portal-e2e:   -  /runner/_work/client-monorepo/client-monorepo/dist/cypress/apps/portal-e2e/scree     (1050x882)
portal-e2e:      nshots/login.spec.ts/automatically redirects to Google SSO auth if query param i
portal-e2e:      s passed -- after each hook (failed).png
portal-e2e:   -  /runner/_work/client-monorepo/client-monorepo/dist/cypress/apps/portal-e2e/scree     (1050x882)
portal-e2e:      nshots/login.spec.ts/automatically redirects to Google SSO auth if query param i
portal-e2e:      s passed -- after each hook (failed) (attempt 2).png
portal-e2e:   -  /runner/_work/client-monorepo/client-monorepo/dist/cypress/apps/portal-e2e/scree     (1050x882)
portal-e2e:      nshots/login.spec.ts/automatically redirects to Google SSO auth if query param i
portal-e2e:      s passed -- after each hook (failed) (attempt 3).png
portal-e2e: CONSOLE: Cypress after run
portal-e2e: CONSOLE Slack: Sending results to Slack
portal-e2e: ====================================================================================================
portal-e2e:   (Run Finished)
portal-e2e:        Spec                                              Tests  Passing  Failing  Pending  Skipped
portal-e2e:   ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
portal-e2e:   │ ✖  login.spec.ts                            00:49        7        5        1        -        1 │
portal-e2e:   └────────────────────────────────────────────────────────────────────────────────────────────────┘
portal-e2e:     ✖  1 of 1 failed (100%)                     00:49        7        5        1        -        1
portal-e2e: Error: [FAIL FAST] - A thread finished with 'failed' status
portal-e2e:     at /runner/_work/client-monorepo/client-monorepo/libs/cypress-runner/src/executors/run-parallel/executor.ts:170:17
portal-e2e:     at async Promise.all (index 1)
portal-e2e:     at runParallel (/runner/_work/client-monorepo/client-monorepo/libs/cypress-runner/src/executors/run-parallel/executor.ts:146:21)

Although, the Suite is reported as failed, but with Error message empty

[juan-fernandez]

I'm not sure what could be happening 😞 . I'll see if I can push a version of the tracer that logs more data so you can use it to run your tests - that might help

[juan-fernandez]

I think I have an idea of what could be happening:

if an afterEach that is defined before datadog's plugin own afterEach handler fails, our necessary code does not execute and we don't calculate the test status correctly.

I think this can be fixed by making sure that datadog's support file is always defined first:

// cypress/support/support.ts
import "dd-trace/ci/cypress/support" // always on top!
import "cypress-fail-fast";
import "./commands"

This is not ideal and we need to find a better way to handle this but in the meantime it should fix your issue.

Could you test it @MaximilianoAdaro ?