protractor-flake seems to rerun ALL tests when only SOME fail about protractor-flake

[tedsiebear]

hm still an issue for me, I am logging stackstrace: displayStacktrace: 'all' or 'specs' and still it is rerunning the whole test suite rather than individual tests that failed, using jasmine2

    for (; i < l; i++) {
        if (args[i] === "-t") {
            src.push("spec/e2e/" + args[++i].replace(/\.spec.js$/, "") + ".spec.js");
        }
        if (args[i] === "-f") {
            src.push("spec/e2e/" + args[++i].replace(/\/$/, "") + "/**/*.spec.js");
        }
    }
    if (src.length > 0) {
        protractorArgs.push("--specs", src.join(","));
    }

[NickTomlin]

I can definitely confirm this happens when --suite is passed to protractor. I'll start working on this but may not be able to get a fix out for a few days due to scheduling. Feel free to PR something in the meantime.

[vdimikj]

Resolved with this PR

• #70
  Thanks

[NickTomlin]

👍 thanks for reporting back in

[NickTomlin]

That should definitely not be happening, thanks for bringing this to my attention! If you have the time could you try running protractor flake with the PROTRACTOR_FLAKE_LOG_LEVEL env variable set to debug and post the output?

I'll do my own investigation as well.

[NickTomlin]

A super hacky fix would be to just remove the --suite from the arguments passed to protractor when re-running the specs. I'll push a branch with this fix if you wouldn't mind testing it.

[NickTomlin]

Here's the branch. I'm going to see if I can get some integration coverage for it. Would love any feedback you have.

[bcherny]

pulling your branch, the behavior is still the same - successful tests are re-run every time.

npm i NickTomlin/protractor-flake#suite
cd node_modules/protractor-flake
npm i && npm run build
cd ../..
export PROTRACTOR_FLAKE_LOG_LEVEL=debug
gulp e2e

also, setting the log level seems to have no effect..

[NickTomlin]

@bcherny interesting, I haven't been able to replicate this 😦 Does your project happen to be open sourced? I'd love to pull it down and take a look.

I've set up a gitter room so we can talk about things in a more real-time manner if that suits you better.

[bcherny]

i won't be able to take another look till possibly later today. will keep you posted!

[NickTomlin]

@bcherny were you able to take another look? I still haven't been able to reproduce, but I want to make sure this isn't happening :)

[NickTomlin]

Looks like others are reproducing this.

[NickTomlin]

It appears like this is an issue with Jasmine2 output. @brandonbeyers and I are investigating further.

[bcherny]

thanks for the updates! sorry about not helping more, it's been a hectic few weeks.

[NickTomlin]

It seems like this was due to jasmine2 output being slightly different than jasmine1 output.
Jasmine one has failed out put like [object Object].<anonymous> (/path/to/spec.js) whereas jasmine2 has output like at Object.<anonymous> (/path/to/spec.js).

I've fixed this in this branch. @bcherny if you wouldn't mind checking that out and seeing if it resolves your issue i'd much appreciate it. Thanks for your patience!

[NickTomlin]

So this is caused by Jasmine2 output (which will be fixed by the changes in the aforementioned branch) or due to a custom reporter that changes the failed spec output from what protractor-flake expects. If you are using a custom reporter you will need to make sure that the original jasmine failures get printed to STDOUT by your tests. @brandonbeyers has a solution for this and is going to PR an addition to the README.

I'm going to merge that branch in and do a new release.

[brandenbyers]

Yes, as @NickTomlin says, if working with a custom reporter such as jasmine-spec-reporter, make sure to display stacktraces:

onPrepare: function() {
    var SpecReporter = require('jasmine-spec-reporter');
    jasmine.getEnv().addReporter(new SpecReporter({
        displayStacktrace: 'specs'
    }));
}

[okram999]

when some test fails due to timeout, there is no stacktrace. Its 'undefined'. And hence it ends up re-running all the test. Is this by design?

[FinKingma]

@okram999 The default parser only supplies the test specs that can be retrieved with an error stack trace. With timeouts, the test specs are not shown in the error stack trace. So by default there is no way to retrieve the specfile. This results in rerunning all your specs.

If you are using sharding or other types of multitests, you can now also use the multi parser, which assumes that the specfile is shown in the log in the following manner:
Specs: /tests/a-good-test.test.js
This way it can retrieve all the test specs that did not pass (so timeouts will also be rerun).

If this is the case, you can use the multi parser by specifying the following parameter:
parser: 'multi'

When you are still unsure of which parser to use, could you share your logging output with the timeout?

[quantizor]

This is still a bug. Using multi and everything gets re-run.

[wswebcreation]

Can you give some more info about what you experience?

• Your config
• Your command you use to run protractor-flake
• The log that needs to be parsed

Tnx in advance

[trevcor]

I'm seeing a similar issue using the multi parser where any spec that has a pending test will be re-run every time.

[NickTomlin]

@trevcor can you provide the following details:

• Your config
• Your command you use to run protractor-flake
• The log that needs to be parsed

[naspeh]

Hey, thanks for hard work!

It works great for other specs, but for this I got all specs running when one spec fails.

this log has spec name

Failures:
1) analytics manage activity reports
  Message:
    Expected 2 to equal 1.
  Stack:
    Error: Failed expectation
        at Object.it (/opt/superdesk/client-core/spec/analytics_spec.js:68:47)
        at /opt/superdesk/client-core/node_modules/jasminewd2/index.js:94:23
        at new ManagedPromise (/opt/superdesk/client-core/node_modules/selenium-webdriver/lib/promise.js:1082:7)
        at controlFlowExecute (/opt/superdesk/client-core/node_modules/jasminewd2/index.js:80:18)
        at TaskQueue.execute_ (/opt/superdesk/client-core/node_modules/selenium-webdriver/lib/promise.js:2913:14)
        at TaskQueue.executeNext_ (/opt/superdesk/client-core/node_modules/selenium-webdriver/lib/promise.js:2896:21)
        at asyncRun (/opt/superdesk/client-core/node_modules/selenium-webdriver/lib/promise.js:2820:25)
        at /opt/superdesk/client-core/node_modules/selenium-webdriver/lib/promise.js:639:7
        at process._tickCallback (internal/process/next_tick.js:103:7)
Pending:

[...]

Using standard to parse output
Re-running tests: test attempt 2

Tests failed but no specs were found. All specs will be run again.

[...]

spec here https://github.com/superdesk/superdesk-client-core/blob/master/spec/analytics_spec.js

---

Updated: one more case

Failures:
1) search can action on items
  Message:
�[31m    Expected false to be true.�[0m
  Stack:
    Error: Failed expectation
        at Object.it (/opt/superdesk/client-core/spec/search_spec.js:194:41)
        at /opt/superdesk/client-core/node_modules/jasminewd2/index.js:94:23
        at new ManagedPromise (/opt/superdesk/client-core/node_modules/selenium-webdriver/lib/promise.js:1082:7)
        at controlFlowExecute (/opt/superdesk/client-core/node_modules/jasminewd2/index.js:80:18)
        at TaskQueue.execute_ (/opt/superdesk/client-core/node_modules/selenium-webdriver/lib/promise.js:2913:14)
        at TaskQueue.executeNext_ (/opt/superdesk/client-core/node_modules/selenium-webdriver/lib/promise.js:2896:21)
        at asyncRun (/opt/superdesk/client-core/node_modules/selenium-webdriver/lib/promise.js:2820:25)
        at /opt/superdesk/client-core/node_modules/selenium-webdriver/lib/promise.js:639:7
Pending:

[...]

Using standard to parse output
Re-running tests: test attempt 2

Tests failed but no specs were found. All specs will be run again.

[...]

spec: https://github.com/superdesk/superdesk-client-core/blob/master/spec/search_spec.js

[wswebcreation]

@NickTomlin,

Looking at the above log it looks like the regex in the standard.js parser doesn't give any match. Only (?:\[object Object\]|Object) gives a match.
If you look at the testdata we use to test the jasmine2-output the regex gives a match on only at Object.<anonymous>.
Do you happen to know where the at Object.it is coming from? If the .it is also an option in jasmine2, we may need to change the regex to

at (?:\[object Object\]|Object)\.(?:<anonymous>|it) \((([A-Za-z]:\\)?.*?):.*\)

Grtz

Wim

[NickTomlin]

@wswebcreation great call; I think you are right. I wasn't actually aware that Object.it was a jasmine built in. I think your solution sound fair; want to PR it?

[wswebcreation]

@NickTomlin,

Will make a PR for it and add a new testcase for it. Hope to do it this weekend. I got 1 question, you say I wasn't actually aware that Object.it was a jasmine built in , do you mean with this that this is new and we need to use it?

Grtz

Wim

[wswebcreation]

@naspeh ,

Can you give a complete anonymized log so I can create a correct test for it?
Tnx in advance!

[naspeh]

@wswebcreation, sure, here is: https://gist.github.com/naspeh/38a89bb2bdc48a98248e422c725f288d

Thanks for attention and quick feedback!

[NickTomlin]

@wswebcreation I was confused because I hadn't seen Object.it in Jasmine before. After doing some quick looking I think it's coming from jasminewd2 so we should definitely expand the regex.

[wswebcreation]

@naspeh

Tnx. Is this the complete log? I only see 1 failure on 1 spec file (on both examples you gave me). Is that correct?
The link to the superdesk project you provided, is that also the testsetup you are using? "protractor": "^4.0.13"?

[naspeh]

Tnx. Is this the complete log? I only see 1 failure on 1 spec file (on both examples you gave me). Is that correct?

yes this is complete log for protractor step, this is false failing which I want to catch up with protractor-flake :)

[wswebcreation]

Added a PR for it #49

[naspeh]

Super. Thanks guys, it works, I've just checked new logs with [email protected]

[wswebcreation]

Awesome 👍

[anuvratashu]

Hi Guys,
Its great work... But its still running all test even there are only 2 test which failed.
node_modules/.bin/protractor-flake --parser multi --node-bin node --max-attempts=2 --color=magenta -- JSOut/config.js",

var SpecReporter = require('jasmine-spec-reporter').SpecReporter;
jasmine.getEnv().addReporter(new SpecReporter({
displayStacktrace: 'specs'
}));

Can you please let me know what else i am missing?

[wswebcreation]

Hi @anuvratashu ,

This is a closed issue, please open a new issue if you want, also provide your config, and your log so we can do a detailed check on your problem.

[vdimikj]

Hi Guys,

I m facing an issue like this one.
The regex is not catching this one.

// when test appear
      - Expected '' to be 'Successfully updated page'.
          at UserContext.<anonymous> (/var/www/admin_panel/src/app/e2e/keeping-the-inserted-data.e2e-spec.ts:43:47)
          at /var/www/admin_panel/node_modules/protractor/node_modules/jasminewd2/index.js:112:25
          at new ManagedPromise (/var/www/admin_panel/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:1067:7)
          at ControlFlow.promise (/var/www/admin_panel/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:2396:12)
          at schedulerExecute (/var/www/admin_panel/node_modules/protractor/node_modules/jasminewd2/index.js:95:18)
          at TaskQueue.execute_ (/var/www/admin_panel/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:2970:14)
          at TaskQueue.executeNext_ (/var/www/admin_panel/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:2953:27)
          at asyncRun (/var/www/admin_panel/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:2860:25)
          at /var/www/admin_panel/node_modules/protractor/node_modules/selenium-webdriver/lib/promise.js:676:7

This is solvable by updating the old regex
at (?:\[object Object\]|Object)\.(?:<anonymous>|it) \((([A-Za-z]:\\)?.*?):.*\)
to
at (?:\[object Object\]|Object|UserContext)\.(?:<anonymous>|it) \((([A-Za-z]:\\)?.*?):.*\)

Though i m using SpecReporter, Jasmine, and starting flake with standard parser

Thanks,
Vlad

[NickTomlin]

@VladimirDimikjCuponation interesting, i'm not sure if this is a common thing? You could solve this by writing a custom parser for yourself, or, if this is part of jasmine, we could just incorporate it into the standard parser.

[wswebcreation]

@NickTomlin

I think this has to do with the SpecReporter, see also here. I read a lot about the spec reporter, should we think about supporting that?

[NickTomlin]

Interesting, the UserContext error seems different than the spec reporter (e.g. the spec reporter output doesn't have any stack trace information) but I agree that it is an issue.

Since Jasmine supports multiple reports (IIRC) I wonder if we could do something like add a custom flake reporter that would write the output to a file or communicate with flake in some other way. Sounds slightly complicated though, do you have any thoughts?

[wswebcreation]

Having a few weeks off so plenty of time to think about it 😊

[vdimikj]

@NickTomlin @wswebcreation any updates on this one?

[wswebcreation]

@vdimikj

Currently not, I'm thinking about a different solution. But because people SpecReporter and want a clean report no stacktrace and and other logging is shown. That extra logging is needed by protractor-flake to read the logging and find the failing spec files.
I would suggest to change some settings in you SpecReporter initialisation, see here to get the right logging that protractor-flake can pick up.