testarmada / magellan Goto Github PK

Ran into this while getting magellan set up.

--> Worker 1, mock port: 12000, running test: test/testfile1_spec.js @phantomjs
Fatal internal error while running a test: SyntaxError: Unexpected token r
(1 / 42) <-- Worker 1 FAIL (will retry) test/testfile1_spec.js @phantomjs
--> Worker 2, mock port: 12003, running test: test/testfile2_spec.js @phantomjs

Looks like a fix would be to also output err.stack at https://github.com/TestArmada/magellan/blob/master/src/test_runner.js#L170

Magellan is great but, if you run in parallel and want to start up a server, things get more difficult. I'd love the ability to create a node file which would execute magellan using an API with a completion callback function. Something like this

var magellan = require('testarmada-magellan');

magellan.start(magellanOptions, function (err) {
  // the magellan process should still exit just as it would have previously (with same error code)
  // but I could use this callback to tear down any services created for testing purposes
});

Investigate if this actually works and what degree of sharing is stable, if any.

When running magellan's own tests using magellan itself, the environment claimed to be running those tests is @phantomjs, but in reality it's just vanilla node. This is incorrect.

Admiral dashboard support may or may not be working properly in non-nightwatch frameworks, so investigate whether this is an issue or not.

npm install on magellan fails due to this error:

npm ERR! [email protected] install: `node-gyp configure build`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script 'node-gyp configure build'.
npm ERR! This is most likely a problem with the node-proxy package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp configure build
npm ERR! You can get their info via:
npm ERR!     npm owner ls node-proxy
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/dcadwal/dev/test7/boilerplate-nightwatch/npm-debug.log

As is often the case, the real error is masked, until you run npm install node-proxy directly:

Agreeing to the Xcode/iOS license requires admin privileges, please re-run as root via sudo.

Error retrieving a new session from the selenium server
Error: socket hang up
    at createHangUpError (http.js:1477:15)
    at Socket.socketOnEnd [as onend] (http.js:1573:23)
    at Socket.g (events.js:180:16)
    at Socket.emit (events.js:117:20)
    at _stream_readable.js:944:16
    at process._tickCallback (node.js:442:13)


Connection refused! Is selenium server started?

/cc @1un4r

If no bail_time is set, then it's possible for magellan to appear to stall forever.

• magellan should probably have a default bail_time set.
• magellan should indicate periodically indicate that it's alive so that a CI process like Jenkins can know whether to kill it as a zombie process or not.

See this boilerplate-nightwatch build which had failures but still exited 0, and reported a green build to TravisCI.

I reproduced the issue with the base magellan test suite itself by adding assert(false); at the beginning of a test. I got this result:

============= Suite Complete =============

     Status: FAILED
    Runtime: 31.1s
Total tests: 15
 Successful: 14 / 15
     Failed: 1 / 15
Error initializing listeners (i.e. reporters, setup/teardown tasks): 

 ...dev/TestArmada/magellan   master ✘ ✹ 
$ echo $? 
0

Note the exit 0 status code, even though there was a failure. This could mean that legitimate failures are getting masked.

/cc @Maciek416 @archlichking

Nightwatch supports providing a JS file instead of a JSON file for configuration. I find this sort of convenient because it lets you specify remove config boilerplate with Object.assign and such.

This is done by using require in loading the config rather than fs.readFileSync JavaScript https://github.com/nightwatchjs/nightwatch/blob/df3794a10927286f20493e3999ce5de297f158b7/lib/runner/cli/clirunner.js#L62

If for example, magellan is set up with rowdy to run tests with SauceLabs. Every test on SauceLabs will have the same name which can make it hard to tell what exactly is failing. It would be great if there was a way to pass the test name to SauceLabs.

localtunnel exposes your localhost to the world for easy testing and sharing! No need to mess with DNS or deploy just to have others test out your changes.

we could potentially look into using this instead of sauce connect, as it would allow a common solution across multiple BaaS providers

Instead of (56/56) show (retry 2/3)

/cc @geekdave

Tunnel support on --sauce mode is a bit underserved at the moment:

• tunnel support only works correctly on nightwatch
• tunnel mode is default for sauce
• tunnel ratio is always 1:1 with workers

Defaulting to --no_tunnels is probably the most urgent issue here.

Magellan passes --debug as an argument to Nightwatch when in debug mode. However Nightwatch's terminology for to output Selenium logging (probably the closest concept it has to "debug") is --verbose. http://nightwatchjs.org/guide#command-line-options

We have an internal selenium grid, and don't use SauceLabs. I'm having trouble getting magellan to run tests against our own grid. It is possible with nightwatch -- is it possible with magellan-nightwatch?

The problem appears to be related to this bit. Despite having configured nightwatch to send session requests to http://my-hub:4444, magellan is sending them to some port other than 4444.

Im basing this project off of the boilerplate-nightwatch.

My nightwatch.json https://gist.github.com/chrismikehogan/4defd025c5f8ac7b8dfb
My magellan.json https://gist.github.com/chrismikehogan/c2c6fc48af7cc48099bb
Example output:

[Github] Test Suite
===================

Running:  Load TestArmada GitHub page
INFO Request: POST http://my-hub:12001/wd/hub/session 
 - data:  {"desiredCapabilities":{"browserName":"chrome","javascriptEnabled":true,"acceptSslCerts":true,"platform":"ANY","name":"Github"}} 
 - headers:  {"Content-Type":"application/json; charset=utf-8","Content-Length":128}


Error retrieving a new session from the selenium server
Error: connect ECONNREFUSED
    at exports._errnoException (util.js:746:11)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1000:19)


Connection refused! Is selenium server started?

Some environments are costly to the respective project to spin up. This could be lessened by allowing multiple tests to reuse a single Sauce VM environment.

Potential strawman:

• Require --serial for sanity is fine
• Allow a --reuse-client option to keep the existing Sauce / whatever environment
• Restart all the tests if any fail to make it simple.

This way, project owners can decide for themselves what is acceptable / too expensive to wait for in terms of Sauce VM startup time without having to completely give up slow starting Sauce VM environments.

Currently, nightwatch support requires users to painfully have to update their nightwatch json to seek out a selenium server path (and other deep-module paths):

  "selenium": {
    "start_process": true,
    "server_path": "./node_modules/testarmada-magellan-nightwatch/node_modules/selenium-server/lib/runner/selenium-server-standalone-2.46.0.jar",
    "log_path": "tests/reports",
    "host": "127.0.0.1",
    "port": 4444,
    "cli_args": {
      "webdriver.chrome.driver": "./node_modules/testarmada-magellan-nightwatch/node_modules/chromedriver/lib/chromedriver/chromedriver",
      "webdriver.ie.driver": ""
    }
  },

and

    "phantomjs" : {
      "desiredCapabilities" : {
        "browserName" : "phantomjs",
        "javascriptEnabled" : true,
        "acceptSslCerts" : true,
        "phantomjs.binary.path" : "./node_modules/testarmada-magellan-nightwatch/node_modules/phantomjs/bin/phantomjs"
      }
    },

Magellan should perhaps just handle this by finding these modules and wiring up these paths for users.

I have magellan up and running against a non-trivial suite of nightwatch tests (written in ES6, using Nightwatch page objects, multiple environments). This is pretty exciting! I PRed some other issues I ran into that didn't seem to require any major design considerations but this one seems a bit bigger.

Acorn is used to parse test files and extract tags from them as static properties on the file. This hardcodes Magellan to only support tests written in ES5 syntax.

I noticed that changing https://github.com/TestArmada/magellan/blob/master/src/test_filter.js#L42 from

    var root = acorn.parse(fs.readFileSync(path.resolve(f)));

to

    var root = acorn.parse(fs.readFileSync(path.resolve(f)), {
      ecmaVersion: 6
    });

fixes the issue but at the cost of potentially more confusion when a syntax error happens. I think this might be the best solution as asking the consumer to opt-in to ES6 as part of their arguments to Magellan seems a bit unnecessary.

Any preference on how this issue might be fixed?

Some users need the ability to add desiredCapabilities to all environments (eg: for dismissing modals, etc).

/cc @1un4r

Admiral integration has some issues at the moment:

• Errors are hard to decipher.
• Functionality is largely untested.

We should address both of these issues and also test the admiral integration's error handling in both misconfiguration and connection problem scenarios.

/cc @geekdave

When I am waiting for a test that I know will fail…….and I CTRL-C cancel. During the next test run, the browser doesn’t launch….I need to kill the terminal window, restart another terminal and trigger the test.
This has happened many times. Is there away I can avoid launching a new terminal window.

Hello,

Are there any tutorials or informations how to set up slack reporter integration? I see no informations on this and there're no examples in config. Would be great to know how to do this.

Best regards,
drptbl.

Ubuntu 14.04 Desktop
java version "1.7.0_79"
node version: v0.10.25
npm version: 2.13.3

./node_modules/.bin/magellan --serial --browser=chrome --bail_fast

Error retrieving a new session from the selenium server
{ status: 13,
sessionId: null,
value:
{ supportUrl: null,
additionalInformation: '\nDriver info: org.openqa.selenium.chrome.ChromeDriver',
cause: null,
systemInformation: 'System info: host: 'asunar-VirtualBox', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'i386', os.version: '3.16.0-30-generic', java.version: '1.7.0_79'',
class: 'org.openqa.selenium.WebDriverException',
suppressed: [ [length]: 0 ],
hCode: 16165686,
message: 'unknown error: cannot find Chrome binary\n (Driver info: chromedriver=2.12.301324 (de8ab311bc9374d0ade71f7c167bad61848c7c48),platform=Linux 3.16.0-30-generic x86) (WARNING: The server did not provide any stacktrace information)\nCommand duration or timeout: 924 milliseconds\nBuild info: version: '2.43.1', revision: '5163bce', time: '2014-09-10 16:27:33'\nSystem info: host: 'asunar-VirtualBox', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'i386', os.version: '3.16.0-30-generic', java.version: '1.7.0_79'\nDriver info: org.openqa.selenium.chrome.ChromeDriver',
buildInformation:
{ releaseLabel: '2.43.1',
buildTime: '2014-09-10 16:27:33',
class: 'org.openqa.selenium.internal.BuildInfo',
buildRevision: '5163bce',
hCode: 33066190 },
localizedMessage: 'unknown error: cannot find Chrome binary\n (Driver info: chromedriver=2.12.301324 (de8ab311bc9374d0ade71f7c167bad61848c7c48),platform=Linux 3.16.0-30-generic x86) (WARNING: The server did not provide any stacktrace information)\nCommand duration or timeout: 924 milliseconds\nBuild info: version: '2.43.1', revision: '5163bce', time: '2014-09-10 16:27:33'\nSystem info: host: 'asunar-VirtualBox', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'i386', os.version: '3.16.0-30-generic', java.version: '1.7.0_79'\nDriver info: org.openqa.selenium.chrome.ChromeDriver' },
state: 'unhandled error',
class: 'org.openqa.selenium.remote.Response',
hCode: 15619088 }

Other than the comment in the webdriver.io boilerplate repo, I can't find a great deal of documentation around getting a multi-file test suite up and running. Would you consider adding more documentation around it?

Specifically I am interested in a multi-file webdriver.io boilerplate.

Thanks!

If you create a folder structure under <project>/tests/<some directory>/<testfile.js> , the test cases are not recognized unless you change the conf/nightwatch.json with that specific directory. This will not be feasible at large scale when teams will like to create folders for their tracks. We need to ensure that the folders are ignored and just .js files are recognized as valid test cases at any level under tests/

This may be only possible for frameworks that send rich data back to magellan, since we need to definitively know that the test has started running in selenium (session id acquisition may be a good way to signal this).

This would allow claims of VMs to more tightly track remote actual usage.

See:

https://github.com/TestArmada/boilerplate-mocha/issues/1

It would be great if Magellan could scan directories for mocha tests the same way mocha can with the recursive option.

Some options:

• scan mocha.opts to see if the user has requested this
• always do it with all supplied folders (possibly dangerous)?

Right now most of the code in the magellan binary in bin is not testable, so we can't write tests for our initialization process, only the test runner. The magellan init process in that binary needs to be separated out into a module that can be tested in our mocha unit tests.

... and perhaps think about adding a code of conduct also :)

$> "./node_modules/.bin/magellan" --browser=firefox --bail_fast
Sauce configuration OK
Magellan-nightwatch test iterator found nightwatch configuration at: ./conf/nightwatch.json

Running 3 tests with 3 workers with firefox

--> Worker 3, mock port: 12006, running test: tests/walmart-shirt.js @Firefox
--> Worker 2, mock port: 12003, running test: tests/walmart-book.js @Firefox
--> Worker 1, mock port: 12000, running test: tests/github.js @Firefox

Opens firefox browser, fails with the following error message:
'Unable to connect to host 127.0.0.1 on port 7055 after 45000 ms........'

Updated configuration to use selenium driver 2.46.0 like this:
"server_path": "./node_modules/testarmada-magellan-nightwatch/node_modules/selenium-server/lib/runner/selenium-server-standalone-2.46.0.jar"

$> "./node_modules/.bin/magellan" --browser=firefox --bail_fast
Sauce configuration OK
Magellan-nightwatch test iterator found nightwatch configuration at: ./conf/nightwatch.json

Running 3 tests with 3 workers with firefox

--> Worker 1, mock port: 12000, running test: tests/github.js @Firefox
--> Worker 2, mock port: 12003, running test: tests/walmart-book.js @Firefox
--> Worker 3, mock port: 12006, running test: tests/walmart-shirt.js @Firefox
(1 / 3) <-- Worker 1 PASS tests/github.js @Firefox
(2 / 3) <-- Worker 2 PASS tests/walmart-book.js @Firefox
(3 / 3) <-- Worker 3 FAIL tests/walmart-shirt.js @Firefox
.......
ERROR: Unable to locate element: "[data-magellan-temp-automation-id='magellan_click_6c0a5e2']" using: css selector
Unable to locate element: "[data-magellan-temp-automation-id='magellan_click_6c0a5e2']" using: css selector

FAILED: 1 errors and 1 passed (5.878s)

Running: Check product details
✖ Selector '.product-short-description' was not visible after 60117 milliseconds. - expected "visible" but got: not visible

λ "./node_modules/.bin/magellan" --browser=chrome --bail_fast
Sauce configuration OK
Magellan-nightwatch test iterator found nightwatch configuration at: ./conf/nightwatch.json

Running 3 tests with 3 workers with chrome

--> Worker 3, mock port: 12006, running test: tests/walmart-shirt.js @chrome
--> Worker 2, mock port: 12003, running test: tests/walmart-book.js @chrome
--> Worker 1, mock port: 12000, running test: tests/github.js @chrome
(1 / 3) <-- Worker 2 PASS tests/walmart-book.js @chrome
(2 / 3) <-- Worker 1 PASS tests/github.js @chrome
(3 / 3) <-- Worker 3 PASS tests/walmart-shirt.js @chrome

============= Suite Complete =============

 Status: PASSED
Runtime: 27.8s

Total tests: 3
Successful: 3 / 3

If google or wikipedia change their page or go down, our tests shouldn't fail.

Let's perhaps borrow the test playground from webdriver.io and spin it up as a localhost server during tests, and assert against that.

Magellan should be able to continue to function if a brief locks outage happens:

• should have a default maximum amount of time spent waiting for locks to recover
• should recover gracefully from 404 / 500 / connection errors, and have reasonably short default timeouts for each of these
• should explain that a build is failing from a locks outage if one happens

So for certain hidden customized device config we can only leave useful key-value pair in config file.

Right now profiles have to be in magellan.json, but it would be useful for an organization to be able to centralize profiles in one place, to avoid having to update dozens of files when transitioning to new tiers of browsers (or devices).

centos 6.4
x64
java 1.7

{ [UnknownError: unknown error: cannot find Chrome binary
(Driver info: chromedriver=2.18.343837 (52eb4041461e46a6b73308ebb19e85787ced4281),platform=Linux 3.13.0-49-generic x86_64)]
code: 13,
state: 'unknown error',
message: 'unknown error: cannot find Chrome binary\n (Driver info: chromedriver=2.18.343837 (52eb4041461e46a6b73308ebb19e85787ced4281),platform=Linux 3.13.0-49-generic x86_64)',
name: 'UnknownError',
stack: 'UnknownError: unknown error: cannot find Chrome binary\n (Driver info: chromedriver=2.18.343837 (52eb4041461e46a6b73308ebb19e85787ced4281),platform=Linux 3.13.0-49-generic x86_64)\n at new bot.Error (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/atoms/error.js:108:18)\n at Object.bot.response.checkResponse (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/atoms/response.js:109:9)\n at /root/node_modules/browsertime/node_modules/selenium-webdriver/lib/webdriver/webdriver.js:160:24\n at promise.ControlFlow.runInFrame_ (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/goog/../webdriver/promise.js:1857:20)\n at goog.defineClass.notify (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/goog/../webdriver/promise.js:2448:25)\n at promise.Promise.notify_ (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/goog/../webdriver/promise.js:564:12)\n at Array.forEach (native)\n at promise.Promise.notifyAll_ (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/goog/../webdriver/promise.js:553:15)\n at goog.async.run.processWorkQueue (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/goog/async/run.js:125:21)\n at runMicrotasksCallback (node.js:337:7)\nFrom: Task: WebDriver.createSession()\n at Function.webdriver.WebDriver.acquireSession_ (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/webdriver/webdriver.js:157:22)\n at Function.webdriver.WebDriver.createSession (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/webdriver/webdriver.js:131:30)\n at new Driver (/root/node_modules/browsertime/node_modules/selenium-webdriver/chrome.js:783:36)\n at Builder.build (/root/node_modules/browsertime/node_modules/selenium-webdriver/builder.js:454:14)\n at Object.module.exports.getDriver (/root/node_modules/browsertime/lib/browsers/chrome.js:41:49)\n at /root/node_modules/browsertime/lib/browsertime.js:107:64\n at /root/node_modules/browsertime/node_modules/async/lib/async.js:744:13\n at /root/node_modules/browsertime/node_modules/async/lib/async.js:46:16\n at /root/node_modules/browsertime/node_modules/async/lib/async.js:260:32\n at /root/node_modules/browsertime/node_modules/async/lib/async.js:37:16' }
unknown error: cannot find Chrome binary
(Driver info: chromedriver=2.18.343837 (52eb4041461e46a6b73308ebb19e85787ced4281),platform=Linux 3.13.0-49-generic x86_64)

There are opportunities to split the testing infrastructure between singletons and per-worker components, along with some potential to decide how to distribute the load amongst workers. Some examples:

• boot up only one sauce connect tunnel for the duration of the run
• similarly, only one local selenium server per test runner (I believe selenium servers have "sessions" so you need only create one and then each worker will use a new session)
• dedicate a worker per browser, then run the entire functional test suite in one worker; OR
• one browser + one file per worker (versus one browser + one "it" case)
• the readme suggests manually using npm scripts to perform setup/teardown operations for the entire test run, this is something that could be baked into magellan so that you need only run "magellan" in any scenario

What are your thoughts - does that sound okay?

To be able to run firefox tests updated configuration to use selenium driver 2.46.0 like this:
"server_path": "./node_modules/testarmada-magellan-nightwatch/node_modules/selenium-server/lib/runner/selenium-server-standalone-2.46.0.jar"

Chrome only => all tests pass
Firefox only => walmart-shirt.js fails, the rest pass.

Chrome,Firefox parallel fails with the following error:

λ "./node_modules/.bin/magellan" --browser=chrome,firefox --bail_fast
...
(1 / 3) <-- Worker 1 FAIL tests/github.js @chrome,firefox
(2 / 3) <-- Worker 3 FAIL tests/walmart-shirt.js @chrome,firefox
(3 / 3) <-- Worker 2 FAIL tests/walmart-book.js @chrome,firefox
...
nightwatch-magellan has found nightwatch configuration at C:\Users\ASunar\boilerplate\temp\build-magellan-13e9e9364_24ccb1041__temp_assets\nightwatch.json
chrome TypeError: undefined is not a function
at emitStartedTest (C:\Users\ASunar\boilerplate\node_modules\testarmada-magellan-nightwatch\lib\base-test-class.js:70:13)
at Object.MagellanBaseTest.beforeEach (C:\Users\ASunar\boilerplate\node_modules\testarmada-magellan-nightwatch\lib\base-test-class.js:144:7)
at TestSuite. (C:\Users\ASunar\boilerplate\node_modules\nightwatch\lib\runner\testsuite.js:90:18)
at TestSuite.makePromise (C:\Users\ASunar\boilerplate\node_modules\nightwatch\lib\runner\testsuite.js:384:8)
at Object.beforeEach (C:\Users\ASunar\boilerplate\node_modules\nightwatch\lib\runner\testsuite.js:88:21)
at TestSuite.beforeEach (C:\Users\ASunar\boilerplate\node_modules\nightwatch\lib\runner\testsuite.js:323:21)
at TestCase.run (C:\Users\ASunar\boilerplate\node_modules\nightwatch\lib\runner\testcase.js:47:6)
at TestSuite.runTestCase (C:\Users\ASunar\boilerplate\node_modules\nightwatch\lib\runner\testsuite.js:284:20)
at TestSuite.runNextTestCase (C:\Users\ASunar\boilerplate\node_modules\nightwatch\lib\runner\testsuite.js:272:21)
at C:\Users\ASunar\boilerplate\node_modules\nightwatch\lib\runner\testsuite.js:220:19

http://www.screencast.com/t/H9Z0SguP

The tunnel construction process can time out and stall. We should have timeouts and continue with what tunnels we have (existing reduced worker mode) if tunnels take too long to construct.

Reproduction

• Download https://github.com/walmartlabs/little-loader and npm install and npm run install-dev
• Try to filter with magellan --tests=./test/func/spec/advanced.spec.js (or any variation to get to just that test)
• Expected Behavior: Runs only that test.
• Actual Behavior: Errors out with:

$ magellan --test=test/func/spec/advanced.spec.js 
Loaded magellan configuration from:  /Users/rye/scm/wm/little-loader/magellan.json
Sauce configuration OK
Using test filter:  test/func/spec/advanced.spec.js
Error: no tests found

For testing IE locally on a mac, would be great to be able to use:

• virtualbox to host a windows instance
• ievms to obtain windows images from microsoft
• iectrl to auto-manage the images (handle re-installing after expiration, etc)

If we have workers available, we should retry in parallel using those spare workers.

If a job is linked up with admiral, we show the link to the admiral URL after a suite is complete. But for jobs not linked with admiral, or if a user wants to jump directly to a sauce result from the console, we should change this:

(1 / 1) <-- Worker 1 PASS  loads basic script @IE_8_Windows_2008_Desktop  

to this:

(1 / 1) <-- Worker 1 PASS  loads basic script @IE_8_Windows_2008_Desktop  https://saucelabs.com/tests/1234567890

Support vanilla nightwatch in addition to the enhanced nightwatch integration we already have.