testarmada / magellan Goto Github PK
View Code? Open in Web Editor NEWLarge Scale Automated Testing for Mocha, Nightwatch, Appium, Nodejs, etc
License: MIT License
Large Scale Automated Testing for Mocha, Nightwatch, Appium, Nodejs, etc
License: MIT License
Ran into this while getting magellan set up.
--> Worker 1, mock port: 12000, running test: test/testfile1_spec.js @phantomjs
Fatal internal error while running a test: SyntaxError: Unexpected token r
(1 / 42) <-- Worker 1 FAIL (will retry) test/testfile1_spec.js @phantomjs
--> Worker 2, mock port: 12003, running test: test/testfile2_spec.js @phantomjs
Looks like a fix would be to also output err.stack
at https://github.com/TestArmada/magellan/blob/master/src/test_runner.js#L170
Magellan is great but, if you run in parallel and want to start up a server, things get more difficult. I'd love the ability to create a node file which would execute magellan using an API with a completion callback function. Something like this
var magellan = require('testarmada-magellan');
magellan.start(magellanOptions, function (err) {
// the magellan process should still exit just as it would have previously (with same error code)
// but I could use this callback to tear down any services created for testing purposes
});
Investigate if this actually works and what degree of sharing is stable, if any.
When running magellan's own tests using magellan itself, the environment claimed to be running those tests is @phantomjs
, but in reality it's just vanilla node. This is incorrect.
Admiral dashboard support may or may not be working properly in non-nightwatch frameworks, so investigate whether this is an issue or not.
npm install
on magellan fails due to this error:
npm ERR! [email protected] install: `node-gyp configure build`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] install script 'node-gyp configure build'.
npm ERR! This is most likely a problem with the node-proxy package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! node-gyp configure build
npm ERR! You can get their info via:
npm ERR! npm owner ls node-proxy
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR! /Users/dcadwal/dev/test7/boilerplate-nightwatch/npm-debug.log
As is often the case, the real error is masked, until you run npm install node-proxy
directly:
Agreeing to the Xcode/iOS license requires admin privileges, please re-run as root via sudo.
Error retrieving a new session from the selenium server
Error: socket hang up
at createHangUpError (http.js:1477:15)
at Socket.socketOnEnd [as onend] (http.js:1573:23)
at Socket.g (events.js:180:16)
at Socket.emit (events.js:117:20)
at _stream_readable.js:944:16
at process._tickCallback (node.js:442:13)
Connection refused! Is selenium server started?
/cc @1un4r
If no bail_time
is set, then it's possible for magellan to appear to stall forever.
bail_time
set.See this boilerplate-nightwatch build which had failures but still exited 0, and reported a green build to TravisCI.
I reproduced the issue with the base magellan test suite itself by adding assert(false);
at the beginning of a test. I got this result:
============= Suite Complete =============
Status: FAILED
Runtime: 31.1s
Total tests: 15
Successful: 14 / 15
Failed: 1 / 15
Error initializing listeners (i.e. reporters, setup/teardown tasks):
...dev/TestArmada/magellan master ✘ ✹
$ echo $?
0
Note the exit 0 status code, even though there was a failure. This could mean that legitimate failures are getting masked.
Nightwatch supports providing a JS file instead of a JSON file for configuration. I find this sort of convenient because it lets you specify remove config boilerplate with Object.assign
and such.
This is done by using require
in loading the config rather than fs.readFileSync
JavaScript https://github.com/nightwatchjs/nightwatch/blob/df3794a10927286f20493e3999ce5de297f158b7/lib/runner/cli/clirunner.js#L62
If for example, magellan is set up with rowdy to run tests with SauceLabs. Every test on SauceLabs will have the same name which can make it hard to tell what exactly is failing. It would be great if there was a way to pass the test name to SauceLabs.
localtunnel exposes your localhost to the world for easy testing and sharing! No need to mess with DNS or deploy just to have others test out your changes.
we could potentially look into using this instead of sauce connect, as it would allow a common solution across multiple BaaS providers
Instead of (56/56)
show (retry 2/3)
/cc @geekdave
Tunnel support on --sauce
mode is a bit underserved at the moment:
Defaulting to --no_tunnels
is probably the most urgent issue here.
Magellan passes --debug
as an argument to Nightwatch when in debug mode. However Nightwatch's terminology for to output Selenium logging (probably the closest concept it has to "debug") is --verbose
. http://nightwatchjs.org/guide#command-line-options
We have an internal selenium grid, and don't use SauceLabs. I'm having trouble getting magellan to run tests against our own grid. It is possible with nightwatch -- is it possible with magellan-nightwatch?
The problem appears to be related to this bit. Despite having configured nightwatch to send session requests to http://my-hub:4444, magellan is sending them to some port other than 4444.
Im basing this project off of the boilerplate-nightwatch.
My nightwatch.json https://gist.github.com/chrismikehogan/4defd025c5f8ac7b8dfb
My magellan.json https://gist.github.com/chrismikehogan/c2c6fc48af7cc48099bb
Example output:
[Github] Test Suite
===================
Running: Load TestArmada GitHub page
INFO Request: POST http://my-hub:12001/wd/hub/session
- data: {"desiredCapabilities":{"browserName":"chrome","javascriptEnabled":true,"acceptSslCerts":true,"platform":"ANY","name":"Github"}}
- headers: {"Content-Type":"application/json; charset=utf-8","Content-Length":128}
Error retrieving a new session from the selenium server
Error: connect ECONNREFUSED
at exports._errnoException (util.js:746:11)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1000:19)
Connection refused! Is selenium server started?
Some environments are costly to the respective project to spin up. This could be lessened by allowing multiple tests to reuse a single Sauce VM environment.
Potential strawman:
--serial
for sanity is fine--reuse-client
option to keep the existing Sauce / whatever environmentThis way, project owners can decide for themselves what is acceptable / too expensive to wait for in terms of Sauce VM startup time without having to completely give up slow starting Sauce VM environments.
Currently, nightwatch support requires users to painfully have to update their nightwatch json to seek out a selenium server path (and other deep-module paths):
"selenium": {
"start_process": true,
"server_path": "./node_modules/testarmada-magellan-nightwatch/node_modules/selenium-server/lib/runner/selenium-server-standalone-2.46.0.jar",
"log_path": "tests/reports",
"host": "127.0.0.1",
"port": 4444,
"cli_args": {
"webdriver.chrome.driver": "./node_modules/testarmada-magellan-nightwatch/node_modules/chromedriver/lib/chromedriver/chromedriver",
"webdriver.ie.driver": ""
}
},
and
"phantomjs" : {
"desiredCapabilities" : {
"browserName" : "phantomjs",
"javascriptEnabled" : true,
"acceptSslCerts" : true,
"phantomjs.binary.path" : "./node_modules/testarmada-magellan-nightwatch/node_modules/phantomjs/bin/phantomjs"
}
},
Magellan should perhaps just handle this by finding these modules and wiring up these paths for users.
I have magellan up and running against a non-trivial suite of nightwatch tests (written in ES6, using Nightwatch page objects, multiple environments). This is pretty exciting! I PRed some other issues I ran into that didn't seem to require any major design considerations but this one seems a bit bigger.
Acorn is used to parse test files and extract tags from them as static properties on the file. This hardcodes Magellan to only support tests written in ES5 syntax.
I noticed that changing https://github.com/TestArmada/magellan/blob/master/src/test_filter.js#L42 from
var root = acorn.parse(fs.readFileSync(path.resolve(f)));
to
var root = acorn.parse(fs.readFileSync(path.resolve(f)), {
ecmaVersion: 6
});
fixes the issue but at the cost of potentially more confusion when a syntax error happens. I think this might be the best solution as asking the consumer to opt-in to ES6 as part of their arguments to Magellan seems a bit unnecessary.
Any preference on how this issue might be fixed?
Some users need the ability to add desiredCapabilities
to all environments (eg: for dismissing modals, etc).
/cc @1un4r
Admiral integration has some issues at the moment:
We should address both of these issues and also test the admiral integration's error handling in both misconfiguration and connection problem scenarios.
/cc @geekdave
When I am waiting for a test that I know will fail…….and I CTRL-C cancel. During the next test run, the browser doesn’t launch….I need to kill the terminal window, restart another terminal and trigger the test.
This has happened many times. Is there away I can avoid launching a new terminal window.
Hello,
Are there any tutorials or informations how to set up slack reporter integration? I see no informations on this and there're no examples in config. Would be great to know how to do this.
Best regards,
drptbl.
Ubuntu 14.04 Desktop
java version "1.7.0_79"
node version: v0.10.25
npm version: 2.13.3
./node_modules/.bin/magellan --serial --browser=chrome --bail_fast
Error retrieving a new session from the selenium server
{ status: 13,
sessionId: null,
value:
{ supportUrl: null,
additionalInformation: '\nDriver info: org.openqa.selenium.chrome.ChromeDriver',
cause: null,
systemInformation: 'System info: host: 'asunar-VirtualBox', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'i386', os.version: '3.16.0-30-generic', java.version: '1.7.0_79'',
class: 'org.openqa.selenium.WebDriverException',
suppressed: [ [length]: 0 ],
hCode: 16165686,
message: 'unknown error: cannot find Chrome binary\n (Driver info: chromedriver=2.12.301324 (de8ab311bc9374d0ade71f7c167bad61848c7c48),platform=Linux 3.16.0-30-generic x86) (WARNING: The server did not provide any stacktrace information)\nCommand duration or timeout: 924 milliseconds\nBuild info: version: '2.43.1', revision: '5163bce', time: '2014-09-10 16:27:33'\nSystem info: host: 'asunar-VirtualBox', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'i386', os.version: '3.16.0-30-generic', java.version: '1.7.0_79'\nDriver info: org.openqa.selenium.chrome.ChromeDriver',
buildInformation:
{ releaseLabel: '2.43.1',
buildTime: '2014-09-10 16:27:33',
class: 'org.openqa.selenium.internal.BuildInfo',
buildRevision: '5163bce',
hCode: 33066190 },
localizedMessage: 'unknown error: cannot find Chrome binary\n (Driver info: chromedriver=2.12.301324 (de8ab311bc9374d0ade71f7c167bad61848c7c48),platform=Linux 3.16.0-30-generic x86) (WARNING: The server did not provide any stacktrace information)\nCommand duration or timeout: 924 milliseconds\nBuild info: version: '2.43.1', revision: '5163bce', time: '2014-09-10 16:27:33'\nSystem info: host: 'asunar-VirtualBox', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'i386', os.version: '3.16.0-30-generic', java.version: '1.7.0_79'\nDriver info: org.openqa.selenium.chrome.ChromeDriver' },
state: 'unhandled error',
class: 'org.openqa.selenium.remote.Response',
hCode: 15619088 }
Other than the comment in the webdriver.io boilerplate repo, I can't find a great deal of documentation around getting a multi-file test suite up and running. Would you consider adding more documentation around it?
Specifically I am interested in a multi-file webdriver.io boilerplate.
Thanks!
If you create a folder structure under <project>/tests/<some directory>/<testfile.js>
, the test cases are not recognized unless you change the conf/nightwatch.json with that specific directory. This will not be feasible at large scale when teams will like to create folders for their tracks. We need to ensure that the folders are ignored and just .js files are recognized as valid test cases at any level under tests/
This may be only possible for frameworks that send rich data back to magellan
, since we need to definitively know that the test has started running in selenium (session id acquisition may be a good way to signal this).
This would allow claims of VMs to more tightly track remote actual usage.
See:
https://github.com/TestArmada/boilerplate-mocha/issues/1
It would be great if Magellan could scan directories for mocha tests the same way mocha can with the recursive
option.
Some options:
mocha.opts
to see if the user has requested thisRight now most of the code in the magellan
binary in bin
is not testable, so we can't write tests for our initialization process, only the test runner. The magellan
init process in that binary needs to be separated out into a module that can be tested in our mocha unit tests.
... and perhaps think about adding a code of conduct also :)
$> "./node_modules/.bin/magellan" --browser=firefox --bail_fast
Sauce configuration OK
Magellan-nightwatch test iterator found nightwatch configuration at: ./conf/nightwatch.json
Running 3 tests with 3 workers with firefox
--> Worker 3, mock port: 12006, running test: tests/walmart-shirt.js @Firefox
--> Worker 2, mock port: 12003, running test: tests/walmart-book.js @Firefox
--> Worker 1, mock port: 12000, running test: tests/github.js @Firefox
Opens firefox browser, fails with the following error message:
'Unable to connect to host 127.0.0.1 on port 7055 after 45000 ms........'
Updated configuration to use selenium driver 2.46.0 like this:
"server_path": "./node_modules/testarmada-magellan-nightwatch/node_modules/selenium-server/lib/runner/selenium-server-standalone-2.46.0.jar"
$> "./node_modules/.bin/magellan" --browser=firefox --bail_fast
Sauce configuration OK
Magellan-nightwatch test iterator found nightwatch configuration at: ./conf/nightwatch.json
Running 3 tests with 3 workers with firefox
--> Worker 1, mock port: 12000, running test: tests/github.js @Firefox
--> Worker 2, mock port: 12003, running test: tests/walmart-book.js @Firefox
--> Worker 3, mock port: 12006, running test: tests/walmart-shirt.js @Firefox
(1 / 3) <-- Worker 1 PASS tests/github.js @Firefox
(2 / 3) <-- Worker 2 PASS tests/walmart-book.js @Firefox
(3 / 3) <-- Worker 3 FAIL tests/walmart-shirt.js @Firefox
.......
ERROR: Unable to locate element: "[data-magellan-temp-automation-id='magellan_click_6c0a5e2']" using: css selector
Unable to locate element: "[data-magellan-temp-automation-id='magellan_click_6c0a5e2']" using: css selector
FAILED: 1 errors and 1 passed (5.878s)
Running: Check product details
✖ Selector '.product-short-description' was not visible after 60117 milliseconds. - expected "visible" but got: not visible
λ "./node_modules/.bin/magellan" --browser=chrome --bail_fast
Sauce configuration OK
Magellan-nightwatch test iterator found nightwatch configuration at: ./conf/nightwatch.json
Running 3 tests with 3 workers with chrome
--> Worker 3, mock port: 12006, running test: tests/walmart-shirt.js @chrome
--> Worker 2, mock port: 12003, running test: tests/walmart-book.js @chrome
--> Worker 1, mock port: 12000, running test: tests/github.js @chrome
(1 / 3) <-- Worker 2 PASS tests/walmart-book.js @chrome
(2 / 3) <-- Worker 1 PASS tests/github.js @chrome
(3 / 3) <-- Worker 3 PASS tests/walmart-shirt.js @chrome
============= Suite Complete =============
Status: PASSED
Runtime: 27.8s
Total tests: 3
Successful: 3 / 3
If google or wikipedia change their page or go down, our tests shouldn't fail.
Let's perhaps borrow the test playground from webdriver.io and spin it up as a localhost server during tests, and assert against that.
Magellan should be able to continue to function if a brief locks
outage happens:
So for certain hidden customized device config we can only leave useful key-value pair in config file.
Right now profiles have to be in magellan.json
, but it would be useful for an organization to be able to centralize profiles in one place, to avoid having to update dozens of files when transitioning to new tiers of browsers (or devices).
centos 6.4
x64
java 1.7
{ [UnknownError: unknown error: cannot find Chrome binary
(Driver info: chromedriver=2.18.343837 (52eb4041461e46a6b73308ebb19e85787ced4281),platform=Linux 3.13.0-49-generic x86_64)]
code: 13,
state: 'unknown error',
message: 'unknown error: cannot find Chrome binary\n (Driver info: chromedriver=2.18.343837 (52eb4041461e46a6b73308ebb19e85787ced4281),platform=Linux 3.13.0-49-generic x86_64)',
name: 'UnknownError',
stack: 'UnknownError: unknown error: cannot find Chrome binary\n (Driver info: chromedriver=2.18.343837 (52eb4041461e46a6b73308ebb19e85787ced4281),platform=Linux 3.13.0-49-generic x86_64)\n at new bot.Error (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/atoms/error.js:108:18)\n at Object.bot.response.checkResponse (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/atoms/response.js:109:9)\n at /root/node_modules/browsertime/node_modules/selenium-webdriver/lib/webdriver/webdriver.js:160:24\n at promise.ControlFlow.runInFrame_ (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/goog/../webdriver/promise.js:1857:20)\n at goog.defineClass.notify (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/goog/../webdriver/promise.js:2448:25)\n at promise.Promise.notify_ (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/goog/../webdriver/promise.js:564:12)\n at Array.forEach (native)\n at promise.Promise.notifyAll_ (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/goog/../webdriver/promise.js:553:15)\n at goog.async.run.processWorkQueue (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/goog/async/run.js:125:21)\n at runMicrotasksCallback (node.js:337:7)\nFrom: Task: WebDriver.createSession()\n at Function.webdriver.WebDriver.acquireSession_ (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/webdriver/webdriver.js:157:22)\n at Function.webdriver.WebDriver.createSession (/root/node_modules/browsertime/node_modules/selenium-webdriver/lib/webdriver/webdriver.js:131:30)\n at new Driver (/root/node_modules/browsertime/node_modules/selenium-webdriver/chrome.js:783:36)\n at Builder.build (/root/node_modules/browsertime/node_modules/selenium-webdriver/builder.js:454:14)\n at Object.module.exports.getDriver (/root/node_modules/browsertime/lib/browsers/chrome.js:41:49)\n at /root/node_modules/browsertime/lib/browsertime.js:107:64\n at /root/node_modules/browsertime/node_modules/async/lib/async.js:744:13\n at /root/node_modules/browsertime/node_modules/async/lib/async.js:46:16\n at /root/node_modules/browsertime/node_modules/async/lib/async.js:260:32\n at /root/node_modules/browsertime/node_modules/async/lib/async.js:37:16' }
unknown error: cannot find Chrome binary
(Driver info: chromedriver=2.18.343837 (52eb4041461e46a6b73308ebb19e85787ced4281),platform=Linux 3.13.0-49-generic x86_64)
There are opportunities to split the testing infrastructure between singletons and per-worker components, along with some potential to decide how to distribute the load amongst workers. Some examples:
What are your thoughts - does that sound okay?
To be able to run firefox tests updated configuration to use selenium driver 2.46.0 like this:
"server_path": "./node_modules/testarmada-magellan-nightwatch/node_modules/selenium-server/lib/runner/selenium-server-standalone-2.46.0.jar"
Chrome only => all tests pass
Firefox only => walmart-shirt.js fails, the rest pass.
Chrome,Firefox parallel fails with the following error:
λ "./node_modules/.bin/magellan" --browser=chrome,firefox --bail_fast
...
(1 / 3) <-- Worker 1 FAIL tests/github.js @chrome,firefox
(2 / 3) <-- Worker 3 FAIL tests/walmart-shirt.js @chrome,firefox
(3 / 3) <-- Worker 2 FAIL tests/walmart-book.js @chrome,firefox
...
nightwatch-magellan has found nightwatch configuration at C:\Users\ASunar\boilerplate\temp\build-magellan-13e9e9364_24ccb1041__temp_assets\nightwatch.json
chrome TypeError: undefined is not a function
at emitStartedTest (C:\Users\ASunar\boilerplate\node_modules\testarmada-magellan-nightwatch\lib\base-test-class.js:70:13)
at Object.MagellanBaseTest.beforeEach (C:\Users\ASunar\boilerplate\node_modules\testarmada-magellan-nightwatch\lib\base-test-class.js:144:7)
at TestSuite. (C:\Users\ASunar\boilerplate\node_modules\nightwatch\lib\runner\testsuite.js:90:18)
at TestSuite.makePromise (C:\Users\ASunar\boilerplate\node_modules\nightwatch\lib\runner\testsuite.js:384:8)
at Object.beforeEach (C:\Users\ASunar\boilerplate\node_modules\nightwatch\lib\runner\testsuite.js:88:21)
at TestSuite.beforeEach (C:\Users\ASunar\boilerplate\node_modules\nightwatch\lib\runner\testsuite.js:323:21)
at TestCase.run (C:\Users\ASunar\boilerplate\node_modules\nightwatch\lib\runner\testcase.js:47:6)
at TestSuite.runTestCase (C:\Users\ASunar\boilerplate\node_modules\nightwatch\lib\runner\testsuite.js:284:20)
at TestSuite.runNextTestCase (C:\Users\ASunar\boilerplate\node_modules\nightwatch\lib\runner\testsuite.js:272:21)
at C:\Users\ASunar\boilerplate\node_modules\nightwatch\lib\runner\testsuite.js:220:19
The tunnel construction process can time out and stall. We should have timeouts and continue with what tunnels we have (existing reduced worker mode) if tunnels take too long to construct.
Reproduction
npm install
and npm run install-dev
magellan --tests=./test/func/spec/advanced.spec.js
(or any variation to get to just that test)$ magellan --test=test/func/spec/advanced.spec.js
Loaded magellan configuration from: /Users/rye/scm/wm/little-loader/magellan.json
Sauce configuration OK
Using test filter: test/func/spec/advanced.spec.js
Error: no tests found
If we have workers available, we should retry in parallel using those spare workers.
If a job is linked up with admiral, we show the link to the admiral URL after a suite is complete. But for jobs not linked with admiral, or if a user wants to jump directly to a sauce result from the console, we should change this:
(1 / 1) <-- Worker 1 PASS loads basic script @IE_8_Windows_2008_Desktop
to this:
(1 / 1) <-- Worker 1 PASS loads basic script @IE_8_Windows_2008_Desktop https://saucelabs.com/tests/1234567890
Support vanilla nightwatch in addition to the enhanced nightwatch integration we already have.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.