coveros-github-sandbox / selenified Goto Github PK

The Selenified Test Framework provides mechanisms for simply testing applications at multiple tiers while easily integrating into DevOps build environments. Selenified provides traceable reporting for both web and API testing, wraps and extends Selenium calls to more appropriately handle testing errors, and supports testing over multiple browsers locally, or in the cloud in parallel. It can be a great starting point for building or improving test automation in your organization.

Home Page: https://www.coveros.com/products/selenified/

License: Apache License 2.0

HTML 0.75% Java 99.25%

sauce-labs selenium-grid testng java selenium failsafe parallelization webpage webservices testing

selenified's People

Stargazers

Watchers

Forkers

otherdevopsgene qakart mattfirtion based2 amareshfnu ramesh-tech akhileshkumarverma ravichandra068 sampadrout youvegotnigel nghieptruong

selenified's Issues

Include Basic Service Integration

Include a class to allow Web Service calls to be simple made and analyzed that is incorporated into the framework

Support for SOAP

Instead of only allowing json payloads, allow xml payloads. This will inherently allow for SOAP API calls to be made

The selected browser PHANTOMJS is not an applicable choice

When trying to use PhantomJS browser, Selenified reports an error that the selected browser is not allowed.

mvn clean verify -Dbrowser=PhantomJS

[Selenified] [ERROR] com.coveros.selenified.exceptions.InvalidBrowserException: The selected browser PHANTOMJS is not an applicable choice
Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.448 sec <<< FAILURE! - in TestSuite

README Updates

Add a section in the readme to better describe how to use locators

Screensize

Would like a cmd parameter to set the screen size

API Test Reports are incorrectly formatted for some endpoints

When running API tests against certain endpoints, the "Actual Result" column of the HTML reports does not line up with the rest of the table.

Here's an example of one such endpoint along with a code snippet and screenshots of the reports.
https://api.github.com/users/johndoe

@BeforeClass(alwaysRun = true)
   public void beforeClass(ITestContext test) {
     setTestSite(this, test, "https://api.github.com/users/johndoe");  
     setAuthor(this, test, "Hammid Funsho\n<br/>[email protected]");
     setVersion(this, test, "0.0.1");
   }
  @Test(groups = { "services" }, description = "A sample services test to verify the response code")
   public void sampleServicesCityTest() {
       Call call = this.calls.get();
       Response response = call.get("");
       response.assertEquals(200);
       response.assertContains("johndoe");
       finish();
   }

Note: This appears to only affect certain endpoint

Make Elements OO

Currently the actions and asserts class is reminiscent of RC. We want to make it object oriented, i.e. have those classes implement the elements, so that each element defined can have any of these actions simply acted upon them

The README has sample code that doesn't include imports

It might be helpful to include imports in the sample code. like this:

import com.coveros.selenified.Locator;
import com.coveros.selenified.Selenified;
import com.coveros.selenified.application.App;
import com.coveros.selenified.element.Element;
import com.coveros.selenified.services.Call;
import com.coveros.selenified.services.Request;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

import java.util.HashMap;
import java.util.Map;

public class SampleTests extends Selenified {

...

Combined Reporting Desired

Right now, the failsafe directory has multiple test reports, all linked together via TestNG.
The request was made to create one combined report, that has all of the test cases executed, and then displays them all

Screenshots with Alerts broken

Currently, if an Alert is present, the takescreenshot method fails. This needs to be fixed

Add ability to reference child elements

Cleanup JavaDocs

When omitting the
-Xdoclint:none
line in the pom, multiple javadoc issues can be seen.
These should all be cleaned up

closeTab doesn't work

When tabs are opened by links in the page, windowHandles are actually created, and the open/close/switch tab functionality doesn't work. There needs to be some closeWindow funtionality to go along with switchToNewWindow and switchToParentWindow methods

Remove dependency on testng.xml file

Integrate this information into the POM, and provide default variables to set as a result, defining group, method, class, or package

Uploading Files Doesn't Work

When uploading a file, new stylizing often times hides the input area.
This means the input area isn't displayed, which means that Selenified throws an error and doesn't allow typing in it. This needs to be fixed

Package Test Results

Add ability to package individual test results

The command line to run the tests didn't work using Gradle

Ran the sample code on Windows 10 on the command line with Gradle, as follows:

$ gradlew seleniumTest -DappURL=google.com -Dbrowser=Firefox

FAILURE: Build failed with an exception.

* What went wrong:
Task 'seleniumTest' not found in root project 'demo'.

* Try:
Run gradlew tasks to get a list of available tasks. Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 1s

Here is the gradle file:


plugins {
    id 'war'

    // gretty is a gradle plugin to make it easy to run a server and hotswap code at runtime.
    id 'org.gretty' version '2.3.1'

    // provides access to a database versioning tool.
    id "org.flywaydb.flyway" version "5.2.4"

    // scans our code for static analysis
    id "org.sonarqube" version "2.6.2"

    // provides Cucumber reporting
    id "com.github.spacialcircumstances.gradle-cucumber-reporting" version "0.0.15"

    // provides unit test coverage
    id 'jacoco'

    // Dependency Check analyzes the dependencies for potential security issues.
    // see https://plugins.gradle.org/plugin/org.owasp.dependencycheck
    id "org.owasp.dependencycheck" version "4.0.1"

    // provides the ability to see a tree of dependent tasks
    // for any particular task.
    // usage: gradle <task 1>...<task N> taskTree
    // see https://github.com/dorongold/gradle-task-tree
    id "com.dorongold.task-tree" version "1.3.1"

    // pitest provides mutation testing coverage - this shows
    // which code-under-test is actually "tested", rather than
    // simply being run during the test.  It does this by
    // changing the code and seeing if the test fails as a result.
    id 'info.solidsoft.pitest' version '1.4.0'
}

// the checker framework adds a bunch of type annotations that
// can insure safer coding practices.  See https://checkerframework.org/manual/
apply from: "$rootDir/gradle/checkerframework.gradle"

// including our script plugin for running our integration tests
apply from: "$rootDir/gradle/fast_integration_test.gradle"

// including our script plugin for running BDD-type tests
apply from: "$rootDir/gradle/bdd_test.gradle"

repositories {
    jcenter()
    mavenCentral()
}

dependencies {
    providedCompile 'javax.servlet:javax.servlet-api:3.1.0'

    // junit, for running unit tests
    testCompile 'junit:junit:4.12'

    // mockito, for mocking objects in our unit tests
    testCompile 'org.mockito:mockito-core:2.7.19'

    // nbvcxz is a tool to determine how robust a password is.  See
    // https://github.com/GoSimpleLLC/nbvcxz
    compile group: 'me.gosimple', name: 'nbvcxz', version: '1.4.2'

    // gives us a driver to connect to postgresql
    // https://mvnrepository.com/artifact/org.postgresql/postgresql
    compile group: 'org.postgresql', name: 'postgresql', version: '42.2.5'

    // the glorious equalsverifier.  This is a tool that tests the contract
    // for equals and hashcode are met.  It's very strict.  Because it's very
    // strict, it forces us to create very safe, very solid code.
    // https://mvnrepository.com/artifact/nl.jqno.equalsverifier/equalsverifier
    testCompile group: 'nl.jqno.equalsverifier', name: 'equalsverifier', version: '3.0.3'

    // The commons-lang3 package has some utility classes that are very helpful
    // for those interested in safer coding.  Particularly, the equalsbuilder and hashcodebuilder.
    // https://mvnrepository.com/artifact/org.apache.commons/commons-lang3
    compile group: 'org.apache.commons', name: 'commons-lang3', version: '3.8.1'

    // a logging framework, so it becomes easily possible to see what is happening
    // in our code in realtime.
    compile group: 'org.apache.logging.log4j', name: 'log4j-api', version: '2.11.1'
    compile group: 'org.apache.logging.log4j', name: 'log4j-core', version: '2.11.1'

    testCompile 'com.coveros:selenified:3.0.3'
}

pitest {
    targetClasses = ['com.coveros.training.*']  //by default "${project.group}.*"
    threads = 4
    outputFormats = ['XML', 'HTML']
    timestampedReports = false
}

// configure jetty to run on port 8080 when we run "./gradlew appRun"
gretty {
    httpPort = 8080
    servletContainer = 'tomcat8'
}

// configuration for the cucumber reports.
cucumberReports {
    outputDir = file('build/reports/bdd')
    buildName = '0'
    reports = files('build/bdd/file.json')
    testTasksFinalizedByReport = false
}

flyway {
    url = 'jdbc:postgresql://localhost:5432/training'
    driver = 'org.postgresql.Driver'
    user = "postgres"
    password = "postgres"
    schemas = ['administrative', 'library', 'auth']
}

jacocoTestReport {
    executionData("$buildDir/jacoco/combineCoverage.exec")
}

// a way to combine the coverage reports for the purpose of seeing the
// whole picture when we run jacocoTestReport
task combineCoverage (type: JacocoMerge) {
    executionData = files("$buildDir/jacoco/test.exec", "$buildDir/jacoco/fastIntegrationTest.exec")
    mustRunAfter fastIntegrationTest
}

// Add the combined report when we run check
check.dependsOn(combineCoverage)
check.dependsOn(jacocoTestReport)


sonarqube {
    properties {
        property "sonar.sources", "src/main/java"
        property "sonar.tests", "src/test/java,src/integration_test/java,src/bdd_test/java"
        property "sonar.java.binaries", "build/classes/java/main"
        property "sonar.junit.reportPaths", "build/test-results/test/,build/test-results/fastIntegrationTest"
        property "sonar.jacoco.reportPaths", "build/jacoco/test.exec,build/jacoco/fastIntegrationTest.exec"
    }
}

Crashed Tests Don't Finish Report

Tests that truly error out don't even finish updating the detailed log report. Additionally, they show as passed, if no steps failed, but an error was thrown. This should be addressed

Add a way to check assertion by regex

There should be a way to check the assertion of a text of an element by regex. For example, if we were to check for a phone number, we expect it to have 10 digits and dashes within the string. We might not necessarily know what the phone number is, but we expect it to be in that format. So something like assertThat(“FooBarBaz”, matchesPattern(“^Foo”)); would be nice!

Update SauceLab runs

If running on SauceLabs, after test completion, SauceLabs just shows complete, not with a pass or fail. We need to call their API to update the result with the pass or fail

Typeo in Readme.md

call.get().("", new Request(params)).assertEquals(200);

In the SampleTests example class, the above line doesn't compile. I think it should probably be

call.get("", new Request(params)).assertEquals(200);

Selenium 3.0 Integration

Requested Features

Refreshing/Reloading the page
Clearing the cache

Wiki Page For Issues

We should have a wiki page for creating an issue. What info do they need to include, etc

Convert Wait Statements

Convert the current WaitFor class to use SE3's wait.until methods. These are more robust than our current wait loops, and can better handle stale elements, websocket responses, etc

Wiki page for Ant

We need a wiki page detailing runtime instructions for Ant, similar to what we have for Maven and Gradle.

PDF Reporting

Currently, the detailed reports are HTML. They have dynamic information displayed, such as images that can be toggled. Could these HTML reports also be produced as PDFs, with imbedded images, and proper formatting, so that they could be easily uploaded/attached as supporting materials

Remove host test site dependency

Cleanly Passing in Cloud

Several IE, Edge and Safari tests are failing. We need to fix these, mainly to ensure cross browser compatibility

Remove Web Services Third Party Dependency

Naming Collitions

When different classes have tests with the same name, iterators for looping through tests get broken. Need to fix the definition for the iterators to include classname as well

Add CONTRIBUTING file

A CONTRIBUTING.md explains how a participant should do things like format code, test fixes, and submit patches. Example: https://github.com/puppetlabs/puppet/blob/master/CONTRIBUTING.md

No Way To Test Table Cell Values

Looking for a compareTableCell in the asserts to verify the text in a particular table cell

Additional Response Asserts

Additional response checks are needed, for nested/more complicated results. For example:

/**
* Asserts that a response does not contain the entered key-value pairs.
* For arrays, this checks each object in the array.
*
* @param expectedPairs key-value pairs the should not be in the response
*/
public void assertNotContains(Map<String, String> expectedPairs)

/**
 * Verifies the response json payload contains an object at the key
 * that contains a nestedKey that contains a value
 * example response: { "books": { "id": 123 }, "author": "John Doe" }
 *
 * @param key       - top level json key eg. books in object above
 * @param nestedKey - key in nested object eg. "id" in object above
 * @param value     - must be String, Integer or Boolean
 */
public void assertContainsNested(String key, String nestedKey, Object value)

/**
 * Verifies the response does not contain a certain key-value pair.
 * For arrays, this checks each object in the array.
 *
 * @param key          key that should not be in the response
 * @param expectedJson value that the key should not have in the response
 */
public void assertNotContains(String key, JsonElement expectedJson)

/**
 * Verifies the response does not contain a certain key-value pair.
 * For arrays, this checks each object in the array.
 *
 * @param key key that should not be in the response
 */
public void assertNotContains(String key)

/**
 * Asserts that the response does not contain a certain JSON element.
 *
 * @param expectedJson JSON that should not be in the response
 */
public void assertNotContains(JsonElement expectedJson)

/**
 * Asserts that the response is a valid JSON array
 */
public void assertValidArrayData()

/**
 * Asserts that the response is an empty JSON array
 */
public void assertEmptyArrayData()

Create Wiki page for blog/vlog

Setting Request Header in API Testing

It would be very useful to be able to set request headers at will during each of the test case. Right now, we can modify the request body and the url parameters, but there's no easy way to set the request headers.

ZAPI Integration

Create a class to enable Zephyr integration

More API details can be found here: http://docs.getzephyr.apiary.io/#reference

Before All Tests

@BeforeSuite
Create a new test cycle
Post to /cycle ZAPI
Name - obtained from Bamboo based on build execution id
Build - obtained from Bamboo
Environment - obtained from Bamboo
Description - obtained from the suite description?
Start Date - today’s date
End Date - today’s date
Project ID - obtained from properties file?
Will return the cycle id - needs to be stored

Before Each Test

@BeforeMethod
Add the test to the test cycle
Post to /execution/addTestsToCycle API
Cycle ID - obtained from above
Issue ID - from annotation description

Add the test execution
Post to /execution API
Cycle ID - obtained from above
Issue ID - from annotation description
Project ID - obtained from properties file?
Assignee - Bamboo
Will return the execution id - needs to be stored

After each test

@AfterMethod
Update the test status
Put to /execution/[ID]/execute API
Execution ID - obtained from above
Status - The Pass/Fail/Skipped result

Update README

Per Alan's feedback/comments

There's a possible problem ingesting very large json list data

I've been getting NPE when trying to ingest a large json list data.

Response res = call.get(url, req);
res.getMessage(); //this prints the json just fine
JsonObject json = res.getObjectData(); //this throws NPE

Fixing logging levels to cleanup output

Increase Coverage

New Features

Include new features such as drawing capabilities, an updated find child elements

Handle Blocking Alerts Gracefully

The README does not emphasize the command to run the test early enough

I found that when following the README, after the sample code it was quite a while further down the document before I found the command to run the tests. It seems like someone just dipping their toes in the water with this framework might not scroll down to nearly the bottom of the document where those instructions are found. Suggest putting those right after the sample code.

Parallelization issues with TestNG 6.10

Maybe look at using 6.9.9

Sample code doesn't pass

When I ran the sample code in IntelliJ, I got no passes and only errors. Here's what I did:

I added Selenified as a dependency in my gradle buildfile.

I then created a file called SampleTests and put the sample code there.

Because there weren't any imports, I made assumptions about what was needed, and I think I got it right. Here were the imports I used:

import com.coveros.selenified.Locator;
import com.coveros.selenified.Selenified;
import com.coveros.selenified.application.App;
import com.coveros.selenified.element.Element;
import com.coveros.selenified.services.Call;
import com.coveros.selenified.services.Request;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

import java.util.HashMap;
import java.util.Map;

In the method "sampleServicesCityTest", there was an error:

call.get("", new Request(params)).assertEquals(200);

This method expected a constructor that didn't exist. I changed it as follows:

@Test(groups = { "sampleServices" }, description = "A sample web services test to verify the response code")
 public void sampleServicesCityTest() {
   Map<String, Object> params = new HashMap<>();
   params.put("address", "chicago");
   // use this object to verify the app looks as expected
   Call call = this.calls.get();
   // retrieve the zip code and verify the return code
   final Request request = new Request();
   call.get("", request.setUrlParams(params)).assertEquals(200);
   // verify no issues
   finish();
 }

Following is the error log when running these tests in Intellij:

[Selenified] [WARN] org.openqa.selenium.WebDriverException: java.net.MalformedURLException
Build info: version: '3.12.0', revision: '7c6e0b3', time: '2018-05-08T14:04:26.12Z'
System info: host: 'DESKTOP-4GITO6A', ip: '192.168.56.1', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_171'
Driver info: driver.version: HtmlUnitDriver

java.lang.NullPointerException
	at com.coveros.training.SampleTests.sampleServicesCityTest(SampleTests.java:59)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
	at org.testng.TestRunner.privateRun(TestRunner.java:648)
	at org.testng.TestRunner.run(TestRunner.java:505)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
	at org.testng.SuiteRunner.run(SuiteRunner.java:364)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
	at org.testng.TestNG.runSuites(TestNG.java:1049)
	at org.testng.TestNG.run(TestNG.java:1017)
	at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
	at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)

[Selenified] [WARN] org.openqa.selenium.WebDriverException: java.net.MalformedURLException
Build info: version: '3.12.0', revision: '7c6e0b3', time: '2018-05-08T14:04:26.12Z'
System info: host: 'DESKTOP-4GITO6A', ip: '192.168.56.1', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_171'
Driver info: driver.version: HtmlUnitDriver

java.lang.AssertionError: Detailed results found at: com_coveros_sampleTestHTMLUNIT.html 
Expected :0 errors
Actual   :2 errors
 <Click to see difference>


	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:101)
	at com.coveros.selenified.Selenified.finish(Selenified.java:447)
	at com.coveros.training.SampleTests.sampleTest(SampleTests.java:30)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:989)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
	at org.testng.TestRunner.privateRun(TestRunner.java:648)
	at org.testng.TestRunner.run(TestRunner.java:505)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
	at org.testng.SuiteRunner.run(SuiteRunner.java:364)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
	at org.testng.TestNG.runSuites(TestNG.java:1049)
	at org.testng.TestNG.run(TestNG.java:1017)
	at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:73)
	at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)

[Selenified] [WARN] org.openqa.selenium.WebDriverException: java.net.MalformedURLException
Build info: version: '3.12.0', revision: '7c6e0b3', time: '2018-05-08T14:04:26.12Z'
System info: host: 'DESKTOP-4GITO6A', ip: '192.168.56.1', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_171'
Driver info: driver.version: HtmlUnitDriver
[Selenified] [WARN] org.openqa.selenium.WebDriverException: java.net.MalformedURLException
Build info: version: '3.12.0', revision: '7c6e0b3', time: '2018-05-08T14:04:26.12Z'
System info: host: 'DESKTOP-4GITO6A', ip: '192.168.56.1', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_171'
Driver info: driver.version: HtmlUnitDriver
[Selenified] [WARN] org.openqa.selenium.WebDriverException: java.net.MalformedURLException
Build info: version: '3.12.0', revision: '7c6e0b3', time: '2018-05-08T14:04:26.12Z'
System info: host: 'DESKTOP-4GITO6A', ip: '192.168.56.1', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_171'
Driver info: driver.version: HtmlUnitDriver

java.lang.AssertionError: Detailed results found at: com_coveros_sampleTestWDataProviderWithOptionPythonHTMLUNIT.html 
Expected :0 errors
Actual   :5 errors
 <Click to see difference>


	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:101)
	at com.coveros.selenified.Selenified.finish(Selenified.java:447)
	at com.coveros.training.SampleTests.sampleTestWDataProvider(SampleTests.java:48)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719)
	at org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:71)
	at org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:14)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: Detailed results found at: com_coveros_sampleTestWDataProviderWithOptionBashHTMLUNIT.html 
Expected :0 errors
Actual   :5 errors
 <Click to see difference>


	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:101)
	at com.coveros.selenified.Selenified.finish(Selenified.java:447)
	at com.coveros.training.SampleTests.sampleTestWDataProvider(SampleTests.java:48)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719)
	at org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:71)
	at org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:14)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)


java.lang.AssertionError: Detailed results found at: com_coveros_sampleTestWDataProviderWithOptionPerlHTMLUNIT.html 
Expected :0 errors
Actual   :5 errors
 <Click to see difference>


	at org.testng.AssertJUnit.assertEquals(AssertJUnit.java:101)
	at com.coveros.selenified.Selenified.finish(Selenified.java:447)
	at com.coveros.training.SampleTests.sampleTestWDataProvider(SampleTests.java:48)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:583)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:719)
	at org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:71)
	at org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:14)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Attached are the html reports generated.

Parent Frame issues

Add select parent and select main frame methods for app

Update Setup Instructions

Include more detailed instructions for maven to include failsafe plugin, and testng setup

Marionette Doesn't Work

Running with the MarionnetteDriver (selecting Marionette for the browser) doesn't work. This means that versions of Firefox above 47 don't work.
Error throw:
startTest(samples.SampleIT): Returned value cannot be converted to WebElement: {stacktrace=stack backtrace

Additional ChromeOptions

When configuring the Chrome browser, the tests should be able to add additional options to the ChromeOptions, i.e. options.addArgument("--my_argument_here").

Basically the person writing the tests should have control of how the browser is setup.