wix / detox

Gray box end-to-end testing and automation framework for mobile apps

Detox

Gray box end-to-end testing and automation framework for mobile apps.

What Does a Detox Test Look Like?

This is a test for a login screen, it runs on a device/simulator like an actual user:

describe('Login flow', () => {
  beforeEach(async () => {
    await device.reloadReactNative();
  });

  it('should login successfully', async () => {
    await element(by.id('email')).typeText('[email protected]');
    await element(by.id('password')).typeText('123456');

    const loginButton = element(by.text('Login'));
    await loginButton.tap();

    await expect(loginButton).not.toExist();
    await expect(element(by.label('Welcome'))).toBeVisible();
  });
});

Get started with Detox now!

About

High velocity native mobile development requires us to adopt continuous integration workflows, which means our reliance on manual QA has to drop significantly. Detox tests your mobile app while it’s running in a real device/simulator, interacting with it just like a real user.

The most difficult part of automated testing on mobile is the tip of the testing pyramid - E2E. The core problem with E2E tests is flakiness - tests are usually not deterministic. We believe the only way to tackle flakiness head on is by moving from black box testing to gray box testing. That’s where Detox comes into play.

• Cross Platform: Write end-to-end tests in JavaScript for React Native apps (Android & iOS).
• Debuggable: Modern async-await API allows breakpoints in asynchronous tests to work as expected.
• Automatically Synchronized: Stops flakiness at the core by monitoring asynchronous operations in your app.
• Made For CI: Execute your E2E tests on CI platforms like Travis CI, Circle CI or Jenkins without grief.
• Runs on Devices: Gain confidence to ship by testing your app on a device/simulator just like a real user (not yet supported on iOS).
• Test Runner Agnostic: Detox provides a set of APIs to use with any test runner without it. It comes with Jest integration out of the box.

Supported React Native Versions

Detox was built from the ground up to support React Native projects.

While Detox should work out of the box with almost any React Native version of the latest minor releases, official support is provided for React Native versions 0.71.x, 0.72.x and 0.73.x without React Native's "New Architecture".

Newer versions, as well as React Native's "New Architecture", may work with Detox, but have not been tested out yet by the Detox team.

Although we do not officially support older React Native versions, we do our best to keep Detox compatible with them.

Also, in case of a problem with an unsupported version of React Native, please submit an issue or write us in our Discord server and we will do our best to help out.

Known Issues with React Native

• Visibility edge-case on Android: see this RN issue.

Get Started with Detox

Read the Getting Started Guide to get Detox running on your app in less than 10 minutes.

Documents Site

Explore further about using Detox from our new website.

Core Principles

We believe that the only way to address the core difficulties with mobile end-to-end testing is by rethinking some of the principles of the entire approach. See what Detox does differently.

Contributing to Detox

Detox has been open-source from the first commit. If you’re interested in helping out with our roadmap, please see issues tagged with the label. If you have encountered a bug or would like to suggest a new feature, please open an issue.

Dive into Detox core by reading the Detox Contribution Guide.

License

• Detox is licensed under the MIT License

Non-English Resources (Community)

• Getting Started (Brazilian Portuguese)

detox's Issues

add not visible matcher

Fixed in #32

Red screen - Invariant Violation

getting this red screen after success installation:

reason for test failure stopped being reported

when a test fails we don't get the reason

Move fbsimctl to a different npm package (detox-tools)

Since it takes a lot of space, we don't want to update it when it doesn't change

Add toMatch by id matcher

In order to select components in the tests we use the testID property of course.

Right now you can't check the text by testID. It'll be great if we could do it.
Another great thing is to implement the toMatch matcher so we won't break on every tiny text change.

Protractor syntax is:

expect($('#my-id').getText()).toMatch('some partial text');

P.S - if the toMatch is too complicated having the ability to check the text by Id is also good enough.

Go team WIX!

Add option to sendkeys to input

The thing that will really let us boost up our tests and give us much better coverage and usage testing is giving the option to set text and clear an input field.

Protractor syntax is:

$('#address-input').sendKeys(value);
$('#address-input').clear()

10x

Create a CLI module

Will make working with the project a breeze. Instead of defining stuff in the package.json, pass arguments to the cli, for example: detox build ios/MyApp.xcproj MyAppDev, detox run e2e/,
then the cli automatically figures out what and where to build.
Then, in the user's package.json, he can define those scripts once like so "e2e": "detox build ios/MyApp.xcproj MyAppDev && detox run e2e/", and forgo all other installation and configuration coupling to the build path

Command line arg to keep simulator open

Need to see where it stopped :-/

Change API to use Promises instead of callbacks

For example, I would like to be able to use async/await when calling simulator.deleteAndRelaunchApp()

Handle iOS permission dialogs

EarlGrey has a hard time dealing with them since they're external to the app

google/EarlGrey#55

this thread offers some interesting workarounds.. we can add them directly into detox

Add detox preStart script in config json

For example:

"detox": {
   **"beforeStart": "npm run fake-server",**
    "session": {
      "server": "ws://localhost:8099",
      "sessionId": "abc"
    },
    "ios-simulator": {
      "app": "ios/DerivedData/WixOneApp/Build/Products/Debug-iphonesimulator/ABC.app",
      "device": "iPhone 6s, iOS 10.1"
    }
  }

Create a simple npm script to fix EarlGrey build issues

if you get build problems, delete the following folders (since EarlGrey downloads them on build):
node_modules/detox/ios/EarlGrey/OCHamcrest.framework
node_modules/detox/ios/EarlGrey/fishhook
node_modules/detox/ios/EarlGrey/Tests/UnitTests/ocmock

Handle NSLog better - output it during tests

we should have NSLog available for crashes.. if detox doesn't work, usually NSLog can explain what went wrong and help us debug

if it's saved on the CI, it could also help us a lot to investigate failed test runs

also, it will be very nice to add tests based on NSLog, for example verify that during the last test, there were no console.errors or console.warnings

RN release build fails in e2e due to sync problems

Add ability to enter type a text in a field

In ios I believe this essentially means supporting grey_typeText

Refactor invoke queue into promise queue and support inline awaits

All lines in a test scenario are naturally asynchronous since the device has to perform each interaction and that takes time. In order to avoid the boilerplate of writing await before every line, we use a concept similar to what protractor does for its Control Flow.

Unlike protractor, our current implementation simply manages an invocation queue and then executes it until the test terminates.

This is not ideal. We need to switch to a pending promise queue like protractor.

In addition, we want to support a manual inline await in the middle of a test. The reason is that you might want to wait on something external, for example restart the simulator in the middle of a test.

make "npm run e2e" work again

cd detox/detox
npm install
npm run build
npm run test-clean
npm run test-install
# don't run this now: npm run test
cd test
npm run e2e

The last line is how end users (not people developing detox itself) will eventually run detox
this means any parameters you want to send need to be designed to make sense there, like:
npm run e2e -- --scheme=ios-simulator.release
npm run e2e -- --verbose

If you want to improve the scripts for people developing detox, this must come as step 2 after the API for end users is finished and working

rnpm link support

Support multiple test schemes - like Debug and Release

each scheme needs its definitions in package.json because the app binary might be located in different places

we should specify which scheme we want using command line arguments to npm run e2e

the default scheme should be ios-simulator.debug

Improve the demo-native shell script that copies binary outside

right now we instruct people to copy-paste the script into xcode as a post build step

cd ${SRCROOT}
if [ ! -d "build" ]; then
    mkdir build
fi
cp -rf ${BUILT_PRODUCTS_DIR}/../ /${SRCROOT}/build/Products

we should instead implement is as a bash script that will be available inside detox, and then just reference the script from xcode

simulator.deleteAndRelaunchApp with fbsimctl is still slower than apple's

fbsimctl uninstall org.reactjs.native.example.example

vs

xcrun simctl uninstall booted org.reactjs.native.example.example

the apple version is still faster, we need to improve fbsimctl

add app reuse option

Done in #45

Add support in iOS alerts

Need a means to detect whether an alert is displayed, and if so - need to be able to verify the content (title, message, buttons).

Add ability to wait for element to exist

Or in general, wait for a matcher/condition to be satisfied.

output UI Hierarchy on test failure

Fixed in #33

what's the best way to match tabs in iOS?

requires some research

what's the best way to match the different tab icons?

need to add this to the api

Improve syncing between tester and testee by introducing isReady and ready

Before starting a new session, the tester will ask isReady and will wait for ready.
When ready for a new session, or when asked isReady, the testee will send ready.
This will make sure the session only starts when the testee is ready

typeText and clearText sometimes fail due to keyboard sync

We sometimes have sync problems with the keyboard not shown yet:

2016-10-08 01:47:39.567 example[38497:4275833] Detox Test Failed: Element matcher: (respondsToSelector(accessibilityIdentifier) && accessibilityID("UniqueId005")) Complete Error: Error Domain=com.google.earlgrey.ElementInteractionErrorDomain Code=1 "Failed to type string "�����������������", because keyboard was not shown on screen." UserInfo={NSLocalizedDescription=Failed to type string "�����������������", because keyboard was not shown on screen.}

Failing tests pass, afterEach fails

When a test fails, the terminal marks it as passed, and the afterEach hook fails.

Expected: Failing tests should fail.

git submodules prevent from switching branches

maybe a new approach?

react-native adds flakiness to tests due to its async nature

the way to fix flakiness is to work on synchronization

EarlGrey synchronizes by waiting for the device to be idle:
https://github.com/google/EarlGrey/blob/master/docs/api.md#synchronization

we should probably add to EarlGrey the same ability to synchronize, but on RN instead:

1. on the bridge (like EG waits for network to be idle, maybe we can wait for the bridge to cool down from pending tasks)
2. on the JS event queue (since there are pending tasks, we should probably wait until they finish before moving forward)

this is not an easy research task

React Native 0.34+ not supported - testee not connected, cannot fw action

I'm still having this this issue #20 with detox 3.2.0. ( Also tried on 3.1.0 )

06:12:37: server listening on localhost:8099...
06:12:41: role=tester login (sessionId=mytestapp)
06:12:47: role=tester action=isReady (sessionId=mytestapp)
06:12:47: role=testee not connected, cannot fw action (sessionId=mytestapp)
06:12:47: role=testee login (sessionId=mytestapp)
06:12:47: role=testee action=ready (sessionId=mytestapp)
06:12:47: role=tester action=reactNativeReload (sessionId=mytestapp)
06:13:07: role=tester action=cleanup (sessionId=mytestapp)
06:13:07: role=testee action=cleanupDone (sessionId=mytestapp)
06:13:07: role=tester disconnect (sessionId=mytestapp)

Simulators starts app but test isn't executed. I tried on iphone 6s, 6, 5s simulator as well and it was the same.

Need to update dependencies - issues with Xcode 8, MacOS Sierra

The symptom you see is that the "testee" does not connect. The underlying reason is that Detox dynamic framework tries to load and fails because one of the dependencies throws an exception in runtime.

This project depends on several "hardcore" projects that are tightly coupled to internals and therefore might break on major version changes of Xcode and the OS:

1. EarlGrey - the version we depend upon does not load successfully after being compiled on Xcode 8
2. FBSimCtl - the version we depend upon does not work well on MacOS Sierra
3. RocketSocket - the version we depend upon also has a few problems under this setting

The solution is obvious, update these dependencies to latest version which solve all the issues. While we're at it, it's a good time to make the dependencies dynamic so they will update automatically by themselves from now on:

1. EarlGrey - rely on it as a git submodule that we can update easily by tag and change releases
2. FBSimCtl - download it pre-compiled from brew, this promises we get a correct ver per OS
3. RocketSocket - also change to rely on it as a git submodule

Improve iOS React Native sync mechanism

EarlGrey on iOS synchronizes with async OS tasks such as network requests using objects that conform to GREYIdlingResource. These objects listen on a resource like the network stack and are polled by EarlGrey to say whether this resource is busy or not (isIdleNow).

Our current phase 1 sync mechanism has 2 new Idling Resources specific for React Native.

The current implementation uses heuristics (moving average of the time that takes a bridge command to travel the full cycle) and this isn't ideal.

We want to improve the sync mechanism, remove the heuristics and make it more robust.

Support more than 1 action on simulator

This code works:

    it('starts with video view on first launch', (done) => {
      global.simulator.deleteAndRelaunchApp(() => {
        expect(element(by.id('VideoView'))).toBeVisible();
        done();
      });
    });

    it('does not display video view on second launch', (done) => {
      global.simulator.reloadReactNativeApp(() => {
        expect(element(by.id('VideoView'))).toNotExist();
        done();
      });
    });

while this fails (prints error: _onTestResult is undefined on testFailed):

    it('starts with video screen on first launch only', (done) => {
      global.simulator.deleteAndRelaunchApp(() => {
        expect(element(by.id('VideoView'))).toBeVisible();
        global.simulator.reloadReactNativeApp(() => {
          expect(element(by.id('VideoView'))).toNotExist();
          done();
        });
      });
    });

I tracked the reason to this

Also, #57 would make this code much prettier

Remote testing

Do you guys have this working remotely in some sort of continuous integration environment? (i.e. CircleCi)

add match by type

fixed in #40

Add mode to _currentScheme

In order to determine if we are in debug or release mode, we need to add a key to the _currentScheme in simulator.js.

Something like:

{
    app: 'ios/build/Build/Products/Debug-iphonesimulator/example.app',
    mode: 'debug',
    device: 'iPhone 6s, iOS 9.3'
}

Android version

Phase 1 - react-native-invoke for Java

Phase 2 - Integrate with Espresso

Add ability to swipe a view

In particular, swipe up a ListView in order to trigger "fetch more items" action.

Add navigation support

Need to be able to check various navigation scenarios, for example:

1. Is this there root screen? - yes/no
2. (Mainly) for inner screens: get navigation bar info (e.g. title text) and nav buttons captions.
3. Same for modals...
4. Get the app name of current screen (e.g. Safari)

Also, need to be able to perform a 'back' navigation action.

Support expectations on app logs

Allow to write tests over the app log

For example, expect that we have no errors or warnings in log

Match by selector

Support CSS-like syntax.
For example: element(by.selector("#parentId @UIImage"))
Will get the first element of type UIImage which is a descendent of an element with testID parentId

error trying to run e2e tests

Darwin 15.5.0
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "run" "e2e"
npm ERR! node v5.8.0
npm ERR! npm v3.8.1
npm ERR! code ELIFECYCLE
npm ERR! [email protected] e2e: mocha e2e --opts ./e2e/mocha.opts
npm ERR! Exit status 2
npm ERR!
npm ERR! Failed at the [email protected] e2e script 'mocha e2e --opts ./e2e/mocha.opts'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the rn-example package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! mocha e2e --opts ./e2e/mocha.opts
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs rn-example
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls rn-example
npm ERR! There is likely additional logging output above.

App freezes on launch

On react-native 0.25.1.
Sometimes, when detox starts the app, it freezes (showing video view screen), prints before all in the console, then after a few seconds stops the app and exits with exception timeout exceeded.

FBSimCtl is very slow

All simulator commands take forever, especially deleting an app
We need to improve its performance
One idea is to work with GUIDs directly and not with the models types, it probably enumerates all devices for every command and that takes longer

role=testee not connected

I'm having the following logs on detox-server

login sessionId=MyApp role=tester
fw sessionId=MyApp action=isReady from role=tester
cannot fw sessionId=MyApp since other role=testee not connected

I'm also having timeout as this issue: #20

ability to select non-accessible elements

For example, the navigator's Back UIAButton

simulator.relaunchApp is very slow

seems like an issue with fbsimctl

facebook/idb#289

we need to fix that because it adds impossible delays to the tests

ideas:

1. research the fbsimctl issue, why is it so slow?
2. add a detox endpoint to kill the app manually - exit(0) - and then launch instead of relaunch
3. use the apple xcrun simctl interface

Don't automatically install fbsimctl

This seems intrusive to do that as a postinstall step, better ask that in the installation instructions, and throw exceptions on the postinstall step.