The Fork project consists of these tools:
- Fork offers the fastest way to execute Android instrumentation test suites.
- Flakiness Reporter produces readable reports about test flakiness on tests suites previously executed by Fork.
- Chimprunner runs performance tests and keeps stats and standard reports (Systrace, memory, CPU, etc).
When running instrumentation tests, there is a significant time overhead for developers, especially on larger test suites. Existing solutions were not satisfactory for quick feedback before pushing code to VCS and for CI purposes.
We are big fans of Spoon and were using it for our plans, so we used it as our starting point. However, Spoon had similar issues to the Gradle and Maven test execution plugins, in the sense that it executes all tests on all of the connected devices (and emulators). We decided to tweak that naive scheduling to achieve much faster test execution.
We introduced the notion of pools of devices. These are now responsible for running a test suite instead of each device running the suite separately. That has two side effects:
- infinite scaling: your tests can speed up by as many devices and emulators as you can dedicate to your CI box.
- because test suites now get scheduled to run on a pool, not all tests will run on all devices. For that reason, we also introduced a way to create a pool per device, which offers full coverage (a.k.a. Spoon-mode) but typically takes longer, so we run it on a nightly basis. Fork works out-of-the-box, without any code changes.
There are two ways to run Fork with your builds.
First, you need to add a build-script dependency. You can have access to snapshot builds, but stable versions are recommended.
buildscript {
dependencies {
classpath 'com.shazam.fork:fork-gradle-plugin:3.16.0'
}
}buildscript {
repositories {
// other repos ...
maven { url 'https://oss.sonatype.org/content/repositories/snapshots' }
}
dependencies {
classpath 'com.shazam.fork:fork-gradle-plugin:3.17.0-SNAPSHOT
}
}Apply the Fork plugin
apply plugin: 'com.shazam.fork'You're now done. If you had any instrumentation test tasks before, the plugin has added Fork tasks. You can verify by running:
gradlew tasks | grep forkYou can use Fork's DSL to configure its execution parameters. For a full list of the properties, have a look at: Configuring pools and runtime and related Examples. It should be as easy as adding a block to your build.gradle:
fork {
title = "My acceptance tests"
testPackage = "com.example.tests"
poolingStrategy {
computed {
characteristic = "sw"
groups {
allDevices = 0
}
}
}
}Note: The testPackage property refers to the base package name which your tests reside in. If you have tests in multiple packages, provide the most common ancestor.
Check out the Fork project and execute:
> gradlew fork-runner:run -Pargs='ARGUMENTS LIST'
Available options (* mandatory).
--sdk Path to Android SDK. Defaults to the ANDROID_HOME environment variable.
--apk (*) Path to application.
--test-apk (*) Path to test application.
--config (*) Location of the configuration file. The properties of the configuration file are described in: Configuring pools and runtime and related Examples.
One of the most useful characteristics of the library is the way it creates the device pools. There are different options, to automatically create pools by API level, shortest width dimension and whether devices are self-described as tablets. On top of that, users can also manually create pools based on serial numbers, for maximum flexibility.
With either way of executing Fork (Gradle / Standalone) you can specify how the pools are created by setting a combination of the properties below. You can also find examples:
| Property Name | Property Type | Default value |
|---|---|---|
| baseOutputDir | File | "fork" |
| ignoreFailures | boolean | false |
| isCoverageEnabled | boolean | false |
| testClassRegex | String | "^((?!Abstract).)*Test$" |
| testPackage | String | (Your instrumentation APK package) |
| title | String | - |
| subtitle | String | - |
| testOutputTimeout | int | 60000 |
| testSize | String | - |
| excludedSerials | Collection<String> | - |
| totalAllowedRetryQuota | int | 0 |
| retryPerTestCaseQuota | int | 1 |
| autoGrantPermissions | boolean | true |
| poolingStrategy | PoolingStrategy | - |
| excludedAnnotation | String | (tests with this annotation are excluded) |
| applicationApkName | String | - |
| instrumentationApkName | String | - |
Poolingstrategy is an object that describes how the device pools are created. You can choose only one strategy from below:
| Property Name | Property Type | Description |
|---|---|---|
| splitTablets | boolean | configure pools depending on their manufacturer's 'tablet' flag (ro.build.characteristics) |
| eachDevice | boolean | create a pool per device (a.k.a. Spoon-mode). This is the default behaviour. |
| manual | ManualPooling | configure pools manually, per serial number |
| computed | ComputedPooling | configure pools automatically, by setting device criteria (API level, shortest width dimension) |
ManualPooling's properties are:
| Property Name | Property Type | Description |
|---|---|---|
| groupings | Map<String, Collection<String>> | map pool names to collection of device serials to be assigned to that pool |
ComputedPooling's properties are:
| Property Name | Property Type | Description |
|---|---|---|
| characteristic | Characteristic | Possible values: "sw"|"api" (shortest width or API level) |
| groups | Map<String, Integer> | map the name of a pool to their lowest dimension for a characteristic |
By default Fork auto-grants all runtime permissions on Android Marshmallow +. It is possible anyway to selectively revoke one or more permissions per single test case.
To do so, you have to add an annotation called RevokePermission. Here is an example:
@Test
@RevokePermission({Manifest.permission.RECORD_AUDIO,
Manifest.permission.ACCESS_FINE_LOCATION})
public void aTestRevokingRecordAudioAndFineLocation() {
//in here RECORD_AUDIO and ACCESS_FINE_LOCATION are *not* granted.
}Remember to add the fork client-side library to your project to have access to the annotation. To do so, in your app's dependencies add:
androidTestImplementation "com.shazam.fork:fork-client:3.16.0"After every test case, all the runtime permissions will be automatically re-granted even if the test fails. This feature will impact only Marshmallow and subsequent devices.
Using Fork you can set metadata on tests and get them back in its JUnit xml reports. The metadata are added as additional property tags on the suite level of the report, as each test produces its own report.
@Test
@TestProperties(keys = {"k1", "k2"}, values = {"v1", "v2"})
public void testWithProperties() {
// Test normally here
}Note that Fork will stop adding pairs after it encounters an unpaired key or value, so make sure you have the same number of keys and values.
A common case can be that you want to create two pools, one for phones & small tablets (7" and below) and one for large tablets. You could add to your build.gradle file:
fork {
title "Fork report"
subtitle "automatically split phones to large tablets"
poolingStrategy {
computed {
characteristic = "sw"
groups {
phablets = 0
tablets = 720
}
}
}
}The above will run tests on 2 pools, one named "phablets" and another called "tablets". The smallest width for the first pool will be 0 and for the latter 720 dpi.
fork {
title "Fork report"
subtitle "manually allocated devices"
poolingStrategy {
manual {
groupings {
phablets = ["ABCDE", "FGHIJ"]
tablets = ["12345"]
emulators = [".*:5555"]
}
}
}
}That will create three pools named "phablets", "tablets" & "emulators" with devices that have the respective serial numbers. Emulators will be selected automatically by a regexp.
Execute the following:
> gradlew fork-runner:run -Pargs='--apk /path/to/production.APK --test-apk /path/to/test.APK --config /path/to/fork-config.json'Where the contents of fork-config.json are:
{
"title" : "Fork Report",
"subtitle" : "automatically split phones to tablets",
"poolingStrategy" : {
"computed" : {
"characteristic" : "sw",
"groups" : {
"phablets" : 0,
"tablets" : 720
}
}
}
}Execute the following:
> gradlew fork-runner:run -Pargs='--apk /path/to/production.APK --test-apk /path/to/test.APK --config /path/to/fork-config.json'Where the contents of fork-config.json are:
{
"title" : "Fork Report",
"subtitle" : "manually allocated devices",
"poolingStrategy" : {
"manual" : {
"phablets" : ["ABCDE", "FGHIJ"],
"tablets" : ["12345"]
}
}
}- The scheduling still works on a single build box with ADB, so there still is a limit by how many devices & emulators can be simultaneously connected to ADB. Eventually, Fork could be tweaked to talk over HTTP with other build agents, that would then be connected to devices over ADB. That model would tie in nicely with multi-agent CI systems, like Jenkins.
One common problem with UI tests is the test flakiness from either the environment they run on or badly written tests. To help track down tests that are misbehaving, we introduced the Flakiness reporter.
The reports produced by the Flakiness Reporter eventually make it trivial to find flaky tests and link to them and their diagnostics. Currently Jenkins is supported and it should be really easy to extend it to other types of CI servers.
The Flakiness Reporter collects Fork output files, matches test runs over previous builds and sorts them according to their flakiness. Links are also created to each test of each test run, for easy navigation to diagnostics.
The output after a successful run of the Flakiness Reporter looks like the following:
The Gradle plugin that allows the Reporter to run can be applied to a standalone project, since it doesn't directly depend on your Android project. For convenience, however, that is a good compromise.
Currently, the Reporter supports Jenkins but plugins can be written to be used with other CI servers.
To be able to use the Flakiness Reporter add these dependencies:
buildscript {
dependencies {
classpath "com.shazam.fork:fork-reporter-jenkins-gradle-plugin:3.16.0"
}
repositories {
maven { url "https://repo.jenkins-ci.org/public/" }
}
}Apply the Jenkins Flakiness Reporter plugin
apply plugin: 'com.shazam.fork.reporter.jenkins'You can easily execute the Reporter with the following command.
gradlew forkJenkinsReportTo allow the Reporter communicate with your Jenkins server, you need to configure it with some basic details about your Jenkins Plan
| Property Name | Property Type | Description |
|---|---|---|
| reportTitle | String | The title you want your report to have |
| jenkinsUrl | String | The base URL of your Jenkins Server |
| jenkinsJobName | String | The name of the job you want to be tracked |
| jenkinsReportTitle | String | Optional, used to link to Fork diagnostics. The report title you use to archive Fork's report folder |
| escapeUnderscores | boolean | Optional, should have the same value as HTML Publisher (default: true) |
An example of a configuration:
forkJenkins {
reportTitle = "My project's awesome flakiness report"
jenkinsUrl = "http://my-jenkins.server.net:8080/"
jenkinsJobName = "Master"
jenkinsReportTitle = "Fork Report"
}In your post-build action section in Jenkins, do the following two actions.
The Reporter works with summary files from Fork runs. For them to be accessible, they need to be archived like below:
This requires Jenkins's HTML Publisher Plugin. To be able to link to the right test runs, use a clear title.
Note: The forkJenkins.jenkinsReportTitle parameter of the gradle configuration has to match the Report Title added here.
At the time of writing, not much is available around automated performance testing. Chimprunner is a very simple test runner that allows recording of somewhat accurate timings on test execution, from process creation to test finish. It all works on the Android instrumentation tests system that developers are familiar with.
buildscript {
dependencies {
classpath 'com.shazam.chimprunner:chimprunner-gradle-plugin:3.16.0'
}
}Apply the Chimprunner plugin
apply plugin: 'com.shazam.chimprunner'New tasks will have been added that allow you to run familiar instrumentation tests as performance tests. Verify the tasks exist with:
gradlew tasks | grep chimprunnerConfiguring Chimprunner is simple. Add the following to your build.gradle file:
chimprunner {
serial "0123456"
testPackage "com.example.performancetests"
}The full list of configurable properties (* are mandatory):
| Property Name | Property Type | Default value | Description |
|---|---|---|---|
| baseOutputDir | File | "chimprunner" | the output folder |
| * serial | String | - | the serial of the device where the performance tests will be executed |
| ignoreFailures | boolean | false | whether failures of the performance tests should fail the build |
| testClassRegex | String | "^((?!Abstract).)*Test$" | classes that will be searched for tests |
| * testPackage | String | (Your instrumentation APK package) | the package where the performance tests are located |
Currently, Chimprunner produces a timings.csv file in the output folder with all the timings of tests that were executed as part of the performance tests and the average time they took after running a number of iterations. That CSV file can be then used for plotting by other tools. Using the Jenkins Plot plugin we can now produce historic diagrams of our startup time like the following diagram:
We would like to add ways of automatically launching Android performance tools & reports developers know and use already, with no or minimal code changes. We will investigate around how to provide systrace, CPU, GPU & memory usage reports. The library will probably provide some annotations that will enable various performance tools. An example could be:
@Test
@Systrace
@Gpu
public void trackListScroll() {
//...
}
@Test
@Timings(iterations=10)
public void startUpSpeed() {
//...
}The system could then provide a systrace & GPU profiling reports for the duration of the trackListScroll() test.
Copyright 2019 Apple Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.
You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent