mmcc007 / sylph Goto Github PK

Test_spec.yaml has this:

Is this limited between 1.6.5 and 1.8.1? Or can I set the latest version (v1.15.1)?

Device pools in sylph.yaml must be of the same device type. Add check in validation that they are of the same type (iOS or android).

Since the Device Farm API restricts runs to either android or iOS, Sylph is required to do two runs, one for android and one for iOS.

Currently Sylph calls these runs one after the other. There appears to be no restriction on running these two runs in parallel. This would result in improved overall Sylph run time.

Update the README with more information about sylph features and how to use them.

Is there a solution to restart the test once the devices are available on the Device farm side ?
Some time I get the error:
Error: device: 'name:Apple iPhone 8 Plus, model:A1864, os:12.1.0, deviceType:ios, formFactor:phone, availability:BUSY' is busy.

Hi,
I was wondering if there's plan to implement integration for gcloud cli or firebase test lab ?

If not, how can sylph be extended to support gcloud cli or firebase test lab ?

Builds in a CI environment requires certain values:

• fastlane's Appfile
  requires apple id, team info and app identifier.
• exportOptions.plist
  requires team id, provisioning profile and certificate.

Currently, some of these values are hard-coded.

Add support for configuring these values by adding them to the environment or the sylph.yaml and making them available to fastlane and the exportOptions.plist.

Remove requirement for the following env vars in a CI environment (when running iOS tests):

• SSH_SERVER host can be parsed from PUBLISHING_MATCH_CERTIFICATE_REPO
• SSH_SERVER_PORT can default to 22 (the default port number for ssh)

Setting the env var will override the default.

Currently assumes one device per pool when handling run artifacts.
Collect and download run artifacts for each device in a pool.

Sir. I would like to learn how to use Sylph for local appium server instead of AWS device farm because of cost. We have collected some devices from friends and would love to use Flutter. How can we modify and contribute to your project so instead of using AWS device farm, we use our homemade local appium server device farm?

I am working on to run our flutter based application on device farm. The Smoke Test suite that we are using have both flutter and appium based test cases. When using flutter along with appium in our test suite, the port 4723 is causing a conflict. to enable appium for my test cases in device farm, I have upfront started appium server on default port 4723 before starting the application by modifying test_spec.yaml.

Appium REST http interface listener started on 0.0.0.0:4723

Now when port forward happened through 4723, the application is not responding-

Local port 4723 forwarded to observatory port 42063
......
Connecting to Flutter application at http://127.0.0.1:4723/ajveTCoR24M=/
[warning] FlutterDriver: It is taking an unusually long time to connect to the VM...
[warning] FlutterDriver: It is taking an unusually long time to connect to the VM...
.......
......
Terminated.

Is there a way I can use another port for forwarding and use 4723 for appium server only?

When tests occurs I can see the following logs:
How to get the last stable version (v1.9.1+hotfix.6) ?

[✓] Flutter (Channel stable, v1.9.1+hotfix.2, on Linux, locale en_US.UTF-8)
• Flutter version 1.9.1+hotfix.2 at /home/device-farm/flutter
• Framework revision 2d2a1ffec9 (3 months ago), 2019-09-06 18:39:49 -0700
• Engine revision b863200c37
• Dart version 2.5.0

Currently only checks aws env vars when only on ios pools. Modify to check when testing on any pool when running on CI.

Flutter tests can generate artifacts such as screenshots and logs. Add these artifacts to the job artifacts to be included in download.

I'm have been trying out the tool but are running into an issue that prevents us from going further. I'm not 100% sure that this is related to the tool or this is a configuration issue but I definitely need some help.

So I've been able to install and run the sylph tool. It is also picking up my sylph.yaml configuration file. The connection to our aws account seems to work as well. The exception I'm running into is the following:

Starting Sylph run 'sylph run 2020-03-11 18:38:05.936' on AWS Device Farm ...
Config file: sylph.yaml
Unhandled exception:
FormatException: Missing expected digit (at character 2)
---------------------------------------------------------------------------...
 ^

#0      _ChunkedJsonParser.fail (dart:convert-patch/convert_patch.dart:1394:5)
#1      _ChunkedJsonParser.parseNumber (dart:convert-patch/convert_patch.dart:1258:9)
#2      _ChunkedJsonParser.parse (dart:convert-patch/convert_patch.dart:926:22)
#3      _parseJson (dart:convert-patch/convert_patch.dart:31:10)
#4      JsonDecoder.convert (dart:convert/json.dart:495:36)
#5      JsonCodec.decode (dart:convert/json.dart:153:41)
#6      jsonDecode (dart:convert/json.dart:96:10)
#7      deviceFarmCmd (package:sylph/src/base/utils.dart:150:10)
#8      getDeviceFarmDevices (package:sylph/src/device_farm.dart:290:30)
#9      isValidConfig (package:sylph/src/validator.dart:15:25)
#10     sylphRun (package:sylph/src/sylph_run.dart:44:8)
#11     run (file:///Users/angela/flutter/.pub-cache/hosted/pub.dartlang.org/sylph-0.7.2/bin/main.dart:97:35)
<asynchronous suspension>
#12     main.<anonymous closure> (file:///Users/angela/flutter/.pub-cache/hosted/pub.dartlang.org/sylph-0.7.2/bin/main.dart:62:13)
#13     AppContext.run.<anonymous closure> (package:tool_base/src/base/context.dart:154:29)
#14     _rootRun (dart:async/zone.dart:1126:13)
#15     _CustomZone.run (dart:async/zone.dart:1023:19)
#16     _runZoned (dart:async/zone.dart:1518:10)
#17     runZoned (dart:async/zone.dart:1465:12)
#18     AppContext.run (package:tool_base/src/base/context.dart:153:18)
#19     runInContext (package:sylph/src/context_runner.dart:11:24)
#20     main (file:///Users/angela/flutter/.pub-cache/hosted/pub.dartlang.org/sylph-0.7.2/bin/main.dart:61:11)
#21     _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:299:32)
#22     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)

By the looks of it it is failing when trying to fetch the aws devices. I've actually put a proxy on my Mac to see what the response was to the http call to fetch the aws-devices and the response seems to be pretty normal.

my sylph.yaml file:

# Config file for Flutter tests on real device pools.
# Auto-creates projects and device pools if needed.
# Configures android and ios test runs.
# Builds app, uploads and runs tests.
# Then monitors tests, returns final pass/fail result and downloads artifacts.
# Note: assumes the 'aws' command line utility is logged-in.
# Note: to build the debug iOS app, certain environment variables are required.

# sylph config
tmp_dir: ./tmp/sylph
artifacts_dir: ./tmp/sylph_artifacts
# local timeout per device farm run
#  sylph_timeout: 720 # seconds approx
# run on ios and android pools concurrently (for faster results)
concurrent_runs: true

# device farm config
project_name: Test_project
default_job_timeout: 10 # minutes, set at project creation

device_pools:

  - pool_name: Google Pixel 2
    pool_type: android
    devices:
      - arn: arn:aws:devicefarm:us-west-2::device:58D6FB12B3624256AED26D0F940D4427

  - pool_name: iPhone11
    pool_type: ios
    devices:
      - name: Apple iPhone11
        model: iPhone11
        os: 13.1.3

test_suites:

  - test_suite: example tests 1
    main: test_driver/bdd.dart
    tests:
      - test_driver/bdd_test.dart
    pool_names:
      - Google Pixel 2
      # - iPhone11
    job_timeout: 15 # minutes, set per job, over-rides default job timeout above

the arn code is actually in the response when the tool is calling the devices.

Any help will be greatly appreciated.

Currently requires a forked flutter to by-pass restriction on building a debug .ipa.

May be possible to enable the by-pass by making the change on the fly by running the build in a shell and capturing the ctrl-c before exiting the shell and undoing the change.

Fastlane script is using the default test app name. This should be replaced by the name in the sylph.yaml.

If sylph is run in an app with falvors, the following occurs:

Starting Flutter app com.ianum.app
.dev
.alpha in debug mode...
Running "flutter pub get" in ianum_flutter...                       0.8s
Android Debug Bridge version 1.0.41
Version 29.0.1-5644136
Installed as /Users/goncalopalma/Library/Android/sdk/platform-tools/adb
/system/bin/sh: .dev: not found
/system/bin/sh: .alpha: not found

As you can see from the multiple lines it went to fetch not only applicationId but also applicationIdSuffix

Additionally, the project assumes in
adb shell am start -a android.intent.action.RUN -f 0x20000000 --ez enable-background-compilation true --ez enable-dart-profiling true --ez enable-checked-mode true --ez verify-entry-points true --ez start-paused true "$app_id/$app_id.MainActivity"

That all applicatuons will:

• have the package name the same as the app id
• Have the same app id in different flavors

This will throw an error since adb will not be able to find that activity

Logging is useful for debugging and support and for machine-readable applications (output as json).

Should support output to stdio and files as well as different levels of logging.

I would like to see the snaphot files I'm taking from the test using the method:
where path is the filename without any directory before.

Is there a specific path to specify to see the snapthot files in the AWS console ?

Future<void> _takeScreenshot(FlutterDriver driver, String path) async { final List<int> pixels = await driver.screenshot(); final File file = new File(path); await file.writeAsBytes(pixels); print(path); }

So Im trying to run sylph for integration test for AWS device farm. I have followed the instructions for installation and configuration but I am unable to get it to work. Some clarification would be needed as to what Im doing wrong. Below is the log

C:\Users\penedeh\Desktop\Digital Bank Mobile>sylph -c sylph.yaml -v
[ +30 ms] Starting Sylph run 'sylph run 2020-03-09 07:19:38.482' on AWS Device Farm ...
[ +10 ms] Config file: sylph.yaml
[+6230 ms] executing: [.] aws devicefarm list-devices
[+2566 ms] executing: [.] aws devicefarm list-projects
[ +2 ms] Creating new project for 'App Integration Tests' ...
[+2709 ms] executing: [.] aws devicefarm create-project --name App Integration Tests --default-job-timeout-minutes 10
[ +5 ms] Unpacking sylph resources to /tmp/sylph
[+4094 ms] Creating test bundle for upload...
Unhandled exception:
Invalid argument(s): Cannot find executable for 7z.
#0 _getExecutable (package:process/src/interface/local_process_manager.dart:127:5)
#1 LocalProcessManager.runSync (package:process/src/interface/local_process_manager.dart:94:30)
#2 cmd (package:sylph/src/base/utils.dart:106:33)
#3 zip (package:sylph/src/base/utils.dart:82:5)
#4 bundleFlutterTests (package:sylph/src/bundle.dart:48:3)
#5 sylphRun (package:sylph/src/sylph_run.dart:59:3)

#6 run (file:///C:/Users/penedeh/AppData/Roaming/Pub/Cache/hosted/pub.dartlang.org/sylph-0.7.2/bin/main.dart:97:35)

#7 main. (file:///C:/Users/penedeh/AppData/Roaming/Pub/Cache/hosted/pub.dartlang.org/sylph-0.7.2/bin/main.dart:56:13)
#8 AppContext.run. (package:tool_base/src/base/context.dart:154:29)
#9 _rootRun (dart:async/zone.dart:1126:13)
#10 _CustomZone.run (dart:async/zone.dart:1023:19)
#11 _runZoned (dart:async/zone.dart:1518:10)
#12 runZoned (dart:async/zone.dart:1465:12)
#13 AppContext.run (package:tool_base/src/base/context.dart:153:18)
#14 runInContext (package:sylph/src/context_runner.dart:11:24)
#15 main (file:///C:/Users/penedeh/AppData/Roaming/Pub/Cache/hosted/pub.dartlang.org/sylph-0.7.2/bin/main.dart:55:11)
#16 _startIsolate. (dart:isolate-patch/isolate_patch.dart:305:32)
#17 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:174:12)
[ +38 ms] Starting Sylph run 'sylph run 2020-03-09 07:20:32.603' on AWS Device Farm ...
[ +13 ms] Config file: sylph.yaml
[+6344 ms] executing: [.] aws devicefarm list-devices
[+6459 ms] executing: [.] aws devicefarm list-projects
[ +2 ms] Unpacking sylph resources to /tmp/sylph
[+3213 ms] Creating test bundle for upload...
Unhandled exception:
Invalid argument(s): Cannot find executable for 7z.
#0 _getExecutable (package:process/src/interface/local_process_manager.dart:127:5)
#1 LocalProcessManager.runSync (package:process/src/interface/local_process_manager.dart:94:30)
#2 cmd (package:sylph/src/base/utils.dart:106:33)
#3 zip (package:sylph/src/base/utils.dart:82:5)
#4 bundleFlutterTests (package:sylph/src/bundle.dart:48:3)
#5 sylphRun (package:sylph/src/sylph_run.dart:59:3)

#6 run (file:///C:/Users/penedeh/AppData/Roaming/Pub/Cache/hosted/pub.dartlang.org/sylph-0.7.2/bin/main.dart:97:35)

#7 main. (file:///C:/Users/penedeh/AppData/Roaming/Pub/Cache/hosted/pub.dartlang.org/sylph-0.7.2/bin/main.dart:56:13)
#8 AppContext.run. (package:tool_base/src/base/context.dart:154:29)
#9 _rootRun (dart:async/zone.dart:1126:13)
#10 _CustomZone.run (dart:async/zone.dart:1023:19)
#11 _runZoned (dart:async/zone.dart:1518:10)
#12 runZoned (dart:async/zone.dart:1465:12)
#13 AppContext.run (package:tool_base/src/base/context.dart:153:18)
#14 runInContext (package:sylph/src/context_runner.dart:11:24)
#15 main (file:///C:/Users/penedeh/AppData/Roaming/Pub/Cache/hosted/pub.dartlang.org/sylph-0.7.2/bin/main.dart:55:11)
#16 _startIsolate. (dart:isolate-patch/isolate_patch.dart:305:32)
#17 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:174:12)

The test_spec.yaml can be added as a resource as it no longer needs to be exposed to the developer. It can then be modified during bundling to run the test suite.

The Android SDK can be uploaded with the bundle to reduce installation (and metered) time

Received notification from GitHub. For details see:
https://nvd.nist.gov/vuln/detail/CVE-2019-13574

For developers who are not on a mac, provide support for running on android-only device pools. These developers can still test on iOS device pools from a CI environment.

I exactly followed all the steps mentioned and everything seemed to work seamlessly until I hit this roadblock.

Looks like it couldn't start the activity on Android.. I'm running with flavor --stg

got myself to this stacktrace on 0.4.0

Running tests concurrently on iOS and Android pools...
Running test suite 'catalogue tests 1'  in project 'Catalogue App' on pool 'android pool 1'...
Creating new device pool 'android pool 1' ...
Unhandled exception:
type 'List<dynamic>' is not a subtype of type 'List<SylphDevice>'
#0      setupDevicePool (package:sylph/src/device_farm.dart:58:34)
#1      runSylphJob (package:sylph/src/sylph_run.dart:99:26)
<asynchronous suspension>
#2      runSylphJobInIsolate (package:sylph/src/concurrent_jobs.dart:39:27)
<asynchronous suspension>
#3      IsolateRunnerRemote._handleCommand.<anonymous closure> (package:isolate/isolate_runner.dart:298:43)
#4      new Future.sync (dart:async/future.dart:224:31)
#5      IsolateRunnerRemote._handleCommand (package:isolate/isolate_runner.dart:298:17)
#6      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:172:12)

The --config arg is being ignored

When building the iOS app in a CI environment, the CI ssh client is used to connect to the match repo. The ssh client must be configured correctly. If the ssh client fails, sylph should generate an error message before exiting.

Currently all tests run in debug mode by default with no option to change it. My main goal using this library and AWS Device farm is collecting profiling reports.

Environment variables are used to build a testable iOS app. However, if only testing on android, these env vars are still required. iOS-specific env vars should not be required if testing on android devices only.

Flutter can be uploaded with the bundle to reduce installation (and metered) time.

My test were all failing on AWS Device farm and it takes hours before the whole process completes. On my local its dont in minutes. Please find below the logs and AWS logs

[+4201761 ms] results=[{result: false}]
[ +4 ms] Concurrent runs completed.
[ +13 ms] Sylph run completed in 1h:11m:19s:566ms.
[ +2 ms] Sylph run 'sylph run 2020-03-09 13:39:49.676' failed.

[DEVICEFARM] Setting up your device. This usually takes 2-3 minutes.
[DEVICEFARM] ########### Start executing testspec ###########

[DEVICEFARM] ########### Entering phase install ###########

[DeviceFarm] cd flutter_app
[DeviceFarm] ls -la
total 128
drwxrwxrwx 16 rwx rwx 4096 Mar 9 13:40 .
drwxrwxrwx 9 rwx rwx 4096 Mar 9 13:40 ..
drwxrwxrwx 5 rwx rwx 4096 Mar 9 13:40 android
drwxrwxrwx 3 rwx rwx 4096 Mar 9 13:40 assets
-rwxrwxrwx 1 rwx rwx 51 Mar 9 13:40 build_to_os.txt
drwxrwxrwx 5 rwx rwx 4096 Mar 9 13:40 country_code_picker
drwxrwxrwx 3 rwx rwx 4096 Mar 9 13:40 .dart_tool
-rwxrwxrwx 1 rwx rwx 502 Mar 9 13:40 .flutter-plugins
-rwxrwxrwx 1 rwx rwx 305 Mar 9 13:40 .flutter-plugins-dependencies
drwxrwxrwx 7 rwx rwx 4096 Mar 9 13:40 .git
-rwxrwxrwx 1 rwx rwx 1613 Mar 9 13:40 .gitignore
drwxrwxrwx 4 rwx rwx 4096 Mar 9 13:40 .idea
drwxrwxrwx 3 rwx rwx 4096 Mar 9 13:40 images
drwxrwxrwx 6 rwx rwx 4096 Mar 9 13:40 ios
drwxrwxrwx 9 rwx rwx 4096 Mar 9 13:40 lib
-rwxrwxrwx 1 rwx rwx 315 Mar 9 13:40 .metadata
-rwxrwxrwx 1 rwx rwx 9542 Mar 9 13:40 .packages
-rwxrwxrwx 1 rwx rwx 16717 Mar 9 13:40 pubspec.lock
-rwxrwxrwx 1 rwx rwx 1414 Mar 9 13:40 pubspec.yaml
-rwxrwxrwx 1 rwx rwx 571 Mar 9 13:40 README.md
drwxrwxrwx 2 rwx rwx 4096 Mar 9 13:40 screenshots
drwxrwxrwx 2 rwx rwx 4096 Mar 9 13:40 script
-rwxrwxrwx 1 rwx rwx 1182 Mar 9 13:40 sylph.yaml
drwxrwxrwx 3 rwx rwx 4096 Mar 9 13:40 test
drwxrwxrwx 3 rwx rwx 4096 Mar 9 13:40 test_driver
drwxrwxrwx 3 rwx rwx 4096 Mar 9 13:40 thirdparty
[DeviceFarm] bash --version
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
[DeviceFarm] echo "DEVICEFARM_APP_PATH=$DEVICEFARM_APP_PATH"
DEVICEFARM_APP_PATH=/tmp/scratchAZfnSI.scratch/share-R0D9nC.scratch/app-Qwt8gN.apk
[DeviceFarm] echo "Install flutter"
Install flutter
[DeviceFarm] FLUTTER_VERSION=v1.12.13+hotfix.8-stable
[DeviceFarm] if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "Android" ]; then
# Install Flutter
curl https://storage.googleapis.com/flutter_infra/releases/stable/linux/flutter_linux_$FLUTTER_VERSION.tar.xz -o flutter_linux.tar.xz
tar xf flutter_linux.tar.xz
fi
if [ $DEVICEFARM_DEVICE_PLATFORM_NAME = "iOS" ]; then
# Install Flutter
curl https://storage.googleapis.com/flutter_infra/releases/stable/macos/flutter_macos_$FLUTTER_VERSION.zip -o flutter_macos.zip
unzip -qq flutter_macos.zip
fi
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 485M 0 412 0 0 663 0 8d 21h --:--:-- 8d 21h 663
1 485M 1 5216k 0 0 3334k 0 0:02:29 0:00:01 0:02:28 3332k
9 485M 9 44.7M 0 0 17.4M 0 0:00:27 0:00:02 0:00:25 17.4M
15 485M 15 74.3M 0 0 20.8M 0 0:00:23 0:00:03 0:00:20 20.8M
21 485M 21 105M 0 0 23.0M 0 0:00:21 0:00:04 0:00:17 23.0M
28 485M 28 136M 0 0 22.8M 0 0:00:21 0:00:05 0:00:16 25.4M
29 485M 29 144M 0 0 20.3M 0 0:00:23 0:00:07 0:00:16 25.1M
31 485M 31 153M 0 0 20.2M 0 0:00:23 0:00:07 0:00:16 21.7M
34 485M 34 168M 0 0 19.2M 0 0:00:25 0:00:08 0:00:17 18.2M
37 485M 37 184M 0 0 19.0M 0 0:00:25 0:00:09 0:00:16 15.4M
39 485M 39 194M 0 0 18.3M 0 0:00:26 0:00:10 0:00:16 12.5M
46 485M 46 224M 0 0 18.4M 0 0:00:26 0:00:12 0:00:14 15.7M
47 485M 47 230M 0 0 18.3M 0 0:00:26 0:00:12 0:00:14 15.3M
53 485M 53 258M 0 0 19.0M 0 0:00:25 0:00:13 0:00:12 18.7M
57 485M 57 280M 0 0 18.3M 0 0:00:26 0:00:15 0:00:11 17.0M
60 485M 60 293M 0 0 18.8M 0 0:00:25 0:00:15 0:00:10 19.9M
62 485M 62 302M 0 0 18.2M 0 0:00:26 0:00:16 0:00:10 17.8M
69 485M 69 336M 0 0 19.1M 0 0:00:25 0:00:17 0:00:08 21.0M
72 485M 72 352M 0 0 18.8M 0 0:00:25 0:00:18 0:00:07 18.2M
78 485M 78 382M 0 0 19.5M 0 0:00:24 0:00:19 0:00:05 23.7M
82 485M 82 400M 0 0 19.4M 0 0:00:24 0:00:20 0:00:04 21.3M
87 485M 87 424M 0 0 19.6M 0 0:00:24 0:00:21 0:00:03 24.3M
94 485M 94 459M 0 0 20.3M 0 0:00:23 0:00:22 0:00:01 24.8M
100 485M 100 485M 0 0 20.7M 0 0:00:23 0:00:23 --:--:-- 28.4M
[DeviceFarm] mv flutter $HOME/flutter
[DeviceFarm] export PATH=$PATH:$HOME/flutter/bin:$HOME/flutter/bin/cache/dart-sdk/bin
[DeviceFarm] export PATH="$PATH":"$HOME/.pub-cache/bin"
[DeviceFarm] flutter doctor -v
Warning! This package referenced a Flutter repository via the .packages file that is no longer available. The repository from which the 'flutter' tool is currently executing will be used instead.
running Flutter tool: /home/device-farm/flutter
previous reference : /C:/src/flutter
This can happen if you deleted or moved your copy of the Flutter repository, or if it was on a volume that is no longer mounted or has been mounted at a different location. Please check your system path to verify that you are running the expected version (run 'flutter --version' to see which flutter is on your path).

╔════════════════════════════════════════════════════════════════════════════╗
║ Welcome to Flutter! - https://flutter.dev ║
║ ║
║ The Flutter tool uses Google Analytics to anonymously report feature usage ║
║ statistics and basic crash reports. This data is used to help improve ║
║ Flutter tools over time. ║
║ ║
║ Flutter tool analytics are not sent on the very first run. To disable ║
║ reporting, type 'flutter config --no-analytics'. To display the current ║
║ setting, type 'flutter config'. If you opt out of analytics, an opt-out ║
║ event will be sent, and then no further information will be sent by the ║
║ Flutter tool. ║
║ ║
║ By downloading the Flutter SDK, you agree to the Google Terms of Service. ║
║ Note: The Google Privacy Policy describes how data is handled in this ║
║ service. ║
║ ║
║ Moreover, Flutter includes the Dart SDK, which may send usage metrics and ║
║ crash reports to Google. ║
║ ║
║ Read about data we send with crash reports: ║
║ https://github.com/flutter/flutter/wiki/Flutter-CLI-crash-reporting ║
║ ║
║ See Google's privacy policy: ║
║ https://www.google.com/intl/en/policies/privacy/ ║
╚════════════════════════════════════════════════════════════════════════════╝

[✓] Flutter (Channel stable, v1.12.13+hotfix.8, on Linux, locale en_US.UTF-8)
• Flutter version 1.12.13+hotfix.8 at /home/device-farm/flutter
• Framework revision 0b8abb4724 (4 weeks ago), 2020-02-11 11:44:36 -0800
• Engine revision e1e6ced81d
• Dart version 2.7.0

[!] Android toolchain - develop for Android devices (Android SDK version 21.1.1)
• Android SDK at /opt/dev/android-sdk-linux
• Android NDK location not configured (optional; useful for native profiling support)
✗ Flutter requires Android SDK 28 and the Android BuildTools 28.0.3
To update using sdkmanager, run:
"/opt/dev/android-sdk-linux/tools/bin/sdkmanager" "platforms;android-28" "build-tools;28.0.3"
or visit https://flutter.dev/setup/#android-setup for detailed instructions.
✗ Android license status unknown.
Try re-installing or updating your Android SDK Manager.
See https://developer.android.com/studio/#downloads or visit https://flutter.dev/setup/#android-setup for detailed instructions.

[!] Android Studio (not installed)
• Android Studio not found; download from https://developer.android.com/studio/index.html
(or visit https://flutter.dev/setup/#android-setup for detailed instructions).

[✓] Connected device (1 available)
• SM G975U1 • R38M60AN7YR • android-arm64 • Android 9 (API 28)

! Doctor found issues in 2 categories.

[DEVICEFARM] ########### Entering phase pre_test ###########

[DEVICEFARM] ########### Entering phase test ###########

[DeviceFarm] TEST_PHASE_STATUS=0
[DeviceFarm] track_test_phase_status() {
local command_exit_code=$?
if [ "$command_exit_code" != "0" ]; then
echo "Most recent command failed with error code: $command_exit_code"
TEST_PHASE_STATUS=$command_exit_code
fi
}
set_test_phase_exit_code() {
return $TEST_PHASE_STATUS
}
[DeviceFarm] echo "Navigate to test package directory"
Navigate to test package directory
[DeviceFarm] cd $DEVICEFARM_TEST_PACKAGE_PATH
[DeviceFarm] echo "Start Flutter integration test"
Start Flutter integration test
[DeviceFarm] cd flutter_app
[DeviceFarm] rm -f .packages
[DeviceFarm] MAIN=C:/Users/penedeh/Desktop/Digital Bank Mobile/test_driver/onboarding/OpenAccountWithoutBvn.dart
/tmp/scratchAZfnSI.scratch/shell-script-DrClsH/shell_script.sh: 132: /tmp/scratchAZfnSI.scratch/shell-script-DrClsH/shell_script.sh: Bank: not found

[DEVICEFARM] ########### Entering phase post_test ###########

[DEVICEFARM] ########### Finish executing testspec ###########

[DEVICEFARM] ########### Setting upload permissions ###########

chmod: cannot access ‘/tmp/customer_log_directory0vut2Y/*’: No such file or directory

[DEVICEFARM] Tearing down your device. Your tests report will come shortly.

Modify aws calls to use json in response

This command doesn't work for me as well : sylph -d android | grep Pixel

sylph -d android | grep Pixel
Unhandled exception:
FormatException: Missing expected digit (at character 2)
---------------------------------------------------------------------------...
 ^

#0      _ChunkedJsonParser.fail (dart:convert-patch/convert_patch.dart:1394:5)
#1      _ChunkedJsonParser.parseNumber (dart:convert-patch/convert_patch.dart:1258:9)
#2      _ChunkedJsonParser.parse (dart:convert-patch/convert_patch.dart:926:22)
#3      _parseJson (dart:convert-patch/convert_patch.dart:31:10)
#4      JsonDecoder.convert (dart:convert/json.dart:495:36)
#5      JsonCodec.decode (dart:convert/json.dart:153:41)
#6      jsonDecode (dart:convert/json.dart:96:10)
#7      deviceFarmCmd (package:sylph/src/base/utils.dart:150:10)
#8      getDeviceFarmDevices (package:sylph/src/device_farm.dart:290:30)
#9      getDeviceFarmDevicesByType (package:sylph/src/device_farm.dart:283:10)
#10     run (file:///Users/angela/flutter/.pub-cache/hosted/pub.dartlang.org/sylph-0.7.2/bin/main.dart:76:32)
#11     main.<anonymous closure> (file:///Users/angela/flutter/.pub-cache/hosted/pub.dartlang.org/sylph-0.7.2/bin/main.dart:62:13)
#12     AppContext.run.<anonymous closure> (package:tool_base/src/base/context.dart:154:29)
#13     _rootRun (dart:async/zone.dart:1126:13)
#14     _CustomZone.run (dart:async/zone.dart:1023:19)
#15     _runZoned (dart:async/zone.dart:1518:10)
#16     runZoned (dart:async/zone.dart:1465:12)
#17     AppContext.run (package:tool_base/src/base/context.dart:153:18)
#18     runInContext (package:sylph/src/context_runner.dart:11:24)
#19     main (file:///Users/angela/flutter/.pub-cache/hosted/pub.dartlang.org/sylph-0.7.2/bin/main.dart:61:11)
#20     _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:299:32)
#21     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)

Originally posted by @AnzhelaAninian in #95 (comment)

Obviously I have flutter in my path 🤷‍♀️

Starting Sylph run 'sylph run 2019-08-27 21:35:56.363' on AWS Device Farm ...
Config file: sylph.yaml
Unpacking sylph resources to /tmp/sylph
Creating test bundle for upload...
Unhandled exception:
ProcessException: No such file or directory
  Command: flutter packages get
#0      _ProcessImpl._runAndWait (dart:io-patch/process_patch.dart:489:7)
#1      _runNonInteractiveProcessSync (dart:io-patch/process_patch.dart:634:18)
#2      Process.runSync (dart:io-patch/process_patch.dart:66:12)
#3      cmd (package:sylph/src/utils.dart:39:26)
#4      bundleFlutterTests (package:sylph/src/bundle.dart:52:3)
<asynchronous suspension>
#5      sylphRun (package:sylph/src/sylph_run.dart:48:9)
<asynchronous suspension>
#6      main (file:///Users/vishna/.pub-cache/hosted/pub.dartlang.org/sylph-0.4.0/bin/main.dart:80:13)
<asynchronous suspension>
#7      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:299:32)
#8      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:172:12)

When running multiple tests on an android device, only the first test runs. Any subsequent tests will time out and the sylph run will report a failure.

Fixing this involves finding an available local port to forward to the remote port.

I have been setup everything on my Bitrise CI environment as recommended by the documentation, but my run gets stuck on the Signing Step and eventually times out after 30 minutes. Here are the logs:

+ sylph
Starting Sylph run 'sylph run 2020-04-10 17:11:57.070' on AWS Device Farm ...
Config file: sylph.yaml
Unpacking sylph resources to /tmp/sylph
iOS build in CI environment detected. Unpacking related resources.
Creating test bundle for upload...
Test bundle created (size 50.3MB)
Running 'main suite' test suite...
Running tests concurrently on iOS pools...
Running test suite 'main suite'  in project 'myBrightlink' on pool 'ios pool 1'  with flavor dev...
Building debug .ipa from test_driver/main.dart with flavor dev...
# [REDACTED]:[REDACTED] SSH-2.0-babeld-ad297116
Successfully installed bundler-2.0.1
Parsing documentation for bundler-2.0.1
Installing ri documentation for bundler-2.0.1
Done installing documentation for bundler after 3 seconds
1 gem installed
Fetching gem metadata from https://rubygems.org/........
Using CFPropertyList 3.0.2
Using public_suffix 2.0.5
Using addressable 2.7.0
Using atomos 0.1.3
Using babosa 1.0.3
Using bundler 2.0.1
Using claide 1.0.3
Using colored 1.2
Using colored2 3.1.2
Using highline 1.7.10
Using commander-fastlane 4.4.6
Using declarative 0.0.10
Using declarative-option 0.1.0
Fetching digest-crc 0.4.1
Installing digest-crc 0.4.1
Fetching unf_ext 0.0.7.6
Installing unf_ext 0.0.7.6 with native extensions
Using unf 0.1.4
Using domain_name 0.5.20190701
Using dotenv 2.7.5
Using emoji_regex 1.0.1
Fetching excon 0.72.0
Installing excon 0.72.0
Using multipart-post 2.0.0
Using faraday 0.17.3
Using http-cookie 1.0.3
Using faraday-cookie_jar 0.0.6
Using faraday_middleware 0.13.1
Using fastimage 2.1.7
Using gh_inspector 1.1.3
Using jwt 2.1.0
Using memoist 0.16.2
Using multi_json 1.14.1
Fetching os 1.0.1
Installing os 1.0.1
Fetching signet 0.13.0
Installing signet 0.13.0
Using googleauth 0.11.0
Using httpclient 2.8.3
Using mini_mime 1.0.2
Using uber 0.1.0
Using representable 3.0.4
Using retriable 3.1.2
Using google-api-client 0.36.4
Fetching google-cloud-env 1.3.0
Installing google-cloud-env 1.3.0
Using google-cloud-errors 1.0.0
Using google-cloud-core 1.5.0
Using google-cloud-storage 1.25.1
Fetching json 2.3.0
Installing json 2.3.0 with native extensions
Using mini_magick 4.10.1
Using multi_xml 0.6.0
Using plist 3.5.0
Using rubyzip 1.3.0
Using security 0.1.3
Using naturally 2.2.0
Using simctl 1.6.8
Using slack-notifier 2.3.2
Using terminal-notifier 2.0.0
Fetching unicode-display_width 1.6.1
Installing unicode-display_width 1.6.1
Using terminal-table 1.8.0
Using tty-screen 0.7.1
Using tty-cursor 0.7.1
Using tty-spinner 0.9.3
Using word_wrap 1.0.0
Using nanaimo 0.2.6
Using xcodeproj 1.15.0
Using rouge 2.0.7
Using xcpretty 0.3.0
Using xcpretty-travis-formatter 1.0.0
Fetching fastlane 2.142.0
Installing fastlane 2.142.0
Bundle complete! 1 Gemfile dependency, 65 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
[17:12:35]: fastlane detected a Gemfile in the current directory
[17:12:35]: However, it seems like you didn't use `bundle exec`
[17:12:35]: To launch fastlane faster, please use
[17:12:35]: 
[17:12:35]: $ bundle exec fastlane enable_match_code_signing mode:debug
[17:12:35]: 
[17:12:35]: Get started using a Gemfile for fastlane https://docs.fastlane.tools/getting-started/ios/setup/#use-a-gemfile
[17:12:41]: Sending anonymous analytics information
[17:12:41]: Learn more at https://docs.fastlane.tools/#metrics
[17:12:41]: No personal or sensitive data is sent.
[17:12:41]: You can disable this by adding `opt_out_usage` at the top of your Fastfile
[17:12:41]: ---------------------------
[17:12:41]: --- Step: opt_out_usage ---
[17:12:41]: ---------------------------
[17:12:41]: Disabled upload of used actions
[17:12:41]: ------------------------------
[17:12:41]: --- Step: default_platform ---
[17:12:41]: ------------------------------
[17:12:41]: --------------------------
[17:12:41]: --- Step: setup_travis ---
[17:12:41]: --------------------------
[17:12:41]: Creating temporary keychain: "fastlane_tmp_keychain".
[17:12:42]: $ security list-keychains -d user
[17:12:42]: ▸ "/Users/vagrant/Library/Keychains/login.keychain-db"
[17:12:42]: ▸ "/Users/vagrant/Library/Keychains/login.keychain-db"
[17:12:42]: Enabling match readonly mode.
[17:12:42]: Driving the lane 'ios enable_match_code_signing' 🚀
[17:12:42]: signing_type: development
[17:12:42]: build_configuration: Debug
[17:12:42]: profile_name: sigh_org.[REDACTED].mybrightlink.dev_development_profile-path
[17:12:42]: app_identifier: org.[REDACTED].mybrightlink.dev
[17:12:42]: Retrieving signing certificates and profiles...

Currently the sylph run will stop if a run on a device pool fails and artifacts will not be downloaded. Modify to continue and report the artifacts as well as continuing to next run on the next device pool.

Finally, at the of sylph run, report a summary of devices tested and the results.

Flutter allows Git repos to be referenced via flutter pub get (https://flutter.dev/docs/development/packages-and-plugins/using-packages) however, when sylph running on AWS tries to flutter pub get it fails due to a missing ssh key. AWS doesn't have a place to put the SSH key (only for CodeCommit) and CodeMagic doesn't pass the ssh key when it pushes to AWS.

Logs:

Starting Flutter app co.glean.app in debug mode...
Running "flutter pub get" in flutter_app...                     
Git error. Command: git clone --mirror [email protected]:{repo}

Cloning into bare repository '/home/device-farm/flutter/.pub-cache/git/cache/glean-audio-library-3a19cf8eeb19f3e3abf0bd839accc81263b56c07'...

ssh_askpass: exec(/usr/local/libexec/ssh-askpass): No such file or directory

Host key verification failed.

fatal: Could not read from remote repository.

Please make sure you have the correct access rights

and the repository exists.

pub get failed (server unavailable) -- attempting retry 1 in 1 second...

Is this something that could be fixed within sylph?

Running the example in this repo with:

sylph -c sylph_ios.yaml -v

output:

[+11760 ms] executing: [./] aws devicefarm get-run --arn arn:aws:devicefarm:us-west-2:917623879487:run:030633b3-e130-4058-b087-30d92066a93f/b0e28b53-d029-4e59-bd90-265f70df28da
[        ] Run status on device pool 'android pool 1': RUNNING (sylph run timeout: 330 of 720)
[+1791 ms] executing: [./] aws devicefarm list-jobs --arn arn:aws:devicefarm:us-west-2:917623879487:run:030633b3-e130-4058-b087-30d92066a93f/b0e28b53-d029-4e59-bd90-265f70df28da
[        ]              device: Google Pixel 2 , passed: 0, failed: 0, minutes: 0.0

Add a check for named pools in sylph.yaml to re-create the pool if the devices in the pool have changed.

On a macOS with an attached iOS device on Device Farms (via web interface) with an uploaded debug .app.

ios-deploy is not validating:

ios-deploy --bundle build/ios/Debug-iphoneos/Runner.app --no-wifi --justlaunch --args '--enable-dart-profiling --start-paused --enable-checked-mode --verify-entry-points'
[....] Waiting for iOS device to be connected
[....] Using bb66dc1943570628375e82055b7770b33883dc73 (D22AP, D22AP, uknownos, unkarch) a.k.a. '9905944754'.
------ Install phase ------
[  0%] Found bb66dc1943570628375e82055b7770b33883dc73 (D22AP, D22AP, uknownos, unkarch) a.k.a. '9905944754' connected through USB, beginning install
[  5%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/META-INF/ to device
[  5%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/META-INF/com.apple.ZipMetadata.plist to device
[  5%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/_CodeSignature/ to device
[  6%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/_CodeSignature/CodeResources to device
[  6%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppFrameworkInfo.plist to device
[  6%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/[email protected] to device
[  7%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon20x20@2x~ipad.png to device
[  7%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/[email protected] to device
[  7%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon20x20~ipad.png to device
[  8%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon29x29.png to device
[  8%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/[email protected] to device
[  8%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon29x29@2x~ipad.png to device
[  9%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/[email protected] to device
[  9%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon29x29~ipad.png to device
[ 10%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/[email protected] to device
[ 10%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon40x40@2x~ipad.png to device
[ 10%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/[email protected] to device
[ 11%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon40x40~ipad.png to device
[ 11%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/[email protected] to device
[ 11%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/[email protected] to device
[ 12%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon76x76@2x~ipad.png to device
[ 12%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon76x76~ipad.png to device
[ 12%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/AppIcon83.5x83.5@2x~ipad.png to device
[ 13%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Assets.car to device
[ 13%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/ to device
[ 13%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/LaunchScreen.storyboardc/ to device
[ 14%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/LaunchScreen.storyboardc/01J-lp-oVM-view-Ze5-6b-2t3.nib to device
[ 14%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/LaunchScreen.storyboardc/Info.plist to device
[ 15%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/LaunchScreen.storyboardc/UIViewController-01J-lp-oVM.nib to device
[ 15%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/Main.storyboardc/ to device
[ 15%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/Main.storyboardc/BYZ-38-t0r-view-8bC-Xf-vdC.nib to device
[ 16%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/Main.storyboardc/Info.plist to device
[ 16%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Base.lproj/Main.storyboardc/UIViewController-BYZ-38-t0r.nib to device
[ 16%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Debug.xcconfig to device
[ 17%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/embedded.mobileprovision to device
[ 17%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/ to device
[ 17%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/ to device
[ 18%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/_CodeSignature/ to device
[ 18%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/_CodeSignature/CodeResources to device
[ 18%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/App to device
[ 19%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/ to device
[ 19%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/AssetManifest.json to device
[ 20%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/FontManifest.json to device
[ 20%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/fonts/ to device
[ 20%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/fonts/MaterialIcons-Regular.ttf to device
[ 21%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/isolate_snapshot_data to device
[ 22%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/kernel_blob.bin to device
[ 27%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/LICENSE to device
[ 28%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/packages/ to device
[ 28%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/packages/cupertino_icons/ to device
[ 29%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/packages/cupertino_icons/assets/ to device
[ 29%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/packages/cupertino_icons/assets/CupertinoIcons.ttf to device
[ 29%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/flutter_assets/vm_snapshot_data to device
[ 30%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/App.framework/Info.plist to device
[ 30%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/Flutter.framework/ to device
[ 30%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/Flutter.framework/_CodeSignature/ to device
[ 31%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/Flutter.framework/_CodeSignature/CodeResources to device
[ 31%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/Flutter.framework/Flutter to device
[ 47%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/Flutter.framework/icudtl.dat to device
[ 48%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Frameworks/Flutter.framework/Info.plist to device
[ 48%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Info.plist to device
[ 49%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/PkgInfo to device
[ 49%] Copying /private/tmp/scratchrBjb_g.scratch/test-packageX4pGG_/flutter_app/build/ios/Debug-iphoneos/Runner.app/Runner to device
[ 52%] CreatingStagingDirectory
[ 57%] ExtractingPackage
[ 60%] InspectingPackage
[ 60%] TakingInstallLock
[ 65%] PreflightingApplication
[ 65%] InstallingEmbeddedProfile
[ 70%] VerifyingApplication
2019-06-02 12:49:43.347 ios-deploy[544:5128] [ !! ] Error 0xe8008015: A valid provisioning profile for this executable was not found. AMDeviceSecureInstallApplication(0, device, url, options, install_callback, 0)

Currently experimenting with installing provisioning profile and developer certificate, etc...

@tvolkert @aristeia

When a test fails, the error code is currently being ignored. Add handling for error code.

Use tmp dir returned from os instead of specifying in config file.

@mmcc007 thanks you for your help, it was configured to return table. Have change it to json, but now i'm running into another issue. Maybe you can give me some advice?

The exception I'm running into is the following:

Starting Sylph run 'sylph run 2020-03-18 20:34:05.781' on AWS Device Farm ... Config file: sylph.yaml Unpacking sylph resources to ./tmp/sylph Creating test bundle for upload... Unhandled exception: Invalid argument(s): Cannot copy from . to ./tmp/sylph/test_bundle/flutter_app #0 _doNothing (package:sylph/src/base/copy_path.dart:15:5) #1 copyPathSync (package:sylph/src/base/copy_path.dart:56:7) #2 LocalPackageManager.copy (package:sylph/src/base/local_packages.dart:104:7) #3 bundleFlutterTests (package:sylph/src/bundle.dart:31:23) #4 sylphRun (package:sylph/src/sylph_run.dart:59:3) <asynchronous suspension> #5 run (file:///Users/angela/flutter/.pub-cache/hosted/pub.dartlang.org/sylph-0.7.2/bin/main.dart:97:35) <asynchronous suspension> #6 main.<anonymous closure> (file:///Users/angela/flutter/.pub-cache/hosted/pub.dartlang.org/sylph-0.7.2/bin/main.dart:62:13) #7 AppContext.run.<anonymous closure> (package:tool_base/src/base/context.dart:154:29) #8 _rootRun (dart:async/zone.dart:1126:13) #9 _CustomZone.run (dart:async/zone.dart:1023:19) #10 _runZoned (dart:async/zone.dart:1518:10) #11 runZoned (dart:async/zone.dart:1465:12) #12 AppContext.run (package:tool_base/src/base/context.dart:153:18) #13 runInContext (package:sylph/src/context_runner.dart:11:24) #14 main (file:///Users/angela/flutter/.pub-cache/hosted/pub.dartlang.org/sylph-0.7.2/bin/main.dart:61:11) #15 _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:299:32) #16 _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:168:12)

Originally posted by @AnzhelaAninian in #95 (comment)

Building a new device pool is difficult. It requires familiarity with the device farm CLI to get the model name.

Should also be possible to get all the device names for each device type (iOS and android) from the sylph CLI and use to build a new device pool.

The suite of tests defined in sylph.yaml can be better integrated into the Device Farm API. Each test can start with a fresh install of the app. Then the results of the test can be reported to the device farm API and subsequently by sylph.

The test_spec.yaml can be added as a resource as it no longer needs to be exposed to the developer. It can then be modified during bundling to run the test suite.

If in our pubspec.yaml file we have a dependency such as:

  tutorial_coach_mark:
    path: /path/to/tutorial_coach_mark

The following error is thrown:

cp: ./ios/.symlinks/plugins/notification_permissions/example/ios/.symlinks/plugins/notification_permissions: directory causes a cycle
cp: ./path/to/tutorial_coach_mark: No such file or directory
Unhandled exception:
command failed: cmd='cp -r . /tmp/sylph/test_bundle/flutter_app'
#0      cmd (package:sylph/src/utils.dart:43:5)
#1      LocalPackageManager.copy (package:sylph/src/local_packages.dart:103:9)
<asynchronous suspension>
#2      bundleFlutterTests (package:sylph/src/bundle.dart:56:23)
<asynchronous suspension>
#3      sylphRun (package:sylph/src/sylph_run.dart:53:9)
<asynchronous suspension>
#4      main (file:///Users/goncalopalma/.pub-cache/hosted/pub.dartlang.org/sylph-0.5.0/bin/main.dart:73:13)
<asynchronous suspension>
#5      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:303:32)
#6      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:172:12)

• Integration tests are dart programs with no app dependencies.
• The debug app is already installed on the device by the time the test is invoked on Device Farm
• But flutter pub get requires the app dependencies

Removing the app dependencies on the fly removes unneeded dependencies on local and private repositories on Device Farm.

This fixes problems described in #99

In our project we use a monorepo and we have a structure like:

• Root
  -- apps
  --- app1 (sylph.yaml and flutter drive test is located here)
  -- pkgs
  --- feature1
  --- feature2

When we run sylph we receive the following error:

Because app1 depends on feature1 from path which doesn't exist (could not find package feature1 at "../../pkgs/feature1"), version solving failed.

The app builds correctly when running flutter build.

Does sylph support our structure at the moment and if not are there any plans for supporting monorepo's ?

Keep up the good work!

stacktrace:

FINE: Exception type: SolveFailure
FINE: package:pub/src/solver/version_solver.dart 308:5   VersionSolver._resolveConflict
    | package:pub/src/solver/version_solver.dart 131:27  VersionSolver._propagate
    | package:pub/src/solver/version_solver.dart 96:9    VersionSolver.solve
    | ===== asynchronous gap ===========================
    | dart:async                                         _AsyncAwaitCompleter.completeError
    | package:pub/src/solver/version_solver.dart         VersionSolver.solve
    | ===== asynchronous gap ===========================
    | dart:async                                         _asyncThenWrapperHelper
    | package:pub/src/solver/version_solver.dart         VersionSolver.solve
    | package:pub/src/solver.dart 35:10                  resolveVersions.<fn>
    | package:pub/src/log.dart 378:18                    progress
    | package:pub/src/solver.dart 32:10                  resolveVersions
    | package:pub/src/entrypoint.dart 203:24             Entrypoint.acquireDependencies
---- End log transcript ----
pub get failed (66)
Unhandled exception:
command failed: cmd='flutter packages get'
#0      cmd (package:sylph/src/utils.dart:43:5)
#1      bundleFlutterTests (package:sylph/src/bundle.dart:52:3)
<asynchronous suspension>
#2      sylphRun (package:sylph/src/sylph_run.dart:48:9)
<asynchronous suspension>
#3      main (file:///Users/user/.pub-cache/hosted/pub.dartlang.org/sylph-0.4.0/bin/main.dart:80:13)
<asynchronous suspension>
#4      _startIsolate.<anonymous closure> (dart:isolate-patch/isolate_patch.dart:299:32)
#5      _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:172:12)

Did you send as issue to the Amazon team about the tooling to try to get it moving with IOS testing ?

Wondering where the cause of this is ?