flank / flank-dashboard

Integrate with popular CI tools to collect software project metrics like performance, build stability, and codebase quality.

Flank

Flank is a massively parallel Android and iOS test runner for Firebase Test Lab.

Flank is YAML compatible with the gcloud CLI. Flank provides extra features to accelerate velocity and increase quality.

Documentation is at flank.github.io/flank

Github action documentation is at repository

flank-dashboard's Issues

Setup Firebase Auth for Google login

Setup Firebase Auth so that users login via Google before seeing the dashboard. The production dashboard should not be publicly visible without Google Auth.

• https://firebase.google.com/docs/auth

Yaml Support - Refactor, add documentation, add tests

Create circle progress widget

• Design widget that reads in data from the model and renders according to the spec
• Write a component spec to verify the widget works as expected
  • https://flutter.dev/docs/testing
  • https://flutter.dev/docs/testing#widget-tests
• Write a FlutterDriver test to verify the widget works in a real browser
  • see #18

See https://github.com/gskinnerTeam/flutter_vignettes/tree/master/vignettes/spending_tracker#budgettracker-graphs for an open source example graph.

• Range should be from 0 to 100%

• implement widget
• add tests
• integrate with state management and dashboard page
• documentation

Slack integration

When Flaky test is detected we want to be able to notify team about the problem, the quick way would be to send Slack message with URL.

• Configure slack integration
• Send a message to Slack

Setup the Dart | Metrics build job

https://app.bitrise.io/app/484e6ac68e24cada#/builds Dart | Metrics

Setup the Dart | Metrics build job. The metrics build job should be updated to only build the metrics project + related tests.

Import Jenkins data to Firestore

Develop a program that uses the Jenkins API to import all builds for a given job to Firebase. This should be idempotent. The program will run on a schedule and constantly update Firestore with the latest builds.

Name	Description
buildId	unique identifier of the build
buildDate	date the build was started
buildQueued	the amount of time the build spent waiting to execute
buildDuration	the amount of time the build took to execute (does not include queue time)
buildSuccessful	boolean for if the build was successful or not

Jira Integration testing

Use testing approach created in #44 to create Jira Integration tests.

Guardian

Slack Integration Testing

Use a testing approach created in #44 to create Slack Integration tests.

Identify strategy for state management

Evaluate https://pub.dev/packages/states_rebuilder

• States Rebuilder – ZERO Boilerplate Flutter State Management (video)
• States Rebuilde (article)
• https://github.com/ResoCoder/flutter-states-rebuilder-tutorial

Compare to BLOC and provider

Widget Theme support

A company should be able to change the theme so that colors in the app reflect the brand colors for that company.

Investigate golden screenshot testing

Screenshot testing to verify widgets visually look the same may be helpful. This feature is in development. After the PR merges, we may investigate this technique for the metrics project.

flutter/flutter@bf347e5

https://medium.com/@martin.kamleithner/test-your-flutter-widgets-using-golden-files-b533ac0de469

Ebay posted about a better way of doing golden image testing.

• https://tech.ebayinc.com/engineering/ebay-motors-screenshot-testing-with-flutter/
• https://github.com/eBay/flutter_glove_box/tree/master/packages/golden_toolkit
• flutter-gold.skia.org
• https://github.com/flutter/flutter/wiki/Writing-a-golden-file-test-for-package:flutter
• https://flutter-gold.skia.org/list
• https://github.com/google/skia-buildbot/tree/master/golden
• flutter/plugins#2904 - screenshots via webdriver, compared using golden

Firestore integration

We need to make sure that basic integration with Firestore from CLI works well:

• Query data from firestore
• Add data to the firestore

No auth needed for now.

JSON reporting for Flutter Driver test results

See if it's possible to save the internal flutter driver results to a JSON file. If successful, we'd be able to transform this format into JUnit XML in the future.

Currently there's no machine readable output for test results.

/// A subclass of [LiveTestWidgetsFlutterBinding] that reports tests results
/// on a channel to adapt them to native instrumentation test format.
class E2EWidgetsFlutterBinding extends LiveTestWidgetsFlutterBinding {
  /// Sets up a listener to report that the tests are finished when everything is
  /// torn down.

• https://github.com/flutter/flutter/pull/49833/files#diff-4af4f37bbda3fe167384f2d69d452ff4R14

context['\$flutterDriverResult'] = json.encode(result);

• https://github.com/flutter/flutter/pull/49833/files#diff-7997ccc9c4f340c6c35520af27c86ad1R28

e2e package

• https://github.com/flutter/plugins/blob/master/packages/e2e/lib/e2e.dart
• https://github.com/flutter/plugins/tree/master/packages/e2e#e2e
• https://github.com/flutter/plugins/blob/master/packages/e2e/example/test_driver/example_e2e.dart

---

Flutter itself uses a JSON reporter and stores results in BigQuery.

https://github.com/flutter/flutter/blob/master/dev/bots/test.dart
https://github.com/flutter/flutter/blob/master/dev/bots/flutter_compact_formatter.dart

Create utility that can parse JUnit XML files

To get test results data we need to be able to read JUnit XML files from different test runners. For that, we need to be able to build a utility that can parse JUnit XML, support different JUnit XML variations and return data in the object model.

Here are sample JUnit XML files:

android_fail.xml

android_pass.xml

ios_pass.xml

Ios_fail.xml

Example JUnit parsing test written in Kotlin for the files above
JUnit parsing test

Research

Prepare detailed specification on how flaky test detection will work

Review relevant information presented in shared word file.

Prepare a detailed plan on how Flaky test detection will work, what components will be used.

Current high-level understanding (can be changed after investigation):

1. Parse JUnit XML
2. For each test - get the previous test run results from Firebase
3. Analyze history & detect Flaky test (specify how the algorithm will work)
4. Display results/Save to Firebase

Implement Flaky Test detection algorithm

As we get detailed Flaky Test detection algorithm in #41 we want to implement a Flaky Test detection algorithm according to specification.

flutter_web_driver.dart improvements

Currently flutter_web_driver.dart doesn't stream the results to stdout and the path to the log files are not printed.

Tasks

• Print results in real time to stdout
• Print log file location to stdout after saving the files

Create working Flutter Web Driver test

• Fix ./metrics/test/driver_test.dart to run Flutter Web Driver test.
• Update test to include a simple smoke test to ensure the app is working
• Run test in CI on Bitrise

flutter/flutter#45951

Trigger 3rd party integrations on Flaky test detection

On Flaky test detection trigger (if configured):

• Jira Integration
• Slack Integration

Import Circle CI data to Firestore

Develop a program that uses the Circle CI API to import all builds for a given job to Firebase. This should be idempotent. The program will run on a schedule and constantly update Firestore with the latest builds.

Name	Description
buildId	unique identifier of the build
buildDate	date the build was started
buildQueued	the amount of time the build spent waiting to execute
buildDuration	the amount of time the build took to execute (does not include queue time)
buildSuccessful	boolean for if the build was successful or not

Port Buildkite Shellwords parser from Go to Dart

The task is to create a standalone Dart port of the shellwords library from Buildkite. Shellwords is useful for parsing strings in the same way as the UNIX Bourne shell.

A golang library for splitting command-line strings into words like a Posix or Windows shell would.

https://github.com/buildkite/shellwords

Tasks

• Configure the shellwords job on Bitrise. https://app.bitrise.io/app/65e2c5934061cc08
• Port parser https://github.com/buildkite/shellwords/blob/master/parser.go
• Use the right quotes depending on OS https://github.com/buildkite/shellwords/blob/master/quote.go#L8
• Port tests

Import Bitrise data to Firestore

Develop a program that uses the Bitrise API to import all builds for a given job to Firebase. This should be idempotent. The program will run on a schedule and constantly update Firestore with the latest builds.

Name	Description
buildId	unique identifier of the build
buildDate	date the build was started
buildQueued	the amount of time the build spent waiting to execute
buildDuration	the amount of time the build took to execute (does not include queue time)
buildSuccessful	boolean for if the build was successful or not

• https://api-docs.bitrise.io/

Setup Jenkins testing environment

Research how to setup a public Jenkins and Buildkite service that can be used to validate the integration logic. The build import scripts will point to these test services and ensure the data is imported to Firebase successfully.

Adopt lint package for linting

Google publicly shares their internal rules as package:pedantic in open-source. It represents what Google is enforcing internally throughout all Dart code. For a lint rule to be added to pedantic, Google has to change all code which doesn't follow the style.

This strict practice results in only 27/150+ rules to be enabled. While some are contradictory to each other and can not enabled together, a big chunk of rules isn't enabled because it requires too much work to update all of Googles existing code.

For developers outside Google, it is the norm to have separate lint rules per project. One project might enable more rules then others.

lint enables a majority of lint rules, leaving out contradictory and very opinionated rules.

https://pub.dev/packages/lint

https://resocoder.com/2020/01/17/dart-linting-tutorial-healthy-flutter-codebase-with-analysis_options-yaml/

• make sure that lint warnings fail build on bitrise
• as we use monrepo check if we can use same lint for all projects

Setup Firebase deploy for Flutter Web

Create an optimized deployment process that uploads the project to Firebase hosting.

• https://firebase.google.com/docs/hosting
• https://flutter.dev/docs/deployment/web
• https://dev.to/valentinprgnd/multiple-environments-with-firebase-hosting-1ao4

Doc for guardian

Commit research doc with info from:

• LinkedIn
• Shopify
• Sony
• Netflix

CLEAN architecture with state rebuilder

• https://resocoder.com/2019/08/27/flutter-tdd-clean-architecture-course-1-explanation-project-structure/

• https://medium.com/flutter-community/state-management-gymnastics-using-states-rebuilder-part-1-3ba3a6abf9c7

• https://medium.com/flutter-community/state-management-gymnastics-using-states-rebuilder-part-2-a7fa0dd7dc51

Firestore integration testing

Use testing approach created in #44 to create Firestore Integration tests.

Create project skeleton

Create a simple skeleton for the web UI.

Import Buildkite data to Firestore via GraphQL API

Develop a program that uses the Buildkite GraphQL API to import all builds for a given job to Firebase. This should be idempotent. The program will run on a schedule and constantly update Firestore with the latest builds from Buildkite.

Name	Description
buildId	unique identifier of the build
buildDate	date the build was started
buildQueued	the amount of time the build spent waiting to execute
buildDuration	the amount of time the build took to execute (does not include queue time)
buildSuccessful	boolean for if the build was successful or not

https://gist.github.com/bootstraponline/f2f76d746233a3281e2d86e93d7508e1

---

• https://github.com/gql-dart/gql
• gql-dart/gql#4
• https://www.reddit.com/r/dartlang/comments/f4e9pv/graphql_a_call_for_contributions/

Create build results widget

• Design widget that reads in data from the model and renders according to the spec
• Write a component spec to verify the widget works as expected
• Write a FlutterDriver test to verify the widget works in a real browser
• Note that the individual bars in the graph should be clickable. Once clicked, they will open a link to the build on Jenkins/Buildkite/etc.

FAQ

1. How many bars should we render?
   Let's start with 14 based on the design, but make it configurable.

2. Is the graph interactive?
   Yes, the user should be able to click onto an individual bar and that will open a new URL. See https://buildkite.com/bazel

3. Should we use real data?
   For now, mock out all the data.

Graph should resize based on screen size. See Google example:
https://google.github.io/charts/flutter/example/bar_charts/spark_bar

Identify routing approach

Routing be decoupled from the buildContext.

Research routing packages and make a recommendation.

• https://resocoder.com/2020/01/10/flutter-zero-boilerplate-router-with-auto-route-flutter-navigation-tutorial/
• https://pub.dev/packages/get

Setup Firebase hosting - free plan

Auto deploy metrics web app to Firebase hosting free plan via the bitrise build.

https://firebase.google.com/docs/hosting

Import JUnit XML data to Firebase

After JUnit XML parsing we want to store test data in Firebase so this data can be used for Flaky Test detection algorithm.

• Firestore structure
• Data model
• Integration with Firestore
• Save data

Note: for now we'll skip Auth as it can be handled later on once all pieces work together well.

Setup Buildkite testing environment

Research how to setup a public Buildkite service that can be used to validate the integration logic. The build import scripts will point to these test services and ensure the data is imported to Firebase successfully.

Define data model

Setup data model the Web UI will use to render the widgets.

Update tickets & design doc

Development workflow:

• Author initial set of epics, tickets, with estimates and assigned milestone
• Author initial design and design doc
• Review existing tickets and add new tickets for missing work
• Update design doc with any missing information

Jira Integration

To act on detected Flaky test we want to be able to integrate with Jira so the new ticket can be created (or existing updated).

For current Jira integration implementation, we assume that Simple workflow is used. Ticket type = "Task".

Jira integration should allow:

• Jira integration configuration (Project, Ticket template)
• Open ticket
• Close ticket
• Find ticket by test key
• ?Reopen ticket

Setup bitrise build for guardian CLI

Web UI

Update readme to include more details about Guardian & Metrics projects

We need to add more information to readme about Guardian & Metrics projects: description, purpose, etc..

Firebase Cloud Firestore

Review 3rd party API testing techniques

As we integrate with 3rd party services like Jira & Slack we want to make sure that our integration is reliable.

As a result of this ticket completion, we want to have documented 3rd party API testing approach we plan to use for Jira & Slack integration testing.

Add design resources to GitHub

Create folder in the repo with the Adobe XD design.

Fix dart analysis warnings

Setup the Dart | Guardian build job

https://app.bitrise.io/app/94d3963118a1f4fe#/builds Dart | Guardian

Setup the Dart | Guardian build job. The guardian build job should be updated to only build the guardian project + related tests.

Create sparkline graph widget

• Design widget that reads in data from the model and renders according to the spec
• Write a component spec to verify the widget works as expected
• Write a FlutterDriver test to verify the widget works in a real browser

See https://github.com/gskinnerTeam/flutter_vignettes/tree/master/vignettes/spending_tracker#budgettracker-graphs for an open source example graph.

https://github.com/aeyrium/bezier-chart also looks interesting

Research/refactor browser_name.dart

https://github.com/software-platform/metrics/blob/c8a8a418c4d695ee0f7872678a3b6deed5c334fe/metrics/test_driver/browser_name.dart#L1-L20