fastlane / ci

Open source, self hosted, mobile optimized CI powered by fastlane

fastlane is a tool for iOS and Android developers to automate tedious tasks like generating screenshots, dealing with provisioning profiles, and releasing your application.

---

✨ All fastlane docs were moved to docs.fastlane.tools ✨

---

Need Help?

Before submitting a new GitHub issue, please make sure to

• Check out docs.fastlane.tools
• Search for existing GitHub issues

If the above doesn't help, please submit an issue on GitHub and provide information about your setup, in particular the output of the fastlane env command.

Note: If you want to report a regression in fastlane (something that has worked before, but broke with a new release), please mark your issue title as such using [Regression] Your title here. This enables us to quickly detect and fix regressions.

fastlane team

Iulian Onofrei	Manu Wallner	Fumiya Nakamura	Helmut Januschka	Jorge Revuelta H
Jan Piotrowski	Daniel Jankowski	Łukasz Grabowski	Maksym Grebenets	Stefan Natchev
Josh Holtz	Manish Rathi	Felix Krause	Kohki Miki	Joshua Liebowitz
Aaron Brager	Roger Oba	Luka Mirosevic	Matthew Ellis	Satoshi Namai
Jimmy Dee	Max Ott	Jérôme Lacoste	Olivier Halligon	Danielle Tomlinson
Andrew McBurney

Special thanks to all contributors for extending and improving fastlane.

Contribute to fastlane

Check out CONTRIBUTING.md for more information on how to help with fastlane.

Code of Conduct

Help us keep fastlane open and inclusive. Please read and follow our Code of Conduct.

Metrics

fastlane tracks a few key metrics to understand how developers are using the tool and to help us know what areas need improvement. No personal/sensitive information is ever collected. Metrics that are collected include:

• The number of fastlane runs
• A salted hash of the app identifier or package name, which helps us anonymously identify unique usage of fastlane

You can easily opt-out of metrics collection by adding opt_out_usage at the top of your Fastfile or by setting the environment variable FASTLANE_OPT_OUT_USAGE. Check out the metrics code on GitHub

License

This project is licensed under the terms of the MIT license. See the LICENSE file.

This project and all fastlane tools are in no way affiliated with Apple Inc. This project is open source under the MIT license, which means you have full access to the source code and can modify it to fit your own needs. All fastlane tools run on your own computer or server, so your credentials or other sensitive information will never leave your own computer. You are responsible for how you use fastlane tools.

---

✨ All fastlane docs were moved to docs.fastlane.tools ✨

---

ci's Issues

fastlane.ci should also init submodules

GitHub integration

• Personal API token generation with proper scope
• support SSH/HTTP/GIT protocols
• Post comments on GitHub with the test failure information #58

Login with GitHub button

Get fastlane.ci running on container for sample repo

Post comments on GitHub with the test failure information, ideally the stack trace + error message. That's a freeby, as we already set the build status

We could start as simple as adding a comment to the PR with Build failed because of 'rspec 34 tests failed' or similar, and then we can slowly expand this to support more.

It kind of conflicts with danger, so let's think about what's the healthiest option of not duplicating danger features

Automatically render the right layout file across all views

Render basic Fastfile metadata in CI

Need to work on open TODOs on https://github.com/fastlane/fastfile-parser

Server configuration

• First time setup
• CI User config
• Create new config repo
• Sanity test all configuration before moving on
• Somehow create github account for fastlane.ci
• Ensure proper scopes for fastlane.ci github user

Drive fastlane from fastlane ci

Setup scss (or similar) for application css with support for variables (compiled to CSS)

Finish Swift upgrade stuff

config_service update get projects

Bundled fastlane and fastlane ci

Project detail page

• Build status
• Rolling logger
• See any build artifacts (ipa, dsyms, zips, screen shots)
• Who made the latest changes
• SHA of latest commit
• Link to diff
• Find a way to resume or re-run pending builds #52

Open Source Checklist

Before open sourcing, we need to tackle these tasks

• Reset fastlanebot ci API Token
• Reset taquitos ci API Token
• Update architecture doc
• Enforce design patterns

Storing user account and loading them into the session

Git Integration

• Clone
• Branch
• Commit as user
• Commit as ci
• amend commits
• poll for changes
• push changes
• handle push failures gracefully
• Fix warning where we somehow chdir in folders multiple times #59

Add infrastructure to handle creating fastlane ci users, and linking providers

Because we can't use the callback-url from github to verify a real user and create a user record and utilize a callback URL, we need to create user accounts to store data like github api tokens. If we could get call backs, we could store github access tokens that are generated during github user authorization. We'd store those with their email automatically, and then check with github if a user is still valid.

But, since we can't get a github access_token (need to support github app callbacks), we need to create an account manually with a password check.

Create something that is flexible and can scale to use other service providers like BitBucket or custom git url.

Integrate Material Design framework

All git commands should use the user's login, or the fastlane.ci login

Alright, quick update on the git pull and git push using the user's session. A few things:

We currently have:

• GitConfigDataSource, which is a data source for configuration files, meaning list of projects, and CI config
• GitHubSource, the connection to GitHub, GitHub specific, used to report build status, list repos, etc.

We need a new class that does the git overhead, around auth, error handling, managing of repos, etc. What I was thinking, having a GitHelper class, that does it. As part of each method call, we require the user of the class to pass a valid GitHub session. The class also verifies that we only work with https:// URLs just to unify the whole thing.

The user of the method call can decide:

• Was it a user action? If yes, they're gonna pass the user's auth token, so their face is gonna appear (e.g. https://github.com/KrauseFx/ci-config/commit/d5db054433552b1a82cfed8c913d67d6d6e320b8)
• Was it not triggered by being on the CI front-end by a user? (e.g. scheduled job, triggered by a commit, triggered by time)? We're gonna need a fastlane.ci user, so that not a user session is being used.

I started looking into how git auth works for temporary pulls and pushes, and it seems like there is no good built-in way to contain the whole thing. Will consult with some of my friends who know more about git.

It's tricky to pass the credentials, without affecting the whole system, and not leaking the data anywhere (build output, error case, etc.)

Open TODOs

• The things above
• When and how do we authenticate the CI itself for the second kind of git commits? Where and how do we store the login?

Via #18

Implement background workers without adding a heavy dependency

For now, this is needed to

• Pull changes from the config repo every minute to avoid conflicts
• Monitor GitHub repos for changes, as we can't listen to callbacks (also easier to start)

The dependency should we as lightweight as possible, and ideally doesn't require a fully featured database, or maybe just an SQLite database. Also, we have to make sure it's MIT licensed, as not all are

Sinatra doesn't seem to like a trailing `/`

http://127.0.0.1:8080/login/

is treated differently than

http://127.0.0.1:8080/login

That's not how most web services work AFAIK, and we should strip trailing /

[Configuration] Use "Login with GitHub" system to also authenticate local command line to pull and push changes in configuration layer

We have to assume an empty Mac Mini

[Discussion] How do we store build artifacts

The goal of having 100% of the fastlane CI configuration based on git is:

• Users know where and how their data is stored
• Our target group already uses Git
• Easy way to rollback if 💩 hits the fan or a new setting broke something
• If the Mac dies, no need for a backup, 100% of the config is there, all you need to do is to setup a new Mac, point it to the config repo, and you're good to
• Human readable format (JSON)
• More advanced options, that don't deserve their own UI in the web front-end, but we still want people to set certain flags/options means they just use the GitHub web editor to add an option to one of the config files

Data we want to store: 100% of the configuration needed to have a running fastlane CI, which inclues

• Projects with each having many attributes, like
• repo_url
• information about the available Fastfile, like the lanes and their description
• mapping of branch to an array of lanes to run (master = > release, any other branch => test)
• fastlane.ci wide settings (currently none)

We talked about how storing the complete build output and build artifacts can't really be done via git, as it doesn't scale.

The question we have now, where do we store the build artifacts (build numbers, build output, artifacts) without making it confusing to the user that they need a git repo + something else.

Some things we talked about:

1. Store it locally on disk
2. A local database (e.g. Postgres)
3. Use Google Cloud or Firebase real time database (some people might not like the lack of openness here)
4. Find a way to store the logs in git, and only have build artifacts on Google Cloud / AWS S3, with the logs having a reference to a remote URL on AWS S3

We should think about how we want users to migrate to a new Mac, or recover after a data loss, and how to communicate what data is stored where.

Write basic configuration layer

Have a layer for configuration, with static values for now

Notification Mechanism

When something is wrong, we should have some sort of dashboard alert mechanism

• Git permission issues
• Xcode issues
• Apple issues

Report CI status to GitHub

PR #32

Add symmetric encryption for API keys

We need symmetric encryption for storing sensitive things in the .json files, and so they aren't visible in github.

Sessions seem to die when restarting the server? Let's see if we can keep them persistent

Find a way to temporarily use specific git credentials

For git clone, git push, etc we need to find a way to use the GitHub auth token, without affect the system installation. I was only able to find a way to set the author information, not the credentials. It doesn't seem to be part of the git Ruby gem

[Implement] Remember what commits we ran fastlane.ci for

So a commit might be ran on multiple project (e.g. one for testing, one for screenshots, etc.). For each of the projects, we have to keep track for which commits we already ran tests for. Assume that when you setup fastlane.ci, the user probably already has 1,000 commits. We don't want to go back longer than the first time the CI was setup.

So from what I understand is that we have to keep track of what commits we already ran for. We can't just use builds I think, as there might be multiple builds for one commit (e.g. when the user re-triggers), unless we say that a re-trigger overwrites the previous content. I don't know what other CI systems do, I guess they just overwrite?
If we overwrite, that makes our lives easier, because then we can easily identify each build, and therefore use builds to check if a commit is a new commit, or if we already ran the job.

So our worker goes through all the available branches and commits, we then iterate over all projects that have the given repo_url we just git fetched, and then check for all the builds of that project, and see if there is a build that ran for the given sha. In git the sha is the unique thing (it seems). We don't need to know the branch or anything, the combination of repo_url + sha is unique. And we just do that for each project.

Based on that, we can easily check if we already have a build for that specific commit, and if not, we can run the tests, and report the build status to GitHub. At the same time we generate a new Build object with the build result, output, sha, etc. and store it wherever we want to store them.

Add an icon pack we can use, with a proper license we can use

Looks like there is a free one part of the material design library we're using: https://getmdl.io/styles/index.html and/or https://google.github.io/material-design-icons/ (search for Icon). We should probably just use it

Add CI system account, and update users with encrypted tokens

CI will always need a system account, for now let's assume it's a CI user.

• Generate a CI user for our development linked to the fastlane test projects
• encrypt API tokens
• add josh to ci users

How do we identify a project?

Right now, we're using

[
  {
    "repo_url": "https://github.com/KrauseFx/bot",
    "enabled": true,
    "project_name": "Bikemap App",
    "id": "e0cee5c2-c175-467d-bcb2-8a37d5177af3"
  }
]

so we kind of use the repo_url (which could be https or git). We need to identify it, so we can access the right GitHub repo using the GitHub api. And also we want to check if we have permission to access a certain project. I added the initial implementation for now 9b70de1

Build list to build details transition

Projects List

• Create project
• See project details summary
• Find a way to resume or re-run pending builds #52

It seems like our current Controller architecture with Sinatra doesn't allow us to efficiently re-use code

Methods applied in fastlane_app.rb inside the class don't apply to the other controllers, this includes setting a global layout file to use, or most recently the usage of sesssions

enable(:sessions)
use(Rack::Session::Cookie, :key => 'rack.session',
                               :path => '/',
                               :secret => ENV["SESSION_SECRET"])

which I seem to have to add to every single controller we have. Do you know if we can change the way we create those classes to allow us to still move fast & in parallel, while still being able to set "global" server attributes, like the layout and the sessions (and probably more in the future)

Add session specific per user, not global like it's now

Every action done by a user uses the underlying GitHub session from this specific user

CI has to have their own GitHub account (bot) for:

• Using git actions as part of your Fastfile

In general, separate if CI does its own work

• Source code and config
  • clone with account information
  • push with account information
• Verify it's https

Define architecture

Fix warning where we somehow `chdir` in folders multiple times

/Users/felixkrause/.rbenv/versions/2.3.3/lib/ruby/gems/2.3.0/gems/git-1.3.0/lib/git/lib.rb:460: warning: conflicting chdir during another chdir block

fastlane integration

• Run selected lane
• Redirect output to where fastlane.ci can read it

Create a doc with the architecture principals

Fix remaining issues with ci parsers (edge cases with platform)

https://github.com/fastlane/fastfile-parser/issues

Clone a user's repo, and manage the branches + commits

fastlane CI: multiple people can login, not just one, and that's your user session, that's also used for config updates, access to repos, etc.

• Also Store confit repo url somewhere
• Session wird gespeichert in session (GitHub token)

Clone repository

Be able to clone a project repository using whatever means needed

Find a way to resume or re-run pending builds

Right now, when a build runs, and the CI server either gets killed or restarted, a Build will stay stuck in pending for the rest of time of the universe. We gotta build something to see if the worker is actually still running, and if not, mark the build as failed.

e.g. https://github.com/KrauseFx/ci-config/commit/6167cdac97f96fc34d1277683652539701cbc638#diff-2d77cf50a00bf53694d062b016527aa0R3

rename get_builds to builds

more ruby, less java

If installed through mac app, present messaging about how to migrate

Name of `load_builds` method

How would you feel about avoiding overhead words like get_, set_ and load_ in methods? instead of load_builds we can just use builds. I think that's more Rubyish