alien-tools / breakbot

BreakBot tracks the introduction of breaking changes in Java libraries on GitHub and their impact on client projects.

BreakBot

If we make this change to our code, how will it impact our clients?

What if I remove this type? What if I deprecate this method? What if I change this field's type? What if I introduce this new method?

It is difficult for library maintainers to answer these simple—yet essential!—questions when evolving their libraries. Understanding the impact of a change on your own codebase is complicated enough, so how can you understand its impact on code that you've never even heard of?

BreakBot is a GitHub App, built with Probot, which automatically tracks the introduction of breaking changes in Java libraries and their impact on client projects. Once installed on a repository, BreakBot listens to all pull requests and creates GitHub check runs. These checks highlight the breaking changes introduced by the pull requests and the impact they have on a selected set of client projects. This information feeds the code review process and helps maintainers to decide whether the changes should be accepted in their current form or not.

Content

1. About BreakBot
2. Configuration
3. Installation
4. Other Resources
5. Future Work

About BreakBot

Reports

BreakBot reports consist of a summary that summarizes the list and impact of breaking changes, e.g.:

This pull request introduces 3 breaking changes, causing 22 detections in client code. 3 of 30 clients are impacted by the changes (10%).

Completion Status

BreakBot ends its analysis with either of these status:

• Success when no breaking change is detected
• Neutral when breaking changes are detected
• Skipped when the analysis is skipped (e.g., no Java files affected by the pull request)

Supported Breaking Changes

The underlying analysis is realized using Maracas, which in turn relies on japicmp.

Once installed on a repository, it will publish a report as a check on each new pull request. This report contains a list of the breaking changes created by this PR, compared to the base branch, and a list of the clients impacted for each PR.

Configuration

Similar to other GitHub Apps, BreakBot configuration must be hosted in a file .github/breakbot.yml directly within the repository.

Build

To conduct the analyses, BreakBot needs to be able to build the library and produce a JAR file for both the base and head branches of the pull requests. Currently, only Maven and Gradle are supported. By default, BreakBot autodetects a build file (pom.xml for Maven and gradlew for Gradle) in repository's root and runs a mvn package or ./gradlew accordingly. It is however possible to configure specific sub-directory, source directories, goals, and properties:

build:
  # Only build a sub-directory
  module: core/
  # Sources directory, if non-standard
  sources: src/main/customjava
  # Custom goals
  goals: [clean jar-goal]
  # Skipping tests and assemblies to speed un the build
  properties:
    maven.test.skip: true
    assembly.skipAssembly: true

Clients

BreakBot needs to know about a list of clients to analyze. Currently, they have to be supplied manually as a list of GitHub repositories (denoted owner/repo). By default, BreakBot looks for client code in the latest version (HEAD) of the repository's default branch, but this can be configured. [TODO: This part is not completely clear. What do you mean by client code here (e.g. tests)? ] Note that, in contrast to the library itself, BreakBot does not need to build and produce a JAR for client projects. Instead, BreakBot needs to know where the source files to analyze are located (by default, src or src/main/java). If the source files are located elsewhere (e.g., in a particular submodule), this can be specified using the sources property.

clients:
  # Analyze the latest version (HEAD) of the default branch of user1/repo1
  - repository: user1/repo1
  # Analyze a particular module from user1/repo2, at a particular commit
  - repository: user1/repo2
    sources: module/src/main/java
    sha: a3b98f
  # Analyze a particular branch of user2/repo1
  - repository: user2/repo1
    branch: dev

Exclude some APIs from the analysis

Some parts of your APIs may be exempt from compatibility guarantees: types annotated with a @Beta annotation, everything within a *.internal.* package, etc. This can also be configured in the breakbot.yml file. All the elements matching the given patterns will be excluded from the analysis: no breaking change will be reported. The patterns are given in the excludes section: patterns starting with @ denote annotations (every element tagged with this annotation is excluded from the analysis), while other regex-like patterns specify that any element whose name matches the regex should be excluded from the analysis.

excludes:
  - '@Beta'
  - '*internal*'

Installation

Other Resources

Contributing

If you have suggestions for how BreakBot could be improved, or want to report a bug, open an issue! We'd love all and any contributions.

For more information, check out the Contributing guide.

Contributors

• Léonard Rizzo
• Lina Ochoa
• Thomas Degueule
• Jean-Rémy Falleri

Future Work

• Automatically discover the clients to analyze, e.g. using Maven's dependency graph or GitHub's dependents list.
• Build the list of breaking changes between two versions of a library from source code only, avoiding the need for building JARs and using japicmp.

breakbot's Issues

User-defined checks conclusions

There should be a way to let the user configure the severity of checks conclusions in breakbot.yml, eg:

checks:
  fail_on_bc: false
  fail_on_impact: true

Subscribe to the `synchronize` webhook

Currently, BreakBot only analyses the PR's code once, when the PR is opened (https://github.com/labri-progress/breakbot/blob/main/src/index.ts#L43).

In order to re-run the checks following every new commit in the PR (or a change of base branch), we shall subscribe to the synchronize event.

BreakBot should complete with status 'success' when no breaking change is detected

Check how to read configuration values from the .breakbot.yml

Authentification du bot sur GitHub à l'API

Design of the Checks report

• Which information do we want to display?
• In which form?
• What's in the title? summary? full report? etc.

Mettre en place le traitement du json de Maracas-Spoon

Design document of .breakbot.yml

BreakBot should subscribe and react to the check_suite webhooks

Ecrire un fichier de configuration initial de breakbot pour Spoon

Subscribe to the event fired when the user selects a different commit of the PR from the Checks webpage

Transferer breakbot sur l'orga breakbot après un petit clean de code

Migrer le bot sur Heroku

Mettre en place un dépôt library test et un dépôt client test

Deployer le bot sur Spoon

Réaction de maracas si la demande a déjà été traitée

Est-ce que maracas appelle toujours une callback après avoir envoyé un 202 ?

BreakBot should skip certain pull requests

BreakBot shouldn't analyse the PR and set its completion status to skipped when:

• The PR does not affect any Java code
• The check is explicitly skipped (no-ci? no-breakbot? etc)
• The PR has a certain label
• etc?

Properly handle and display errors returned from Maracas

Gérer le cas où la pull request vient d'un repo différent

• Breakbot: doit savoir sur quelle repo mettre ses checks
• Maracas: doit prendre en compte des adresses de repos différentes pour la branch head et la base
• Communication: on doit échanger les informations de baseRepo et headRepo

413 Payload Too Large

The JSON returned by Maracas for large PRs with large delta models and/or detections sometimes trigger a 413 Payload Too Large on the client (BreakBot) side.

Configuring express with something similar to what follows should do the trick:

app.use(bodyParser.json());
app.use(bodyParser({limit: '5mb'}));

Mettre à jour les tests sur le bot

Etudier l'API des checks et intégrer breakbot à cette API

Move .breakbot.yml to .github/ to match good practices

Display BCs as Checks annotations on the repository's source code

Find a new name for detections

Identifier les projets reverse-ci de Spoon : https://github.com/SpoonLabs https://ci.inria.fr/sos/

Migrate breakbot to the alien-tools organization

Forker un Spoon pour tester l'infra

Document breaking changes in the report

Ensure the check finishes

If there is an error on Maracas side, set a timer that will finish the check.