getsentry / probot-report

Probot app that sends periodic reminder emails for outstanding PR reviews

Probot: Report

a GitHub App built with probot that sends out periodic reports

Disclaimer: This report Bot is heavily focused on our setup, so in order to use it you probably need to fork it or help us make it more general purpose.

Usage

The bot is activated by the file .github/report.yml in the settings repository. If not configured otherwise, this is the probot-settings repository in your organization.

The file can be empty, or it can override any of these default settings:

# Local times at which the bot will send automatic reports
reportTimes:
  - 09:00
  - 12:30
# Timezone offset for all users where the timezone cannot be inferred
# Defaults to PDT (UTC-07:00)
defaultTimezone: -420
# Maximum number of days to report new issues
newIssueDays: 7
# Ignores all issues that match this regular expression in their title
ignoreRegex: "\\bwip\\b"
# Ignores all issues with these lables
ignoreLabels:
  - duplicate
  - wontfix
  - invalid
# Mailer configuration, can be omitted to disable email
email:
  # Name of the email sender
  sender: '"🤖 Eos - Github Bot" <[email protected]>'
  # E-Mail subject
  subject: "Github needs your attention"
  # Template for the entire email body
  bodyTemplate: >
    Hi <%- user.name %>,
    <%= toReview %>
    <%= toComplete %>
    <%= newIssues %>
  # Template to render a single issue
  issueTemplate: >
    <% _.forEach(issues, function (issue) { %>
      <li>
          <b><a href="<%= issue.html_url %>">
            <%- issue.repository_url.match('[^/]+/[^/]+$')[0] %>#<%- issue.number %>
          </a></b>:
          <%- issue.title %><br />
          <small>
            opened <%- moment(issue.created_at).fromNow() %>,
            updated <%- moment(issue.updated_at).fromNow() %>
            by <a href="<%= issue.user.html_url %>"><%- issue.user.login %></a>
          </small>
      </li>
    <% }) %>
  # Template to format the section of PRs to review
  toReviewTemplate: >
    <p>These pull requests need to be <b>reviewed</b>:</p>
    <ul>
      <%= issues %>
    </ul>
  # Template to format the section of PRs to complete
  toCompleteTemplate: >
    <p>These pull requests need to be <b>handled</b>:</p>
    <ul>
      <%= issues %>
    </ul>
  # Template to format the section of new issues
  newIssuesTemplate: >
    <p>There are issues you could <b>label and assign</b>:</p>
    <ul>
      <%= issues %>
    </ul>

Setup

This Probot app requires authentication tokens and credentials for third party apps in environment variables. The project contains a template for environment variables located at .env.example. Copy this file to .env in the project root and adjust all environment variables.

Github App

First, create a GitHub App by following the instructions here. Then, make sure to download the private key and place it in the root directory of this application or set it via the PRIVATE_KEY environment variable. Finally, set the following environment variables:

Name	Description
APP_ID	Unique ID of the GitHub App
WEBHOOK_SECRET	Random webhook secret configured during app creation
SETTINGS_REPO	optional. Repository to store configs in.

Within your organization, create a repository to store the configuration file. If not configured otherwise, it defaults to probot-settings.

Sendgrid mailing

The bot can send report emails to all organization members with configured email addresses (defaulting to their public email address). This requires a Sendgrid account. Once created, configure the API token as SENDGRID_TOKEN environment variable.

Leave this value empty to skip report emails.

Slack

The bot can connect to a Slack team and send summaries there. To do so, it needs to be registered as Slack bot. Once it has been created, configure its token in the SLACK_TOKEN environment variable.

Leave this value empty to skip connection to Slack.

Development

To start the development server, make sure the following environment variables are set:

Name	Description
DRY_RUN	Disables actual releases. Set to true
SUBDOMAIN	Subdomain for localtunnel to receive webhooks
LOG_LEVEL	Sets the loggers output verbosity. Set to debug

Then, install dependencies and run the bot with:

# Install dependencies
yarn

# Run the bot
yarn start

# Run test watchers
yarn test:watch

We highly recommend to use VSCode and install the recommended extensions. They will configure your IDE to match the coding style, invoke auto formatters every time you save and run tests in the background for you. No need to run the watchers manually.

Testing

The bot includes an automated test suite that includes unit tests, linting and formating checks. Additionally, this command generates a coverage report in coverage/. You can run it with npm:

yarn test

We use prettier for auto-formatting and eslint as linter. Both tools can automatically fix most issues for you. To invoke them, simply run:

yarn fix

Deployment

If you would like to run your own instance of this app, see the docs for deployment.

This app requires these Permissions for the GitHub App:

• Repository contents: Read & write
• Organization members: Read-only

Also, the following Events need to be subscribed:

• Push: Git push to a repository
• Membership: Team membership added or removed

Also, make sure all required environment variables are present in the production environment.

probot-report's Issues

Diff the config before writing

Do not attempt to write (i.e. commit) the config if it hasn't changed.

Configurable filters for PRs to review

It would be very helpful if we could customize the filters for filters to review via the settings YAML file. Some of the default filters don't make sense for different teams, and each team will probably have their own unique set of filters that they want to use.

(original post follows)

---

Hi! First of all, thanks for this awesome project, I wanted to build something similar, but now I'm happy to not have to do so! It's actually working pretty well for our organization, we're just using the Slack notifications for now since we don't use Sendgrid.

This commit introduced a change that I don't understand: 8b1d4bb

To me, it seems like you don't need review:none because if you did make a review on that PR, the PR wouldn't be returned by review-requested:${login}, because making a review dismisses the review request. Certainly when I use the Github web UI, and click on "awaiting review from you" from the dropdown, it doesn't add review:none to the query. The commit even adds a comment that implies that the behavior is a bug on Github's part, but to me it seems like expected behavior if the query includes review:none.

I'd appreciate any clarification on this matter. Thanks again!

Make issue sort order configurable

Ignore WIP pull requests

Either by label or by "WIP" in title

Case-insensitive slack commands

The bot only accepts commands in lowercase, that's impractical on mobile where autocorrect usually capitalizes the first word.

Add DRY_RUN to Slack

Our Slack integration currently does not honor the DRY_RUN flag set via the environment, but it should. When DRY_RUN is set to true, the bot may read commands and perform internal actions but not reply or whisper.

Improve config handling

• The default config should be deep-merged to avoid missing nested keys (e.g. email templates)
• Propagate changes in config explicitly to the mailer and Slack to ensure that they are up to date
• Only listen to pushes on the default branch of the settings repository

Make slack commands, case insensitive

Improve Slack Integration

Initially, this bot was planned to work only with GitHub and integrate optionally with Slack. However, this introduces some problems:

• Not every user has a public email configured in their account. In that case, the bot is not able to reach them at all
• There is no way to obtain the email configured for the GitHub organization (e.g. work email). However, people don't like to receive those notifications on their personal email.
• The GitHub API does not offer a reliable way to determine a user's time zone
• Many people prefer communication via Slack over email

Instead, the bot could focus on Slack for user management:

• Users are read from the Slack team instead of GitHub
• The email used in the Slack team is more likely to be related to the work in the organization
• Once a new user gets added, the bot can immediately message them. This also increases the level of engagement
• Users still have to link to GitHub first. The GitHub team check can still be performed, then
• As far as I know, the Slack API also exposes users' time zones

Use Sendgrid's sandbox mode with DRY_RUN

https://sendgrid.com/docs/Classroom/Send/v3_Mail_Send/sandbox_mode.html

Suggest add `probot-app` into topics for this Github repo

Update cached timezones

Users' timezones are only read on initial import. However, it should be updated as soon as the user pushes a commit from a new time zone.

Honor "do not disturb" times for all users

Mainly weekends, maybe also holidays

Also check for org scope instead of listing all open PRs

https://github.com/getsentry/probot-report/blob/master/lib/reporter.js#L88

Create a README and instructions