openedx / openedx-webhooks

Webhooks for the Open edX GitHub and JIRA

Open edX Webhooks Handlers (and Other JIRA/GitHub Utilities)

Webhooks for Open edX integrating JIRA and GitHub. Designed to be deployed at Heroku.

Access the app at https://openedx-webhooks.herokuapp.com.

|Build Status|

Set Up Development Environment

Prerequisites

Make sure you've installed:

• Python 3.10.x development environment (virtualenv strongly recommended)

• Heroku Command Line

  All heroku commands can be performed through the Heroku web-based dashboard as well, if you don't want to use the CLI.

Set up

Log in using the email address and password you used when creating your Heroku account:

make deploy-configure

Authenticating is required to allow both the heroku and git commands to operate.

Alternatively, to authenticate with SSH keys:

make deploy-configure DEPLOY_USE_SSH=true

You should see output similar to:

heroku  https://git.heroku.com/openedx-webhooks-staging.git (push)
heroku  https://git.heroku.com/openedx-webhooks-staging.git (fetch)
origin  [email protected]:edx/openedx-webhooks.git (fetch)
origin  [email protected]:edx/openedx-webhooks.git (push)

Develop

This app relies on the following addons from Heroku:

• Redis
• Papertrail
• Scheduler

While it's possible to replicate the entire stack locally, it'll be difficult to ensure consistent experience for each developer. Instead, we utilize the pipeline facility offered by Heroku to handle our development needs.

The general development cycle is:

Code → Deploy branch to staging → Test → Iterate

To deploy your current working branch to staging:

make deploy-stage-branch

To deploy an arbitrary branch:

make deploy-stage-branch DEPLOY_STAGING_BRANCH=feat/my-branch

Once you're satisfied with your changes, go ahead and open a pull request per normal development procedures.

Smoke test the deployment.

If the application isn't running, visit the openedx-webhooks-staging Resources page to make sure there are dynos running.

Run Tests

make install-dev-requirements test

If you are testing a change to repo-tools-data-schema, you need to coordinate changes in both repos:

• Name your branch here the same as your branch in repo-tools-data-schema.
• Be sure your changes in repo-tools-data-schema are pushed to your branch on GitHub.
• Run make testschema to install your branch of repo-tools-data-schema in this virtualenv.
• Run tests here as usual.

Deploy

In most cases, you'll want to deploy by promoting from staging to production. The general workflow is:

Merge to master → Deploy master to staging → Test → Promote to production

Prior to the promotion, make sure all the changes have been merged to master, and you've deployed the master branch successfully to staging:

make deploy-stage

When you're ready to promote from staging to production:

make deploy-prod

Make sure the same git commit is deployed to both environments.

Make sure the abbreviated git SHAs match.

Smoke test the deployment.

Other things to know

This should no longer be an issue as of July 9th, 2018.

If you re-process pull requests, an unfortunate thing can happen: it will find stale pull requests that were written by edX employees who have now left. The bot will see that the author has no contributor agreement, and will make a new JIRA issue for the pull request. This is needless noise.

The bot looks for comments it wrote that have a JIRA issue id in them. You can leave the bot comment on the stale pull request so that at least it won't happen again in the future.

Configuring a webhook

On GitHub, visit the repo webhooks (https://github.com/<ORG>/<REPO>/settings/hooks) or organization webhooks (https://github.com/organizations/<ORG>/settings/hooks) page.

Create or edit a webhook.

• Payload URL: https://openedx-webhooks.herokuapp.com/github/hook-receiver
• Content type: application/json
• Secret: same as setting GITHUB_WEBHOOK_SECRET in Heroku

• Events:

  • Issue comments
  • Pull requests
  • Pull request reviews
  • Pull request review comments

Changelog

Unreleased

See the fragment files (if any) in the changelog.d directory.

2023-12-07

• When a pull request is closed, if it doesn't have a "open-source-contribution" label on it, it is considered an internal pull request. We assume that it was processed when it was opened, and since it didn't get the label then, the author must have been internal at the time.

2023-11-27

• Now issues created in Jira will have a label of "from-GitHub" on them. Closes issue 279.
• Two possible errors with "jira:foo" labels now create bot comments so people understand why there's no Jira issue. Closes issue 280.
• Fix: we no longer comment twice on a pull request closed with a comment. Closes issue 277.

2023-11-03

• Don't add a "no contributions accepted" comment if the pull request is being closed. It's needlessly discouraging. Fixed #273.

2023-10-31

• Adding a label like jira:xyz to a pull request will look in a private registry of Jira servers for one nicknamed xyz, and then create a Jira issue there to correspond to the pull request.

2023-09-13

• The CLA check used to fail if a pull request had more than 100 commits. Now the head sha is retrieved directly without listing all commits, so the number is irrelevant.

2023-08-30

• Adding a comment to pull request now re-runs the checks and updates the state of the pull request. Previously, we'd edit the title of the PR to trigger updates.

2023-08-11

• Removed: the bot no longer understands the past state of users. This data hasn't been maintained for the last few years. There's no point relying on stale data, so the capability is removed.
• Removed: any understanding of who is a core contributor. The bot no longer makes comments or labels particular to core contributors. Issue 227.

• Removed: we no longer read people.yaml.
• Removed: we no longer add "jenkins ok to test" to start testing, since that comment was only read by Jenkins, which we no longer use.

2023-08-03

• Jira authentication now uses the JIRA_USER_EMAIL and JIRA_USER_TOKEN environment variables. OAuth authentication is removed. These settings are now obsolete and can be deleted:
  • DATABASE_URL
  • GITHUB_OAUTH_CLIENT_ID
  • GITHUB_OAUTH_CLIENT_SECRET
  • JIRA_OAUTH_CONSUMER_KEY
  • JIRA_OAUTH_RSA_KEY
  • SQLALCHEMY_DATABASE_URI
• Stopped the bot from updating a repository's labels based on labels.yaml, as this is now handled by the repo_checks tool. The labels.yaml file is now unused and can be safely deleted from any openedx-webhooks data repositories.
• Stopped the bot from adding the core committer GitHub label to pull requests to repos on which the bot believes the author to have write access. The bot's data source for repository access, people.yaml, is outdated, we do not yet have a strategy for keeping it updated. Until further notice, coding Core Contributors are asked to add the core contributor label to their pull requests manually.

2023-03-03

• Removed the code that used the now-obsolete internal setting in orgs.yaml.
• Tweaked the CLA messages.

2023-03-02

• The "internal" setting is being replaced by an "internal-ghorgs" list on an institution. A pull request is now internal if the author's associated institution (in orgs.yaml) has the org the PR is being made to as an internal-ghorgs org. The old "internal" setting is still used, but we'll be deleting it once the new code is in place.

2023-01-30

• Added: contribution pull requests will be added to GitHub projects if the base repo says to by adding an "openedx.org/add-to-projects" annotation in its catalog-info.yaml file.

2022-07-21

• Adding a pull request to a project could fail if the two are in different GitHub orgs (like edx and openedx). This failure used to stop the bot from making further changes, but now we log the exception and continue.

2022-06-13

• Removing the JIRA_SERVER setting will disable Jira access for the bot. No Jira issues will be created or updated.

2022-06-03

• Blended pull requests now go into a separate project, specified with the GITHUB_BLENDED_PROJECT setting.

2022-06-01

• The JIRA server is now configurable with the JIRA_SERVER environment variable.
• New external pull requests will be added to a GitHub project. The project is configurable with the GITHUB_OSPR_PROJECT environment variable.
• Removed mention of unused JIRA credentials JIRA_ACCESS_TOKEN and JIRA_ACCESS_TOKEN_SECRET.

2022-04-06

• Repos with more than 30 labels might not have properly labelled pull requests that transitioned into late-alphabet statuses (like Open edX Community Review). This is now fixed.

2022-04-05

• Load yaml and csv data files from the openedx/openedx-webhooks-data repo.

2022-03-25

• Pull requests can now be closed and re-opened. When the pull request is re-opened, the survey comment that was added on closing is deleted. The Jira ticket is returned to the state it was in before the pull request was closed.

2022-01-27

• Removed the code that handled "contractor" pull requests, where the bot couldn't know if an OSPR ticket was needed or not.
• The CLA check is now applied to all pull requests, even edX internal ones.

2021-12-20

• The bot now ignores any private repo in the edx organization.

2021-12-17

• We no longer use OAuth authentication for GitHub. All access is with a personal access token.
• The bot now depends on a csv generated by Salesforce to inform which users have signed the Contributor License Agreement (CLA)

• After processing a pull request, the GitHub rate limit is checked and logged:

  Rate limit: 5000, used 29, remaining 4971. Reset is at 2021-12-16 23:26:48

• The "needs CLA" message now includes the possibility that you've signed before and need to re-sign.

2021-09-29

• Removed the NEED-CLA label. We have a check now, which is better.

2021-09-14

• Due to an internal refactoring, now rescanning pull requests will add the end-of-pull-request survey comment if needed.

• Four Jira fields are no longer updated:

  'Github PR Last Updated At' 'Github PR Last Updated By' 'Github Latest Action' 'Github Latest Action by edX'

2021-09-13

• A GitHub check indicates whether the author has a contributor agreement or not.

2021-09-02

• Fix an assertion error that could happen if a pull request had no body (description). The assertion was:

  File "/app/openedx_webhooks/tasks/jira_work.py", line 117, in update_jira_issue

  assert fields

• Change error handling so that more actions can complete even if one fails.

2021-08-30

• Removed one setting: JIRA_OAUTH_PRIVATE_KEY, which was just JIRA_OAUTH_RSA_KEY base64 encoded.

2021-08-18

• fix: all UI pages are now protected with basic auth.

2021-02-25

• Update the CLA link to go to https://openedx.org/cla, which currently redirects to our new Docusign form. If we have to change the form in the future, we can change the redirect on openedx.org.

2021-01-22

• When considering a pull request, we won't update the Jira extra fields if none of our desired fields are different. We used to update a Jira issue if (for example) it had platform map info, but we didn't want to add platform map info.

2021-01-21

• More control over rescanning:

  • You can provide an earliest and latest date to consider. Only pull requests created within that window will be rescanned.

    Rescanning never considers pull requests created before 2018. This is a quick fix to deal with contractor comments.

    Because we don't track when companies started and stopped being contractors, we can't decide now if a pull request should have had a contractor comment when it was created.

    The latest contractor comment on one of our pull requests was in December 2017. So don't consider pull requests that old. Later we can implement a better solution if we need to rescan those old pull requests.

  • Rescanning now has a dry-run mode which records what would have been done, but takes no action.

• Before-clauses in people.yaml are now handled differently. Previously, only one before clause was found, the earliest one that applied to the date we're interested in. Now, all before clauses that apply (with dates after the date we are interested in) are layered together starting with now and working back in time to build a dict of data.

• Updates to Jira tickets will try not to notify users unless the title or body (summary or description) change. This requires that the bot Jira user be an administrator of the projects it is updating.

2021-01-08

• Rescanning changes:
  • Now you have the option to include closed pull requests.
  • Pull requests are fetched in full to ensure all the needed fields will be available.

2020-11-24

• The bot used to create a Jira issue to replace an issue that had been deleted. This interfered with rescanning, so the bot no longer does this. If a Jira issue mentioned in the bot comment has been deleted, it will not be recreated.

2020-10-29

• The number of lines added and deleted by a pull request are recorded in custom Jira fields.

2020-10-15

• Core Committer pull requests now start with a Jira status of "Waiting on Author" rather than "Open edX Community Review".

2020-09-23

• Draft pull requests start with a status of "Waiting on Author". Once the pull request is no longer a draft, the status is set to the initial status it would have originally had.

2020-08-08

• BUG: if the PR description was edited, the Jira issue status would be incorrectly reset to its initial value [OPENEDX-424]. This is now fixed.

2020-08-07

• When a core committer merges a pull request, the bot will add a comment pinging the committer's edX champions to let them know the merge has happened.
• BUG: previously the bot could clobber ad-hoc labels on Jira issues when it set its own labels. This is now fixed. The bot will preserve any labels it didn't make.
• Removed the code that managed webhooks in repos.
• Refactored some code that handles pull requests being closed, so now it operates on any change to the pull request. The behavior should be the same, except now if a pull request is closed or merged after the Jira issue has been manually deleted, the bot will create a new issue so that it can mark it Rejected or Merged.

2020-07-24

• BUG: previously, the bot might change GitHub labels and incorrectly drop ad-hoc labels that people had put on the pull request. This is now fixed.

2020-07-23

• GitHub very occasionally sends us a pull request event, but then serves us a 404 error when we ask it about the pull request. Now the bot will retry GET requests that return 404, to give GitHub a chance to get its act together.
• BUG: when a pull request was edited, the associated Jira issue would be reset to its initial status. This is now fixed: the Jira status is unchanged.

2020-07-21

• Previously, if an OSPR issue had been manually moved to BLENDED, and then the title of the pull request amended to have "[BD-xx]", the bot would try and fail to delete the moved issue. Now it understands the move, and doesn't try to delete the original issue. It also updates the issue with Blended information.

2020-07-20

• Changes to the title or description of a pull request are copied over to the associated Jira issue to keep them in sync.
• If a change to a pull request requires a different Jira issue, the old issue is deleted, and a new one made. For example, if a blended pull request doesn't have "[BD-xx]" in the title, an OSPR issue gets made initially. Now when the developer updates the title, the OSPR issue is deleted, and a new BLENDED issue is created for it.

2020-07-14

• The "expires_on" key in people.yaml is officially obsolete, and no longer interpreted.
• Some incorrect CLA logic was fixed. An entry in people.yaml with no "expires_on" key would be considered to have a signed CLA, even if the agreement was "none".

2020-07-02

• If an opened pull request has a CLA, then the bot will comment "jenkins ok to test" on it to get the tests started automatically.

2020-07-01

• Blended workflow: if "[BD-XX]" is found in the title of an opened pull request, then the Jira ticket will be in the BLENDED project, with links to the correct epic, etc.

2020-06-25

• Core committer logic has to be particular to specific repos, it's not a blanket right. Now "committer" isn't a simple boolean, it's an object with subkeys: "repos" is a list of repos the user can commit to, and "orgs" is a list of GitHub organizations the user can commit to (any repo).

2020-06-24

• Slight change to people.yaml schema: "internal:true" is used to indicate edX people (or Arbisoft). The "committer:true" flag indicates core committers.
• Core committer pull request handling: a different welcome message is used, OSPR issues are started in the "Open edX Community Review" status, and "core committer" GitHub and Jira labels are applied.

2020-06-19

• We used to have two GitHub webhooks. They have been combined. Only /github/hook-receiver is needed now. The obsolete /github/pr endpoint still exists just to log unneeded webhook action so we can fix the GitHub configuration.

2020-06-15

• Labels in GitHub repos are synchronized from repo-tools-data/labels.yaml before any labels are adjusted in the repo.
• Data read from repo-tools-data (people.yaml, label.yaml) is only cached for 15 minutes. It used to be until the bot was restarted.

2020-06-08

• Pull requests that need a CLA signed now create Jira tickets in the "Community Manager Review" status.

TODO

• Describe the different processes that are run on Heroku
• Describe how to access logs
• Make sure docs/ is up to date

openedx-webhooks's Issues

Update welcome message from OSPR bot with info and checklist for authors

Story

"As an author of an OSPR (open source pull request), I want to get a useful welcome message after posting my PR so that:

• I know where to look for relevant information about the OSPR process if/when I need it.
• I know which steps I need to complete to get my PR ready for engineering review.
• (maintained repos only) I know who to ping for engineering review."

Full description

One important aspect of reducing time-to-merge for OSPRs is to empower PR authors to manage more parts of the OSPR process themselves.

For that, they'll need:

• Easy access to information about the OSPR process as a whole that they can reference as necessary.
• A clear list of expectations that maintainers have of them, i.e., a checklist of steps/requirements that they must complete before their PRs can be considered ready for engineering review (including creating product proposals and feature tickets retroactively if their PRs include user-facing changes).

For maintained repositories, they should also get information about who to ping when they're done completing prerequisites for engineering review (e.g. @2u-aurora).

• Note: As determined by the maintenance working group, the OSPR bot should not ping maintainers of a repository directly in the welcome message. Instead, it should enclose mentions of maintainers in backticks: `@2u-aurora`.

Information about the maintainers of a repository can be extracted from its catalog-info.yml (example).

Completion criteria

• Draft new welcome message.
• Have the bot post the new welcome message instead of the existing one.
• Implement support for retrieving maintainer info and adding it to the welcome message (if available).

Behavioral specifications

N/A (doesn't change current behavior of OSPR bot, just its messaging)

Documentation updates & improvements criteria

Left to the assignee’s appreciation

Relevant repositories

• https://github.com/openedx/openedx-webhooks, and in particular:
  • https://github.com/openedx/openedx-webhooks/tree/master/openedx_webhooks/templates
  • https://github.com/openedx/openedx-webhooks/blob/master/openedx*webhooks/templates/github_community_pr*comment.md.j2

Review timeline

Flexible.

Notes

According to Feanil:

openedx-webhooks code is gnarly, feel free to ping #openedx-webhooks for help

Bot shouldn't say "we don't accept contributions" when someone closes a PR

See what happened here: edx/dbt-schema-builder#54

A PR was closed. The author no longer works for edX, so was considered external. The bot added a comment (twice?) saying that contributions aren't accepted. This is just extra noise, the contribution was being cancelled already.

When making Jira issues, add a "from github" label in Jira.

Update openedx-webhooks to not mention repo-tools-data

The datafiles for openedx-webhooks have moved to a new repo.

• Update the openedx-webhooks repo to not mention repo-tools as the source of the data.

Pull Org Data from the Export CSV instead of orgs.yaml

Context: openedx/repo-tools-data-schema#15

We have stopped updating orgs.yaml because it is now redundant with the data we're getting from the salesforce export file. We need to update the openedx-webhooks code to read orgs out of the salesforce export CSV instead of orgs.yaml or potentially in addition to orgs.yaml.

Feature: Support CLA checks in merge queues

GitHub has added a feature called "merge queues" (docs) that feel like a very good solution to some of the race condition issues with automatic PRs I've encountered on https://github.com/openedx/openedx-translations/

When adding a PR to a merge queue, it will try to get the CLA status, but the hook will never return anything

I have added "Merge groups" to the events that trigger openedx-webhooks, but I have not started looking into what changes need to happen in code here to allow running the CLA check for PRs in a queue.

Acceptance Criteria:

• When a PR is added to a merge queue, the CLA check runs and correctly reports the status.

Deprecate Label Creation in openedx-webhooks

The openedx-webhooks repo creates labels for any repos that it receives PRs from. This is useful but now that we're also using a lot of repos as a way of holding related github issues, we need something a bit more robust.

openedx-unsupported/terraform-github#44 adds a robust way to create labels on all repos within the organization.

To reduce the chances of confusion and dueling automation, we should only have one way to manage labels we want to systematically create for all the repos in the org. Since the terraform-github check is more robust, we should move forward with that and migrate any labels that openedx-webhooks creates to that script. This includes the labels defined in https://github.com/openedx/openedx-webhooks-data/blob/main/labels.yaml as well as any defined in the openedx-webhooks code.

Tasks

Beta Give feedback

1. feat: add a bunch of organization-wide labels openedx-unsupported/terraform-github#61
   +0
2. https://github.com/openedx/openedx-webhooks-data/pull/191
3. feat: Make it valid for ORGS, LABELS, and PEOPLE to be empty repo-tools-data-schema#28
   +0
4. feat!: stop syncing labels from labels.yaml #245
   +0
5. feat!: remove validate_labels repo-tools-data-schema#30
   +0
6. https://github.com/openedx/openedx-webhooks-data/pull/203

Followup

Beta Give feedback

1. Regularly Check that the repo checks are passing on all the repos repo-tools#385
   +0
2. [DEPR]: Make OSPR bot unaware of which repositories CCs can merge to #227
   depr +1
   
3. Once blended and open-source-contribution are the only labels being added by openedx-webhooks, consider if there's a simpler way to handle that label creation, or if we're happy how openedx-webhooks is doing that.

Use ownership data to decide whether to make Jira issues

The conditionalization from #189 should be based on OEP-55 official ownership data.

"product review" label gets deleted

(From a Slack discussion: https://openedx.slack.com/archives/C03R320AFJP/p1672944142897759 or 2U: https://twou.slack.com/archives/C03R320AFJP/p1672944142897759)

The bot has a set of labels intended to mirror Jira ticket statuses:

GITHUB_STATUS_LABELS = {
    "architecture review",
    "awaiting prioritization",
    "blocked by other work",
    "changes requested",
    "community manager review",
    "engineering review",
    "merged",
    "needs triage",
    "open edx community review",
    "product review",
    "rejected",
    "waiting on author",
}

When the bot can't find the status from Jira (as it can't now because the connection isn't re-established yet), it doesn't know the status, and so removes any of these labels. When someone manually adds a label, the bot removes it during it's next pass over the pull request.

More Correctly fix paging issues with getting the latest PR commit

#216 was a quick fix for PRs that have more than 30 commits. But we should fix this part of the code to be more resilient now that CLA checks are required on all PRs.

One thing to think about: Instead of getting all the commits, can we just take the head sha from the Pull Request endpoint

Do something about the OSPR survey

Background

OSPRs, when closed or merged, ask the author to fill out a survey: openedx/edx-platform#33204 (comment)

• Fortunately, we still have access to the data, and it's still coming in (private Google Sheets link): https://docs.google.com/spreadsheets/d/15qkfy2Df2TjggKg1pYExZzaEyhN0ac3_Wx36rogUYJM/edit#gid=104229719
• As of right now, we're not using the data.
• We don't have admin access to the form itself. It's probably in the retired edX Google account. @nedbat confirmed that 2U IT could not recover it.

Questions & tasks

• Do we (Axim, or maybe a working group) want to keep doing the survey?
  • If yes ->
    • Re-make the survey in Axim's Google account, and then update the OSPR bot with the new survey link.
    • Think: How should we use the results?
    • Note: if anyone except Axim can see the survey data, we should make that clear in the survey.
  • If no ->
    • Remove the survey functionality from the bot. We had started doing this but stopped once we found the survey data

[DEPR]: Make OSPR bot unaware of which repositories CCs can merge to

Proposal Date

2023-02-03

Target Ticket Acceptance Date

2023-03-13

Earliest Open edX Named Release Without This Functionality

March/April 2023 (not tied to a release)

Rationale

Background

This application looks at a people.yml file for community data. Here's a sample people.yml. If you have access, here's the real people.yml. The only use people.yml currently is the list of repos that Coding Core Contributors (aka Core Committers) have access to. When a CC opens a PR, it uses this data to decide:

• do I add the core committer label or not?
• do I show a special CC OSPR message or a normal OSPR message?

Problem

Since it is no longer the source of truth for CLA data (that's salesforce-export.csv, now), people.yml is liable to falling very out of date. At time of writing this ticket, its last sync with reality was 3+ months ago, via manual update.

There are already two other places where CC repos need to updated: GitHub permissions and the CC wiki page. We (tCRIL) are not keen on keeping the repos in people.yml up-to-date unless there is compelling reason for us to do so.

Removal

#246

(this section needs updating)

• Delete people.yml: https://github.com/openedx/openedx-webhooks-data/blob/main/people.yaml.

• Remove the data structures, utility functions, and tests for people.yml from https://github.com/openedx/openedx-webhooks.

• Remove all logic from https://github.com/openedx/openedx-webhooks that pertains to Core Committer access. The only thing the bot should care about is whether a CLA has been signed.

• Remove the core committer label from labels.yml

• Optional: Reword the generic PR message:

  Thanks for the pull request, @...! Please note that it may take us up to several weeks or months to complete a review and merge your PR.

  to something that makes sense for core contributors. For example, it might say:

  Thanks for the pull request, @...! Please note that it may take us up to several weeks or months to complete a review and merge your PR. Of course, Core Contributors to this repository may merge PRs themselves
  ...

Replacement

Core Contributors can manually add the core contributor label to their PRs with label: core contributor.

There is no other planned replacement. However, here are some alternatives if we wanted to keep the bot aware of core committer repo access:

• Have the OSPR bot use the GitHub permissions API rather than people.yml to determine whether someone has write access to a repository.
• Automatically sync CC repository access information into people.yml. For example, maybe we could store CC access information in Salesforce, and then percolate down to both GitHub and this application. I don't really know how this would be done.
• Have tCRIL on-call manually keep people.yml in sync with GitHub and the wiki.

Deprecation

No response

Migration

Before accepting this deprecation

Get consensus from @feanil , @sarina , @nedbat , and @mphilbrick211, and @e0d that this change makes sense. I am aware that this change might not make sense, but I felt like a DEPR ticket was the best way to discuss it.

After accepting the deprecation

Inform CCs in Slack and on the forums about the workflow changes. Ned could send an email to 2U folks to inform them that PRs from CCs and non-CCs will now look identical.

Additional Info

No response

Update CC PR message to reflect decoupling?

Rather than referring CCs to their edX champions for merge coordination, the bot should point them to the #cc- Slack channels and/or a page of docs with the most up-to-date guidance for CC merges.

If a repo isn't configured for `jira:company` labels, add a comment saying so

Fix openedx-webhooks staging

openedx-webhooks-staging is throwing an exception when receiving a PR webhook. Debug the issue and fix it so that we can use stage for testing again.

2022-10-04T15:54:44.946309+00:00 app[worker.1]: [2022-10-04 15:54:44,945: ERROR/ForkPoolWorker-8] openedx_webhooks.tasks.github.pull_request_changed_task[3a301fdd-0d04-4e03-bef8-a85f6c87af97]: Couldn't pull_request_changed_task
2022-10-04T15:54:44.946341+00:00 app[worker.1]: Traceback (most recent call last):
2022-10-04T15:54:44.946349+00:00 app[worker.1]: File "/app/openedx_webhooks/tasks/github.py", line 34, in pull_request_changed_task
2022-10-04T15:54:44.946349+00:00 app[worker.1]: ret = pull_request_changed(pull_request)
2022-10-04T15:54:44.946349+00:00 app[worker.1]: File "/app/openedx_webhooks/tasks/github.py", line 71, in pull_request_changed
2022-10-04T15:54:44.946350+00:00 app[worker.1]: fixer.fix()
2022-10-04T15:54:44.946352+00:00 app[worker.1]: File "/app/openedx_webhooks/tasks/pr_tracking.py", line 374, in fix
2022-10-04T15:54:44.946353+00:00 app[worker.1]: self.fix_ospr()
2022-10-04T15:54:44.946353+00:00 app[worker.1]: File "/app/openedx_webhooks/tasks/pr_tracking.py", line 422, in fix_ospr
2022-10-04T15:54:44.946353+00:00 app[worker.1]: self._make_jira_issue()
2022-10-04T15:54:44.946354+00:00 app[worker.1]: File "/app/openedx_webhooks/tasks/pr_tracking.py", line 482, in _make_jira_issue
2022-10-04T15:54:44.946354+00:00 app[worker.1]: new_issue = self.actions.create_ospr_issue(
2022-10-04T15:54:44.946354+00:00 app[worker.1]: File "/app/openedx_webhooks/tasks/pr_tracking.py", line 799, in create_ospr_issue
2022-10-04T15:54:44.946355+00:00 app[worker.1]: custom_fields = get_jira_custom_fields(get_jira_session())
2022-10-04T15:54:44.946355+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/cachetools/func.py", line 62, in wrapper
2022-10-04T15:54:44.946356+00:00 app[worker.1]: v = func(*args, **kwargs)
2022-10-04T15:54:44.946356+00:00 app[worker.1]: File "/app/openedx_webhooks/utils.py", line 301, in get_jira_custom_fields
2022-10-04T15:54:44.946356+00:00 app[worker.1]: field_resp.raise_for_status()
2022-10-04T15:54:44.946356+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/requests/models.py", line 1022, in raise_for_status
2022-10-04T15:54:44.946357+00:00 app[worker.1]: raise HTTPError(http_error_msg, response=self)
2022-10-04T15:54:44.946357+00:00 app[worker.1]: requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://nosuchserver.atlassian.net/rest/api/2/field
2022-10-04T15:54:44.959753+00:00 app[worker.1]: [2022-10-04 15:54:44,959: INFO/ForkPoolWorker-8] Task openedx_webhooks.tasks.github.pull_request_changed_task[3a301fdd-0d04-4e03-bef8-a85f6c87af97] succeeded in 2.5614383260253817s: 'Traceback (most recent call last):
2022-10-04T15:54:44.959754+00:00 app[worker.1]: File "/app/openedx_webhooks/__init__.py", line 95, in __call__
2022-10-04T15:54:44.959755+00:00 app[worker.1]: return self.run(*args, **kwargs)
2022-10-04T15:54:44.959755+00:00 app[worker.1]: File "/app/openedx_webhooks/tasks/github.py", line 34, in pull_request_changed_task
2022-10-04T15:54:44.959756+00:00 app[worker.1]: ret = pull_request_changed(pull_request)
2022-10-04T15:54:44.959756+00:00 app[worker.1]: File "/app/openedx_webhooks/tasks/github.py", line 71, in pull_request_changed
2022-10-04T15:54:44.959756+00:00 app[worker.1]: fixer.fix()
2022-10-04T15:54:44.959757+00:00 app[worker.1]: File "/app/openedx_webhooks/tasks/pr_tracking.py", line 374, in fix
2022-10-04T15:54:44.959758+00:00 app[worker.1]: self.fix_ospr()
2022-10-04T15:54:44.959758+00:00 app[worker.1]: File "/app/openedx_webhooks/tasks/pr_tracking.py", line 422, in fix_ospr
2022-10-04T15:54:44.959758+00:00 app[worker.1]: self._make_jira_issue()
2022-10-04T15:54:44.959759+00:00 app[worker.1]: File "/app/openedx_webhooks/tasks/pr_tracking.py", line 482, in _make_jira_issue
2022-10-04T15:54:44.959759+00:00 app[worker.1]: new_issue = self.actions.create_ospr_issue(
2022-10-04T15:54:44.959759+00:00 app[worker.1]: File "/app/openedx_webhooks/tasks/pr_tracking.py", line 799, in create_ospr_issue
2022-10-04T15:54:44.959760+00:00 app[worker.1]: custom_fields = get_jira_custom_fields(get_jira_session())
2022-10-04T15:54:44.959760+00:00 app[worker.1]: File "/app/.heroku/python/lib/python3.8/site-packages/cachetools/func.py", line 62, in wrapper
2022-10-04T15:54:44.959760+00:00 app[worker.1]: v = func(*args, **kwargs)
2022-10-04T15:54:44.959761+00:00 app[worker.1]: File "/app/openedx_webhooks/utils.py", line 301, in...'

Reproducing the error:

• heroku logs --tail -a openedx-webhooks-staging
• Edit the title of nedbat/webhook-testing#121
• See the exception occur.

Publish CC Data In salesforce-export.csv

See #200 and #227 for some background.

Things we probably need to do as a part of this work:

• Add a new fields to salesforce to track CC related data.
  • CCs are tracked at the repo level and at the org level so we may need both but might be able to get away with just the repo field to start with.
• Add/Update documentation around addition and removal of CCs and what to do in salesforce at those times.
• Update the export so that the new fields are being publish to the CSV.
• Update openedx-webhooks to pull this new data from the CSV.
• Update openedx-webhooks to pull org data from the CSV and drop reading data from orgs.yaml
• Add back core-contributor-specific OSPR bot comments. This feature was removed in #251, since the OSPR bot was relying on an outdated data source (people.yaml) to leave such comments.
• Remove the people.yaml and orgs.yaml files from the openedx-webhooks-data repo.
• Figure out if we still need repo-tools-data-schema and update it as necessary to work with the new data setup.
  • With validation happening in salesforce we may not need repo-tools-data-schema.

Update OSPR bot for the Atlassian migration

• Remove most of the hard-coding of openedx.atlassian.net
• Change the JIRA server to 2u-internal.atlassian.net
• Make JIRA optional (JIRA_SERVER="" turns off Jira activity)

For using GitHub projects:

• Add pull requests to a GitHub project
• Add blended pull requests to a separate GitHub project

Openedx-webhooks graphql is deprecated and needs to be updated

The projects Graphql Endpoint we're using looks like it's no longer supported and we need to update to the new v2 api for all our graphql queries.

2022-11-02T14:41:46.627901+00:00 app[worker.1]: [2022-11-02 14:41:46,627: ERROR/ForkPoolWorker-8] openedx_webhooks.tasks.github.pull_request_changed_task[5e9d5b86-626f-4508-a139-40cc18626b4a]: Couldn't pull_request_changed_task
2022-11-02T14:41:46.627918+00:00 app[worker.1]: Traceback (most recent call last):
2022-11-02T14:41:46.627919+00:00 app[worker.1]: File "/app/openedx_webhooks/tasks/github.py", line 34, in pull_request_changed_task
2022-11-02T14:41:46.627919+00:00 app[worker.1]: ret = pull_request_changed(pull_request)
2022-11-02T14:41:46.627920+00:00 app[worker.1]: File "/app/openedx_webhooks/tasks/github.py", line 69, in pull_request_changed
2022-11-02T14:41:46.627920+00:00 app[worker.1]: current = current_support_state(pr)
2022-11-02T14:41:46.627920+00:00 app[worker.1]: File "/app/openedx_webhooks/tasks/pr_tracking.py", line 221, in current_support_state
2022-11-02T14:41:46.627921+00:00 app[worker.1]: current.github_projects = set(pull_request_projects(pr))
2022-11-02T14:41:46.627921+00:00 app[worker.1]: File "/app/openedx_webhooks/gh_projects.py", line 54, in pull_request_projects
2022-11-02T14:41:46.627922+00:00 app[worker.1]: data = graphql_query(query=PROJECTS_FOR_PR, variables=variables)
2022-11-02T14:41:46.627922+00:00 app[worker.1]: File "/app/openedx_webhooks/utils.py", line 243, in graphql_query
2022-11-02T14:41:46.627923+00:00 app[worker.1]: raise Exception(f"GraphQL error: {returned!r}")
2022-11-02T14:41:46.627925+00:00 app[worker.1]: Exception: GraphQL error: {'data': {'repository': {'pullRequest': None}}, 'errors': [{'type': 'NOT_FOUND', 'path': ['repository', 'pullRequest', 'projectNextItems'], 'locations': [{'line': 8, 'column': 7}], 'message': 'The `ProjectNext` API is deprecated in favour of the more capable `ProjectV2` API. Follow the ProjectV2 guide at https://github.blog/changelog/2022-06-23-the-new-github-issues-june-23rd-update/, to find a suitable replacement.'}]}

Update openedx-webhooks from heroku-18 stack to heroku-22

This app is using the Heroku-18 stack which is deprecated. It will End-of-life in April 2023. After that it will stop receiving security updates so at some point we should just update to the latest stack.

https://devcenter.heroku.com/articles/upgrading-to-the-latest-stack
https://help.heroku.com/X5OE6BCA/heroku-18-end-of-life-faq

Relatedly, we're currently running python runtime 3.8.12 but 3.8.14 is available, we should also upgrade this.

Auto-add pull requests and issues to GitHub projects

Task: update the bot to automatically add new pull requests and issues to projects.

Back story: we did this with GitHub Actions (for example: openedx/credentials-themes#559), but there are two problems that would be intricate to solve:

• it adds draft pull requests to the project
• it will not add pull requests from other forks
  • issue about this: actions/add-to-project#163
  • that points to a security concern: https://securitylab.github.com/research/github-actions-preventing-pwn-requests

The bot should do this work. We need to design how it should work:

• The bot currently only looks at pull requests, so it will only add pull requests, not issues.
  • Issues can be handled by the shared GitHub action: add-issue-to-a-project.yml
• Should the bot only add "external contribution" PRs (OSPRs) to the project, or all pull requests?
  • I'm assuming it should be only OSPRs.
• Draft pull requests: should it never add them, always add them, or do different projects have strong opinions that differ?
  • It will not add draft pull requests to project.
  • It will add a pull request to a project if it transitions from draft to ready.
  • "Draft" is determined the same way the bot already does:
    • Either the pull request is literally a draft (in GitHub terms),
    • or it has the words "wip" or "WIP" in the title.
• Configuration: the project to add to will be indicated by metadata in the catalog-info.yml file.
  • Do we have a reason now to need to add to more than one project?
    • It will support multiple projects (easy enough to do right the first time)
• Should it ever remove items from a project? For example, if a pull request is switched to draft mode?
  • For the first iteration, we won't remove from projects. Project automation can remove closed pull requests from the project.

Upgrade openedx-webhooks redis version

openedx-webhooks uses redis for talking to its workers. The versions on both stage and production need upgrades.

Stage is running on Redis 6
Production is running on Redis 4

Note on TLS

Because of the size of the prod redis, upon upgrade to redis 6, it will require that we connect to redis over TLS. This does not happen on stage because on stage, we are using a dev tier redis instance which does not require TLS(but does offer it)

Before we upgrade either, we need to ensure that there will be no issues connecting with redis over TLS in stage.

Approximate Runbook

1. Update the code to use the REDIS_TLS_URL instead of the REDIS_URL in Stage to ensure we won't have any issues with the TLS URL. This should fallback to REDIS_URL if the REDIS_TLS_URL is not set.
2. Deploy to staging and verify everything is working.
3. Upgrade stage to the latest redis version(7.x right now)
4. Verify that things still work(example)
5. Deploy the change to production.
6. Do the redis maintenance in production
7. Ensure that everything is still working as expected.

References

• https://devcenter.heroku.com/articles/heroku-redis-version-upgrade
• https://devcenter.heroku.com/articles/connecting-heroku-redis#connecting-in-python

Note from Heroku

Redis (4.0.14) will not be supported in less than a month.

Upgrading from 4.0.14 to 7.0.11 may break your app as Redis 6 requires TLS. Take action now! See Dev Center for more info on upgrading to Redis 6 in the following articles: - Heroku Redis Version Upgrade - Connecting to Heroku Redis

If you are using stunnel for secure connection, this is no longer used with versions from 6.0 and must be removed. Follow this guide for connecting to a Redis instance.

We recommend upgrading your Redis database to the latest version supported by Heroku, which is currently 7.0.11. We perform heavy testing and validation work on all new Redis releases, including testing the upgrade procedure, and are careful to only recommend versions that we trust.

You can upgrade your Redis database to the latest default version by using heroku redis:upgrade command. You can read more about this procedure in our Dev Center article.

If you do not take action, we will schedule an upgrade to your Redis database on 30 May, 2023.

The upgrade will create a maintenance that will run during your next maintenance window. You can trigger it yourself beforehand by using the maintenance cli.

A `jira:company` label should work on issues as well as pull requests

Add a maintenance Label to all Repos so that we can tag maintenance work that needs to be done.

Get bot credentials for 2u-internal.atlassian.net

The bot can do Jira work, but only if it has credentials for 2u-internal.atlassian.net.

Figure out how we're gonna use people.yaml and salesforce-export.csv

Right now we have two ways of tracking people in openedx-webhooks, with people.yaml and salesforce-export.csv . people.yaml is getting pretty out of date.

Figure out what peopl.yaml still being used for and get rid of it if possible.

If we're using it to see which contributors are CCs, we can instead look at what repos they have write permission to and what org they're from to determine that.

Don't add an "open source contribution" label if a PR is being closed

This happens when we close an ex-edXer pull request. But we should never do it, because if the label should have been on the PR, it would have been added when it was opened.

Add a label when the CLA is missing

If a PR author has no CLA, add a label to the pull request. When the CLA is signed, and the check passes, remove the label. There won't be a "has cla" label, just a "needs cla" label.

Bot double-commenting

(From #273)

BTW, it comments twice because the "Close with comment" button causes two events to fire (comment created and pull request closed). They are both sent to the webhook at the same time, so two requests are handled, and each sees the pull request with no comment on it, so both add a comment.

The comment should be ignored if the bot created it, and I thought the bot already did ignore those?

Use ownership data to determine whether something is an "ospr"

Today the bot decides a PR is an "ospr" if it is authored by someone outside of 2U. This has to be updated: a PR is an OSPR if it is in a 2U-maintained repo, and the author is outside of 2U.

Conditionally create JIRA issues for pull requests

Some teams want JIRA issues, some don't. We need a way to decide and direct the bot.

Bot is labelling and commenting on PRs in private repos in the edx GH org

The bot is adding comments to pull requests in private repos in the edx organization, and labelling them as open source contributions if the author isn't in Salesforce. (example, though obviously, it's a private link: https://github.com/edx/bisupport_queries/pull/47)

Create a docker image for local testing

It could be a good addition for testing the bot locally.

Update the webhooks bot message to tell people how to get help

Note: Different teams for different problems. eg. @test-approval vs @cla-problems

Create an endpoint to purposefully cause a Python error in Celery

This is to test error reporting.

Figure out the future of the OSPR Bot

Decoupling questions

In light of the edX-tCRIL decoupling and increasing distribution of code ownership...

Who will depend on the bot, and for what?

• edX: creating and linking Jira tickets for each OSPR on the Jira project of the owning team.
• tCRIL: putting OSPRs on our team project board for code we own?
• others firms: pushing to their preferred issue tracking system when a PR is opened on owned code?
• tCRIL: automating the CLA flow by hooking into Salesforce.
• anything else?

Who will maintain the bot?

• edX Arch-BOM?
• tCRIL?

Where do bot-related issues go?

• edX Jira, in the BOM project?
• this tcril-engineering GitHub repo?

Other issues

Cataloging these here until we have a proper place for issues.

Personal access token vs App

Until recently, the OSPR bot was implemented as an OAuth App in GitHub. Because the openedx GitHub org is configured not to allow arbitrary OAuth Application access, we either needed to
(i) do the legwork to make the OSPR bot an approved OAuth App, or
(ii) change the bot to use a personal access token from the openedx-webhooks account, or
(iii) change the openedx GitHub org to allow arbitrary OAuth application access.

We took approach (ii).

We have some concern that because this is not the "proper" way to integrate with GitHub (they technically have a one-user-account-per-human policy) that this may expose us to being rate-limited more aggressively than if it were a proper OAuth App or GitHub App (which, mind you, are different things). For now, we've decided to proceed with a personal access token, with Ned keeping an eye on the logs to see if we're approaching or hitting a rate limit.

In the future, though, it would probably be prudent to turn the OSPR bot into an approved GitHub App or OAuth App.