packit / agile

This repository contains all the information and tooling for the agile practices of the Packit team.

Packit Agile Repository

This repository contains all the information and tooling for the agile practices of the Packit team.

agile's Issues

Kanban Lead

• Watch Jira board is up to date

• Announce the new roles on Monday in the chat.

• Be a moderator for stand-ups, refinements, architecture and retrospective meetings.

• Stand-ups:

  • Think of a question and ask it at the beginning
  • Run standup
  • Ask about discussion topics after everyone presents their update
  • On Monday, ask about blocked cards, in-progress cards and priorities in the Refinement column - make sure that everything is up to date!

• If it's an odd week (check with date '+%V')

  • Create a retro board (use this template in Miro).
  • Copy actions/improvements from previous retro board, so we can check whether we did everything we wanted to.
  • Attach the link to the retrospective event in our Google calendar.
  • On retro ask about EPICs and whether we're working on the right ones

• Manual: An overview of how Packit Team kanbanz

CI Hacker

This is a role card for the one responsible throughout the sprint for keeping the CI green, that is to look for and drive the resolution of systematic CI failures.

It can happen that a CI system has an outage. For problems related to Zuul, please reach out to the team at #rhos-ops internal IRC channel.

Service Guru

Every Monday:

• (On stand-up) Check with the team whether deploying new changes from main branches is safe.
• Make sure stage deployment is OK, check:
  • Openshift cluster
  • Sentry
  • Dashboard

If all is fine and things are good to go:

• Create a blog post about what's been changed in packit & packit-service main branches since their stable branch. Use scripts/move_stable.py github-query to get a link to all the PRs from the past week which are marked to have release notes. You can copy those for a start, and check with other team members to make sure nothing was left out. You can also use the blogpost template provided by make move-stable. A blog post should meet the following requirements:
  • This is meant to be read by general public - make it easy to read for everyone.
  • Focus on new features, notable bug fixes, documentation updates, UX improvements.
  • Don't talk about internal things (e.g. most ogr/specfile changes), refactoring, CI changes etc.
  • Write it in a way so that people with a little knowledge of the project can get a clue what is the change about:
    • NO: packit's behaviour in dealing with spec files was improved
    • YES: packit can now find a spec file inside a bottle of rum which is very handy if you're thirsty
• Move content from main branches to stable branches.
  • From a clone of the deployment repo run make move-stable command. Your local ssh keys have to be associated with your github user to be able to use this script.
  • Go to the Actions tab (example of the respective repos) and make sure the images from the stable branches are built & pushed to Quay.io.
• If there were any changes in the Kubernetes/Openshift objects or in the vault/Packit/! Changelog ! that have not been already applied to prod (ask the author of the change), do so with DEPLOYMENT=prod make deploy.
• If you're doing this AFTER new images have already been built (see previous point) you should run DEPLOYMENT=prod make import-images first (or after), to avoid possible Error: ImagePullBackOff.

Monday night & Tuesday morning:

• Newer images are automatically imported from Quay.io and pods are recreated from the new images.

On Tuesday:

• Make sure prod runs as expected - revert the deployment if all jobs fail or there is significant regression in the functionality.
• Merge/publish your blog post.

Service Guru

Every Monday:

• (On stand-up) Check with the team whether deploying new changes from main branches is safe.
• Make sure stage deployment is OK, check:
  • Openshift cluster
  • Sentry
  • Dashboard

If all is fine and things are good to go:

• Create a blog post about what's been changed in packit & packit-service main branches since their stable branch. Use scripts/move_stable.py github-query to get a link to all the PRs from the past week which are marked to have release notes. You can copy those for a start, and check with other team members to make sure nothing was left out. You can also use the blogpost template provided by make move-stable. A blog post should meet the following requirements:
  • This is meant to be read by general public - make it easy to read for everyone.
  • Focus on new features, notable bug fixes, documentation updates, UX improvements.
  • Don't talk about internal things (e.g. most ogr/specfile changes), refactoring, CI changes etc.
  • Write it in a way so that people with a little knowledge of the project can get a clue what is the change about:
    • NO: packit's behaviour in dealing with spec files was improved
    • YES: packit can now find a spec file inside a bottle of rum which is very handy if you're thirsty
• Move content from main branches to stable branches.
  • From a clone of the deployment repo run make move-stable command. Your local ssh keys have to be associated with your github user to be able to use this script.
  • Go to the Actions tab (example of the respective repos) and make sure the images from the stable branches are built & pushed to Quay.io.
• If there were any changes in the Kubernetes/Openshift objects or in the vault/Packit/! Changelog ! that have not been already applied to prod (ask the author of the change), do so with DEPLOYMENT=prod make deploy.
• If you're doing this AFTER new images have already been built (see previous point) you should run DEPLOYMENT=prod make import-images first (or after), to avoid possible Error: ImagePullBackOff.

Monday night & Tuesday morning:

• Newer images are automatically imported from Quay.io and pods are recreated from the new images.

On Tuesday:

• Make sure prod runs as expected - revert the deployment if all jobs fail or there is significant regression in the functionality.
• Merge/publish your blog post.

Chief of Monitors

Watch Sentry for issues:

• Fix recurring alerts
  • Deploy fixes to production for issues that cause major disruption or complete downtime
  • Verify no alerts are being triggered any more
• More complicated issues should be brought to the team and prioritized correctly.
• Clean all the past alerts so we have easy to navigate dashboard
• Link github and sentry issues

React to alerts arriving through email and check the SLO monitoring page (Packit section) and respond to the email so others know what is happening.

Watch our other two Grafana dashboards as well:

• Celery Monitoring
• Packit Monitoring

Chief of Monitors

Watch Sentry for issues:

• Fix recurring alerts
  • Deploy fixes to production for issues that cause major disruption or complete downtime
  • Verify no alerts are being triggered any more
• More complicated issues should be brought to the team and prioritized correctly.
• Clean all the past alerts so we have easy to navigate dashboard
• Link github and sentry issues

React to alerts arriving through email and check the SLO monitoring page (Packit section).

Watch our other two Grafana dashboards as well:

• Celery Monitoring
• Packit Monitoring

CI Hacker

This is a role card for the one responsible throughout the sprint for keeping the CI green, that is to look for and drive the resolution of systematic CI failures.

It can happen that a CI system has an outage. For problems related to Zuul, please reach out to the team at #rhos-ops internal IRC channel.

CI Hacker

This is a role card for the one responsible throughout the sprint for keeping the CI green, that is to look for and drive the resolution of systematic CI failures.

It can happen that a CI system has an outage. For problems related to Zuul, please reach out to the team at #rhos-ops internal IRC channel.

Community Shepherd

• Be present in #packit:fedora.im, #packit:libera.chat, and Packit gchat channel to answer community questions or ping some other team member to help answer those questions.

• Keep an eye on incoming issues opened by our users. The unprocessed issues can be found in the new column of our Kanban board. React to them trying to clarify them, if needed, or ask the team to help give a meaningful reaction to them. Use appropriate labels and move the issue to the backlog column once the issue is clear.

• Help new users trying to enable Packit Service to configure their repositories.

• Watch out for incoming issues in github.com/packit/notifications if someone needs help with the automatic approval process. Delete old, unanswered issues, or the ones which look like spam.

• Watch auto-packit-shared-infra mailing list for planned cluster upgrades. Once one is scheduled:

  • Create a new status post
  • Write to the #packit channel to inform everyone about the possible service disruption during the upgrade
  • Work with the Chief of Monitors to handle sentry alerts that may happen during the upgrade

Community Shepherd

• Be present in #packit:fedora.im, #packit:libera.chat, and Packit gchat channel to answer community questions or ping some other team member to help answer those questions.

• Keep an eye on incoming issues opened by our users. The unprocessed issues can be found in the new column of our Kanban board. React to them trying to clarify them, if needed, or ask the team to help give a meaningful reaction to them. Use appropriate labels and move the issue to the backlog column once the issue is clear.

• Help new users trying to enable Packit Service to configure their repositories.

• Watch out for incoming issues in github.com/packit/notifications if someone needs help with the automatic approval process. Delete old, unanswered issues, or the ones which look like spam.

• Watch auto-packit-shared-infra mailing list for planned cluster upgrades. Once one is scheduled:

  • Create a new status post
  • Write to the #packit channel to inform everyone about the possible service disruption during the upgrade
  • Work with the Chief of Monitors to handle sentry alerts that may happen during the upgrade

Service Guru

Every Monday:

• (On stand-up) Check with the team whether deploying new changes from main branches is safe.
• Make sure stage deployment is OK, check:
  • Openshift cluster
  • Sentry
  • Dashboard

If all is fine and things are good to go:

• Create a blog post about what's been changed in packit & packit-service main branches since their stable branch. Use scripts/move_stable.py github-query to get a link to all the PRs from the past week which are marked to have release notes. You can copy those for a start, and check with other team members to make sure nothing was left out. You can also use the blogpost template provided by make move-stable. A blog post should meet the following requirements:
  • This is meant to be read by general public - make it easy to read for everyone.
  • Focus on new features, notable bug fixes, documentation updates, UX improvements.
  • Don't talk about internal things, refactoring, CI changes etc.
  • Write it in a way so that people with a little knowledge of the project can get a clue what is the change about:
    • NO: packit's behaviour in dealing with spec files was improved
    • YES: packit can now find a spec file inside a bottle of rum which is very handy if you're thirsty
• Move content from main branches to stable branches.
  • From a clone of the deployment repo run make move-stable command. Your local ssh keys have to be associated with your github user to be able to use this script.
  • Go to the Actions tab (example of the respective repos) and make sure the images from the stable branches are built & pushed to Quay.io.
• If there were any changes in the Kubernetes/Openshift objects or in the vault/Packit/! Changelog ! that have not been already applied to prod (ask the author of the change), do so with DEPLOYMENT=prod make deploy.
• If you're doing this AFTER new images have already been built (see previous point) you should run DEPLOYMENT=prod make import-images first (or after), to avoid possible Error: ImagePullBackOff.

Monday night & Tuesday morning:

• Newer images are automatically imported from Quay.io and pods are recreated from the new images.

On Tuesday:

• Make sure prod runs as expected - revert the deployment if all jobs fail or there is significant regression in the functionality.
• Merge/publish your blog post.

Service Guru

Every Monday:

• (On stand-up) Check with the team whether deploying new changes from main branches is safe.
• Make sure stage deployment is OK, check:
  • Openshift cluster
  • Sentry
  • Dashboard

If all is fine and things are good to go:

• Create a blog post about what's been changed in packit & packit-service main branches since their stable branch. Use scripts/move_stable.py github-query to get a link to all the PRs from the past week which are marked to have release notes. You can copy those for a start, and check with other team members to make sure nothing was left out. You can also use the blogpost template provided by make move-stable. A blog post should meet the following requirements:
  • This is meant to be read by general public - make it easy to read for everyone.
  • Focus on new features, notable bug fixes, documentation updates, UX improvements.
  • Don't talk about internal things (e.g. most ogr/specfile changes), refactoring, CI changes etc.
  • Write it in a way so that people with a little knowledge of the project can get a clue what is the change about:
    • NO: packit's behaviour in dealing with spec files was improved
    • YES: packit can now find a spec file inside a bottle of rum which is very handy if you're thirsty
• Move content from main branches to stable branches.
  • From a clone of the deployment repo run make move-stable command. Your local ssh keys have to be associated with your github user to be able to use this script.
  • Go to the Actions tab (example of the respective repos) and make sure the images from the stable branches are built & pushed to Quay.io.
• If there were any changes in the Kubernetes/Openshift objects or in the vault/Packit/! Changelog ! that have not been already applied to prod (ask the author of the change), do so with DEPLOYMENT=prod make deploy.
• If you're doing this AFTER new images have already been built (see previous point) you should run DEPLOYMENT=prod make import-images first (or after), to avoid possible Error: ImagePullBackOff.

Monday night & Tuesday morning:

• Newer images are automatically imported from Quay.io and pods are recreated from the new images.

On Tuesday:

• Make sure prod runs as expected - revert the deployment if all jobs fail or there is significant regression in the functionality.
• Merge/publish your blog post.

Community Shepherd

• Be present in #packit:fedora.im, #packit:libera.chat, and Packit gchat channel to answer community questions or ping some other team member to help answer those questions.

• Keep an eye on incoming issues opened by our users. The unprocessed issues can be found in the new column of our Kanban board. React to them trying to clarify them, if needed, or ask the team to help give a meaningful reaction to them. Use appropriate labels and move the issue to the backlog column once the issue is clear.

• Help new users trying to enable Packit Service to configure their repositories.

• Watch out for incoming issues in github.com/packit/notifications if someone needs help with the automatic approval process. Delete old, unanswered issues, or the ones which look like spam.

• Watch auto-packit-shared-infra mailing list for planned cluster upgrades. Once one is scheduled:

  • Create a new status post
  • Write to the #packit channel to inform everyone about the possible service disruption during the upgrade
  • Work with the Chief of Monitors to handle sentry alerts that may happen during the upgrade

CI Hacker

This is a role card for the one responsible throughout the sprint for keeping the CI green, that is to look for and drive the resolution of systematic CI failures.

It can happen that a CI system has an outage. For problems related to Zuul, please reach out to the team at #rhos-ops internal IRC channel.

Chief of Monitors

Watch Sentry for issues:

• Fix recurring alerts
  • Deploy fixes to production for issues that cause major disruption or complete downtime
  • Verify no alerts are being triggered any more
• More complicated issues should be brought to the team and prioritized correctly.
• Clean all the past alerts so we have easy to navigate dashboard
• Link github and sentry issues

React to alerts arriving through email and check the SLO monitoring page (Packit section).

Watch our other two Grafana dashboards as well:

• Celery Monitoring
• Packit Monitoring

CI Hacker

This is a role card for the one responsible throughout the sprint for keeping the CI green, that is to look for and drive the resolution of systematic CI failures.

It can happen that a CI system has an outage. For problems related to Zuul, please reach out to the team at #rhos-ops internal IRC channel.

Chief of Monitors

Watch Sentry for issues:

• Fix recurring alerts
  • Deploy fixes to production for issues that cause major disruption or complete downtime
  • Verify no alerts are being triggered any more
• More complicated issues should be brought to the team and prioritized correctly.
• Clean all the past alerts so we have easy to navigate dashboard
• Link github and sentry issues

React to alerts arriving through email and check the SLO monitoring page (Packit section) and respond to the email so others know what is happening.

Watch our other two Grafana dashboards as well:

• Celery Monitoring
• Packit Monitoring

Kanban Lead

• Watch Jira board and GitHub project are up to date

• Announce the new roles on Monday in the chat.

• Be a moderator for stand-ups, refinements, architecture and retrospective meetings.

• Stand-ups:

  • Think of a question and ask it at the beginning
  • Run standup
    • Remind people to describe their cards/work when providing an update on their progress.
  • Ask about discussion topics after everyone presents their update
  • On Monday, ask about blocked cards, in-progress cards, urgent cards and priorities in the Refinement column - make sure that everything is up to date!

• If it's an odd week (check with date '+%V')

  • Create a retro board (use this template in Miro).
  • Copy actions/improvements from previous retro board, so we can check whether we did everything we wanted to.
  • Attach the link to the retrospective event in our Google calendar.
  • On retro ask about EPICs and whether we're working on the right ones

• Manual: An overview of how Packit Team kanbanz

Release Responsible

Check if there are changes worth releasing for all our RPM-packaged projects: packit, ogr, specfile

(packit is used as an example below)

• Run Prepare a new release workflow via Run workflow button: choose main branch and type the correct version number for the new release. In a while, you should see the action running and creating a pull request in the repository.
• Review the created PR - you can ask other team members to help you with changelog grammar if needed. You can check out the PR locally, update it and push it back to the PR.
• Merge the pull request once you're fine with the content (and there are no pending review requests).
• The merge of the PR should trigger an action that creates a draft release.
• You should be able to find the draft release as a first release in the releases list. Make sure the content is correct and then publish the release by clicking Publish release.
• Make sure the package is uploaded to PyPI
• Check (wait a few minutes) & merge PRs in dist-git created by packit: merge those created by the stage instance and close the ones created by the production instance. (But check that both services proposed the same set of changes.)
• Wait a few minutes and verify that RPMs are built in Koji
• Wait a bit and verify that updates have been created in Bodhi
• Celebrate another release 🚀🎉

Chief of Monitors

Watch Sentry for issues:

• Fix recurring alerts
  • Deploy fixes to production for issues that cause major disruption or complete downtime
  • Verify no alerts are being triggered any more
• More complicated issues should be brought to the team and prioritized correctly.
• Clean all the past alerts so we have easy to navigate dashboard
• Link github and sentry issues

React to alerts arriving through email and check the SLO monitoring page (Packit section) and respond to the email so others know what is happening.

Watch our other two Grafana dashboards as well:

• Celery Monitoring
• Packit Monitoring

Community Shepherd

• Be present in #packit:fedora.im, #packit:libera.chat, and Packit gchat channel to answer community questions or ping some other team member to help answer those questions.

• Keep an eye on incoming issues in github.com/packit opened by our users and react to them trying to clarify them, if needed, or asking the team to help give a meaningful reaction to them. (note: you'll need to update the date in the GitHub filter to be the first day of the sprint).

• Help new users trying to enable Packit Service to configure their repositories.

• Watch out for incoming issues in github.com/packit/notifications if someone needs help with the automatic approval process. Delete old, unanswered issues, or the ones which look like spam.

• Watch auto-packit-shared-infra mailing list for planned cluster upgrades. Once one is scheduled:

  • Create a new status post
  • Write to the #packit channel to inform everyone about the possible service disruption during the upgrade
  • Work with the Chief of Monitors to handle sentry alerts that may happen during the upgrade

Kanban Lead

• Watch Jira board is up to date

• Announce the new roles on Monday in the chat.

• Be a moderator for stand-ups, refinements, architecture and retrospective meetings.

• Stand-ups:

  • Think of a question and ask it at the beginning
  • Run standup
  • Ask about discussion topics after everyone presents their update
  • On Monday, ask about blocked cards, in-progress cards and priorities in the Refinement column - make sure that everything is up to date!

• If it's an odd week (check with date '+%V')

  • Create a retro board (use this template in Miro).
  • Copy actions/improvements from previous retro board, so we can check whether we did everything we wanted to.
  • Attach the link to the retrospective event in our Google calendar.
  • On retro ask about EPICs and whether we're working on the right ones

• Manual: An overview of how Packit Team kanbanz

Release Responsible

Check if there are changes worth releasing for all our RPM-packaged projects: packit, ogr, specfile

(packit is used as an example below)

• Run Prepare a new release workflow via Run workflow button: choose main branch and type the correct version number for the new release. In a while, you should see the action running and creating a pull request in the repository.
• Review the created PR - you can ask other team members to help you with changelog grammar if needed. You can check out the PR locally, update it and push it back to the PR.
• Merge the pull request once you're fine with the content (and there are no pending review requests).
• The merge of the PR should trigger an action that creates a draft release.
• You should be able to find the draft release as a first release in the releases list. Make sure the content is correct and then publish the release by clicking Publish release.
• Make sure the package is uploaded to PyPI
• Check (wait a few minutes) & merge PRs in dist-git created by packit: merge those created by the stage instance and close the ones created by the production instance. (But check that both services proposed the same set of changes.)
• Wait a few minutes and verify that RPMs are built in Koji
• Wait a bit and verify that updates have been created in Bodhi
• Celebrate another release 🚀🎉

Community Shepherd

• Be present in #packit:fedora.im, #packit:libera.chat, and Packit gchat channel to answer community questions or ping some other team member to help answer those questions.

• Keep an eye on incoming issues in github.com/packit opened by our users and react to them trying to clarify them, if needed, or asking the team to help give a meaningful reaction to them. (note: you'll need to update the date in the GitHub filter to be the first day of the sprint).

• Help new users trying to enable Packit Service to configure their repositories.

• Watch out for incoming issues in github.com/packit/notifications if someone needs help with the automatic approval process. Delete old, unanswered issues, or the ones which look like spam.

• Watch auto-packit-shared-infra mailing list for planned cluster upgrades. Once one is scheduled:

  • Create a new status post
  • Write to the #packit channel to inform everyone about the possible service disruption during the upgrade
  • Work with the Chief of Monitors to handle sentry alerts that may happen during the upgrade

Release Responsible

Check if there are changes worth releasing for all our RPM-packaged projects: packit, ogr, specfile

(packit is used as an example below)

• Run Prepare a new release workflow via Run workflow button: choose main branch and type the correct version number for the new release. In a while, you should see the action running and creating a pull request in the repository.
• Review the created PR - you can ask other team members to help you with changelog grammar if needed. You can check out the PR locally, update it and push it back to the PR.
• Merge the pull request once you're fine with the content (and there are no pending review requests).
• The merge of the PR should trigger an action that creates a draft release.
• You should be able to find the draft release as a first release in the releases list. Make sure the content is correct and then publish the release by clicking Publish release.
• Make sure the package is uploaded to PyPI
• Check (wait a few minutes) & merge PRs in dist-git created by packit: merge those created by the stage instance and close the ones created by the production instance. (But check that both services proposed the same set of changes.)
• Wait a few minutes and verify that RPMs are built in Koji
• Wait a bit and verify that updates have been created in Bodhi
• Celebrate another release 🚀🎉

Service Guru

Every Monday:

• (On stand-up) Check with the team whether deploying new changes from main branches is safe.
• Make sure stage deployment is OK, check:
  • Openshift cluster
  • Sentry
  • Dashboard

If all is fine and things are good to go:

• Create a blog post about what's been changed in packit & packit-service main branches since their stable branch. Use scripts/move_stable.py github-query to get a link to all the PRs from the past week which are marked to have release notes. You can copy those for a start, and check with other team members to make sure nothing was left out. You can also use the blogpost template provided by make move-stable. A blog post should meet the following requirements:
  • This is meant to be read by general public - make it easy to read for everyone.
  • Focus on new features, notable bug fixes, documentation updates, UX improvements.
  • Don't talk about internal things (e.g. most ogr/specfile changes), refactoring, CI changes etc.
  • Write it in a way so that people with a little knowledge of the project can get a clue what is the change about:
    • NO: packit's behaviour in dealing with spec files was improved
    • YES: packit can now find a spec file inside a bottle of rum which is very handy if you're thirsty
• Move content from main branches to stable branches.
  • From a clone of the deployment repo run make move-stable command. Your local ssh keys have to be associated with your github user to be able to use this script.
  • Go to the Actions tab (example of the respective repos) and make sure the images from the stable branches are built & pushed to Quay.io.
• If there were any changes in the Kubernetes/Openshift objects or in the vault/Packit/! Changelog ! that have not been already applied to prod (ask the author of the change), do so with DEPLOYMENT=prod make deploy.
• If you're doing this AFTER new images have already been built (see previous point) you should run DEPLOYMENT=prod make import-images first (or after), to avoid possible Error: ImagePullBackOff.

Monday night & Tuesday morning:

• Newer images are automatically imported from Quay.io and pods are recreated from the new images.

On Tuesday:

• Make sure prod runs as expected - revert the deployment if all jobs fail or there is significant regression in the functionality.
• Merge/publish your blog post.

Community Shepherd

• Be present in #packit:fedora.im, #packit:libera.chat, and Packit gchat channel to answer community questions or ping some other team member to help answer those questions.

• Keep an eye on incoming issues opened by our users. The unprocessed issues can be found in the new column of our Kanban board. React to them trying to clarify them, if needed, or ask the team to help give a meaningful reaction to them. Use appropriate labels and move the issue to the backlog column once the issue is clear.

• Help new users trying to enable Packit Service to configure their repositories.

• Watch out for incoming issues in github.com/packit/notifications if someone needs help with the automatic approval process. Delete old, unanswered issues, or the ones which look like spam.

• Watch auto-packit-shared-infra mailing list for planned cluster upgrades. Once one is scheduled:

  • Create a new status post
  • Write to the #packit channel to inform everyone about the possible service disruption during the upgrade
  • Work with the Chief of Monitors to handle sentry alerts that may happen during the upgrade

Service Guru

Every Monday:

• (On stand-up) Check with the team whether deploying new changes from main branches is safe.
• Make sure stage deployment is OK, check:
  • Openshift cluster
  • Sentry
  • Dashboard

If all is fine and things are good to go:

• Create a blog post about what's been changed in packit & packit-service main branches since their stable branch. Use scripts/move_stable.py github-query to get a link to all the PRs from the past week which are marked to have release notes. You can copy those for a start, and check with other team members to make sure nothing was left out. You can also use the blogpost template provided by make move-stable. A blog post should meet the following requirements:
  • This is meant to be read by general public - make it easy to read for everyone.
  • Focus on new features, notable bug fixes, documentation updates, UX improvements.
  • Don't talk about internal things (e.g. most ogr/specfile changes), refactoring, CI changes etc.
  • Write it in a way so that people with a little knowledge of the project can get a clue what is the change about:
    • NO: packit's behaviour in dealing with spec files was improved
    • YES: packit can now find a spec file inside a bottle of rum which is very handy if you're thirsty
• Move content from main branches to stable branches.
  • From a clone of the deployment repo run make move-stable command. Your local ssh keys have to be associated with your github user to be able to use this script.
  • Go to the Actions tab (example of the respective repos) and make sure the images from the stable branches are built & pushed to Quay.io.
• If there were any changes in the Kubernetes/Openshift objects or in the vault/Packit/! Changelog ! that have not been already applied to prod (ask the author of the change), do so with DEPLOYMENT=prod make deploy.
• If you're doing this AFTER new images have already been built (see previous point) you should run DEPLOYMENT=prod make import-images first (or after), to avoid possible Error: ImagePullBackOff.

Monday night & Tuesday morning:

• Newer images are automatically imported from Quay.io and pods are recreated from the new images.

On Tuesday:

• Make sure prod runs as expected - revert the deployment if all jobs fail or there is significant regression in the functionality.
• Merge/publish your blog post.

Community Shepherd

• Be present in #packit:fedora.im, #packit:libera.chat, and Packit gchat channel to answer community questions or ping some other team member to help answer those questions.

• Keep an eye on incoming issues opened by our users. The unprocessed issues can be found in the new column of our Kanban board. React to them trying to clarify them, if needed, or ask the team to help give a meaningful reaction to them. Use appropriate labels and move the issue to the backlog column once the issue is clear.

• Help new users trying to enable Packit Service to configure their repositories.

• Watch out for incoming issues in github.com/packit/notifications if someone needs help with the automatic approval process. Delete old, unanswered issues, or the ones which look like spam.

• Watch auto-packit-shared-infra mailing list for planned cluster upgrades. Once one is scheduled:

  • Create a new status post
  • Write to the #packit channel to inform everyone about the possible service disruption during the upgrade
  • Work with the Chief of Monitors to handle sentry alerts that may happen during the upgrade

Chief of Monitors

Watch Sentry for issues:

• Fix recurring alerts
  • Deploy fixes to production for issues that cause major disruption or complete downtime
  • Verify no alerts are being triggered any more
• More complicated issues should be brought to the team and prioritized correctly.
• Clean all the past alerts so we have easy to navigate dashboard
• Link github and sentry issues

React to alerts arriving through email and check the SLO monitoring page (Packit section) and respond to the email so others know what is happening.

Watch our other two Grafana dashboards as well:

• Celery Monitoring
• Packit Monitoring

Kanban Lead

• Watch Jira board and GitHub project are up to date

• Announce the new roles on Monday in the chat.

• Be a moderator for stand-ups, refinements, architecture and retrospective meetings.

• Stand-ups:

  • Think of a question and ask it at the beginning
  • Run standup
    • Remind people to describe their cards/work when providing an update on their progress.
  • Ask about discussion topics after everyone presents their update
  • On Monday, ask about blocked cards, in-progress cards, urgent cards and priorities in the Refinement column - make sure that everything is up to date!

• If it's an odd week (check with date '+%V')

  • Create a retro board (use this template in Miro).
  • Copy actions/improvements from previous retro board, so we can check whether we did everything we wanted to.
  • Attach the link to the retrospective event in our Google calendar.
  • On retro ask about EPICs and whether we're working on the right ones

• Manual: An overview of how Packit Team kanbanz

Community Shepherd

• Be present in #packit:fedora.im, #packit:libera.chat, and Packit gchat channel to answer community questions or ping some other team member to help answer those questions.

• Keep an eye on incoming issues in github.com/packit opened by our users and react to them trying to clarify them, if needed, or asking the team to help give a meaningful reaction to them. (note: you'll need to update the date in the GitHub filter to be the first day of the sprint).

• Help new users trying to enable Packit Service to configure their repositories.

• Watch out for incoming issues in github.com/packit/notifications if someone needs help with the automatic approval process. Delete old, unanswered issues, or the ones which look like spam.

• Watch auto-packit-shared-infra mailing list for planned cluster upgrades. Once one is scheduled:

  • Create a new status post
  • Write to the #packit channel to inform everyone about the possible service disruption during the upgrade
  • Work with the Chief of Monitors to handle sentry alerts that may happen during the upgrade

Release Responsible

Check if there are changes worth releasing for all our RPM-packaged projects: packit, ogr, specfile

(packit is used as an example below)

• Run Prepare a new release workflow via Run workflow button: choose main branch and type the correct version number for the new release. In a while, you should see the action running and creating a pull request in the repository.
• Review the created PR - you can ask other team members to help you with changelog grammar if needed. You can check out the PR locally, update it and push it back to the PR.
• Merge the pull request once you're fine with the content (and there are no pending review requests).
• The merge of the PR should trigger an action that creates a draft release.
• You should be able to find the draft release as a first release in the releases list. Make sure the content is correct and then publish the release by clicking Publish release.
• Make sure the package is uploaded to PyPI
• Check (wait a few minutes) & merge PRs in dist-git created by packit: merge those created by the stage instance and close the ones created by the production instance. (But check that both services proposed the same set of changes.)
• Wait a few minutes and verify that RPMs are built in Koji
• Wait a bit and verify that updates have been created in Bodhi
• Celebrate another release 🚀🎉

Release Responsible

Check if there are changes worth releasing for all our RPM-packaged projects: packit, ogr, specfile

(packit is used as an example below)

• Run Prepare a new release workflow via Run workflow button: choose main branch and type the correct version number for the new release. In a while, you should see the action running and creating a pull request in the repository.
• Review the created PR - you can ask other team members to help you with changelog grammar if needed. You can check out the PR locally, update it and push it back to the PR.
• Merge the pull request once you're fine with the content (and there are no pending review requests).
• The merge of the PR should trigger an action that creates a draft release.
• You should be able to find the draft release as a first release in the releases list. Make sure the content is correct and then publish the release by clicking Publish release.
• Make sure the package is uploaded to PyPI
• Check (wait a few minutes) & merge PRs in dist-git created by packit: merge those created by the stage instance and close the ones created by the production instance. (But check that both services proposed the same set of changes.)
• Wait a few minutes and verify that RPMs are built in Koji
• Wait a bit and verify that updates have been created in Bodhi
• Celebrate another release 🚀🎉

Kanban Lead

• Watch Jira board is up to date

• Announce the new roles on Monday in the chat.

• Be a moderator for stand-ups, refinements, architecture and retrospective meetings.

• Stand-ups:

  • Think of a question and ask it at the beginning
  • Run standup
  • Ask about discussion topics after everyone presents their update
  • On Monday, ask about blocked cards, in-progress cards, urgent cards and priorities in the Refinement column - make sure that everything is up to date!

• If it's an odd week (check with date '+%V')

  • Create a retro board (use this template in Miro).
  • Copy actions/improvements from previous retro board, so we can check whether we did everything we wanted to.
  • Attach the link to the retrospective event in our Google calendar.
  • On retro ask about EPICs and whether we're working on the right ones

• Manual: An overview of how Packit Team kanbanz

Release Responsible

Check if there are changes worth releasing for all our RPM-packaged projects: packit, ogr, specfile

(packit is used as an example below)

• Create & merge (after a review) a PR with updated Changelog (use our script to generate new changelog entries) for the spec file
• Create a new release on Github with:
  • Target: main
  • Tag: the version - identical to what's inside spec file, example 0.42.1
  • Release title: the same as Tag
• Make sure the package is uploaded to PyPI
• Check (wait a few minutes) & merge PRs in dist-git created by packit: merge those created by the stage instance and close the ones created by the production instance. (But check that both services proposed the same set of changes.)
• Wait a few minutes and verify that RPMs are built in Koji
• Wait a bit and verify that updates have been created in Bodhi
• Celebrate another release 🚀🎉

Release Responsible

Check if there are changes worth releasing for all our RPM-packaged projects: packit, ogr, specfile

(packit is used as an example below)

• Run Prepare a new release workflow via Run workflow button: choose main branch and type the correct version number for the new release. In a while, you should see the action running and creating a pull request in the repository.
• Review the created PR - you can ask other team members to help you with changelog grammar if needed. You can check out the PR locally, update it and push it back to the PR.
• Merge the pull request once you're fine with the content (and there are no pending review requests).
• The merge of the PR should trigger an action that creates a draft release.
• You should be able to find the draft release as a first release in the releases list. Make sure the content is correct and then publish the release by clicking Publish release.
• Make sure the package is uploaded to PyPI
• Check (wait a few minutes) & merge PRs in dist-git created by packit: merge those created by the stage instance and close the ones created by the production instance. (But check that both services proposed the same set of changes.)
• Wait a few minutes and verify that RPMs are built in Koji
• Wait a bit and verify that updates have been created in Bodhi
• Celebrate another release 🚀🎉

Kanban Lead

• Watch Jira board is up to date

• Be a moderator for stand-ups, refinements, architecture and retrospective meetings.

• Stand-ups:

  • Think of a question and ask it at the beginning
  • Run standup
  • Ask about discussion topics after everyone presents their update
  • On Monday, ask about blocked cards, in-progress cards and priorities in the Refinement column - make sure that everything is up to date!

• If it's an odd week (check with date '+%V')

  • Create a retro board (use this template in Miro).
  • Copy actions/improvements from previous retro board, so we can check whether we did everything we wanted to.
  • Attach the link to the retrospective event in our Google calendar.

• Manual: An overview of how Packit Team kanbanz

Chief of Monitors

Watch Sentry for issues:

• Fix recurring alerts
  • Deploy fixes to production for issues that cause major disruption or complete downtime
  • Verify no alerts are being triggered any more
• More complicated issues should be brought to the team and prioritized correctly.
• Clean all the past alerts so we have easy to navigate dashboard
• Link github and sentry issues

React to alerts arriving through email and check the SLO monitoring page (Packit section) and respond to the email so others know what is happening.

Watch our other two Grafana dashboards as well:

• Celery Monitoring
• Packit Monitoring

Service Guru

Every Monday:

• (On stand-up) Check with the team whether deploying new changes from main branches is safe.
• Make sure stage deployment is OK, check:
  • Openshift cluster
  • Sentry
  • Dashboard

If all is fine and things are good to go:

• Create a blog post about what's been changed in packit & packit-service main branches since their stable branch. Use scripts/move_stable.py github-query to get a link to all the PRs from the past week which are marked to have release notes. You can copy those for a start, and check with other team members to make sure nothing was left out. You can also use the blogpost template provided by make move-stable. A blog post should meet the following requirements:
  • This is meant to be read by general public - make it easy to read for everyone.
  • Focus on new features, notable bug fixes, documentation updates, UX improvements.
  • Don't talk about internal things (e.g. most ogr/specfile changes), refactoring, CI changes etc.
  • Write it in a way so that people with a little knowledge of the project can get a clue what is the change about:
    • NO: packit's behaviour in dealing with spec files was improved
    • YES: packit can now find a spec file inside a bottle of rum which is very handy if you're thirsty
• Move content from main branches to stable branches.
  • From a clone of the deployment repo run make move-stable command. Your local ssh keys have to be associated with your github user to be able to use this script.
  • Go to the Actions tab (example of the respective repos) and make sure the images from the stable branches are built & pushed to Quay.io.
• If there were any changes in the Kubernetes/Openshift objects or in the vault/Packit/! Changelog ! that have not been already applied to prod (ask the author of the change), do so with DEPLOYMENT=prod make deploy.
• If you're doing this AFTER new images have already been built (see previous point) you should run DEPLOYMENT=prod make import-images first (or after), to avoid possible Error: ImagePullBackOff.

Monday night & Tuesday morning:

• Newer images are automatically imported from Quay.io and pods are recreated from the new images.

On Tuesday:

• Make sure prod runs as expected - revert the deployment if all jobs fail or there is significant regression in the functionality.
• Merge/publish your blog post.

Service Guru

Every Monday morning:

• Check with the team whether deploying new changes from main branches is safe (e.g. on stand-up).
• Make sure stage deployment is OK, check:
  • Openshift cluster
  • Sentry
  • Dashboard

If all is fine and things are good to go:

• Create a blog post about what's been changed in packit & packit-service main branches since their stable branch. Use scripts/move_stable.py github-query to get a link to all the PRs from the past week which are marked to have release notes. You can copy those for a start, and check with other team members to make sure nothing was left out. You can also use the blogpost template provided by make move-stable. A blog post should meet the following requirements:
  • This is meant to be read by general public - make it easy to read for everyone.
  • Focus on new features, notable bug fixes, documentation updates, UX improvements.
  • Don't talk about internal things, refactoring, CI changes etc.
  • Write it in a way so that people with a little knowledge of the project can get a clue what is the change about:
    • NO: packit's behaviour in dealing with spec files was improved
    • YES: packit can now find a spec file inside a bottle of rum which is very handy if you're thirsty
• Move content from main branches to stable branches.
  • From a clone of the deployment repo run make move-stable command. Your local ssh keys have to be associated with your github user to be able to use this script.
  • Go to the Actions tab (example of the respective repos and make sure the images from the stable branches are built & pushed to Quay.io.
• Run a proper prod deployment (DEPLOYMENT=prod make deploy) to sync all the Openshift objects.
  • Optional, if the objects or secrets did not change in the week, you can skip this step.
• If you're doing this AFTER new images have already been built (see previous point) you should run DEPLOYMENT=prod make import-images first (or after), to avoid possible Error: ImagePullBackOff.

Monday night & Tuesday morning:

• Prod is being automatically re-deployed from new images on Tue morning.

On Tuesday:

• Make sure prod runs as expected - revert the deployment if all jobs fail or there is significant regression in the functionality.
• Merge/publish your blog post.

Kanban Lead

• Watch Jira board is up to date

• Be a moderator for stand-ups, refinements, architecture and retrospective meetings.

• Stand-ups:

  • Think of a question and ask it at the beginning
  • Run standup
  • Ask about discussion topics after everyone presents their update
  • On Monday, ask about blocked cards, in-progress cards and priorities in the Refinement column - make sure that everything is up to date!

• If it's an odd week (check with date '+%V')

  • Create a retro board (use this template in Miro).
  • Copy actions/improvements from previous retro board, so we can check whether we did everything we wanted to.
  • Attach the link to the retrospective event in our Google calendar.

• Manual: An overview of how Packit Team kanbanz

Release Responsible

Check if there are changes worth releasing for all our RPM-packaged projects: packit, ogr, specfile

(packit is used as an example below)

• Run Prepare a new release workflow via Run workflow button: choose main branch and type the correct version number for the new release. In a while, you should see the action running and creating a pull request in the repository.
• Review the created PR - you can ask other team members to help you with changelog grammar if needed. You can check out the PR locally, update it and push it back to the PR.
• Merge the pull request once you're fine with the content (and there are no pending review requests).
• The merge of the PR should trigger an action that creates a draft release.
• You should be able to find the draft release as a first release in the releases list. Make sure the content is correct and then publish the release by clicking Publish release.
• Make sure the package is uploaded to PyPI
• Check (wait a few minutes) & merge PRs in dist-git created by packit: merge those created by the stage instance and close the ones created by the production instance. (But check that both services proposed the same set of changes.)
• Wait a few minutes and verify that RPMs are built in Koji
• Wait a bit and verify that updates have been created in Bodhi
• Celebrate another release 🚀🎉

CI Hacker

This is a role card for the one responsible throughout the sprint for keeping the CI green, that is to look for and drive the resolution of systematic CI failures.

It can happen that a CI system has an outage. For problems related to Zuul, please reach out to the team at #rhos-ops internal IRC channel.

Chief of Monitors

Watch Sentry for issues:

• Fix recurring alerts
  • Deploy fixes to production for issues that cause major disruption or complete downtime
  • Verify no alerts are being triggered any more
• More complicated issues should be brought to the team and prioritized correctly.
• Clean all the past alerts so we have easy to navigate dashboard
• Link github and sentry issues

React to alerts arriving through email and check the SLO monitoring page (Packit section) and respond to the email so others know what is happening.

Watch our other two Grafana dashboards as well:

• Celery Monitoring
• Packit Monitoring

Create checklist for one-off tasks

I am a kanban lead for this week and there are a few one-off tasks to do. Since I already completed them, I'd love to tick a box:

• so others know I'm done
• so I know that I did it
• to get that sense of completing work

Since other roles have also one-time tasks, WDYT about updating the roles?

Kanban Lead

• Watch Jira board is up to date

• Announce the new roles on Monday in the chat.

• Be a moderator for stand-ups, refinements, architecture and retrospective meetings.

• Stand-ups:

  • Think of a question and ask it at the beginning
  • Run standup
  • Ask about discussion topics after everyone presents their update
  • On Monday, ask about blocked cards, in-progress cards and priorities in the Refinement column - make sure that everything is up to date!

• If it's an odd week (check with date '+%V')

  • Create a retro board (use this template in Miro).
  • Copy actions/improvements from previous retro board, so we can check whether we did everything we wanted to.
  • Attach the link to the retrospective event in our Google calendar.
  • On retro ask about EPICs and whether we're working on the right ones

• Manual: An overview of how Packit Team kanbanz

CI Hacker

This is a role card for the one responsible throughout the sprint for keeping the CI green, that is to look for and drive the resolution of systematic CI failures.

It can happen that a CI system has an outage. For problems related to Zuul, please reach out to the team at #rhos-ops internal IRC channel.

Service Guru

Every Monday morning:

• Check with the team whether deploying new changes from main branches is safe (e.g. on stand-up).
• Make sure stage deployment is OK, check:
  • Openshift cluster
  • Sentry
  • Dashboard

If all is fine and things are good to go:

• Create a blog post about what's been changed in packit & packit-service main branches since their stable branch. Use scripts/move_stable.py github-query to get a link to all the PRs from the past week which are marked to have release notes. You can copy those for a start, and check with other team members to make sure nothing was left out. You can also use the blogpost template provided by make move-stable. A blog post should meet the following requirements:
  • This is meant to be read by general public - make it easy to read for everyone.
  • Focus on new features, notable bug fixes, documentation updates, UX improvements.
  • Don't talk about internal things, refactoring, CI changes etc.
  • Write it in a way so that people with a little knowledge of the project can get a clue what is the change about:
    • NO: packit's behaviour in dealing with spec files was improved
    • YES: packit can now find a spec file inside a bottle of rum which is very handy if you're thirsty
• Move content from main branches to stable branches.
  • From a clone of the deployment repo run make move-stable command. Your local ssh keys have to be associated with your github user to be able to use this script.
  • Go to the Actions tab (example of the respective repos and make sure the images from the stable branches are built & pushed to Quay.io.
• Run a proper prod deployment (DEPLOYMENT=prod make deploy) to sync all the Openshift objects.
  • Optional, if the objects or secrets did not change in the week, you can skip this step.
• If you're doing this AFTER new images have already been built (see previous point) you should run DEPLOYMENT=prod make import-images first (or after), to avoid possible Error: ImagePullBackOff.

Monday night & Tuesday morning:

• Prod is being automatically re-deployed from new images on Tue morning.

On Tuesday:

• Make sure prod runs as expected - revert the deployment if all jobs fail or there is significant regression in the functionality.
• Merge/publish your blog post.

CI Hacker

This is a role card for the one responsible throughout the sprint for keeping the CI green, that is to look for and drive the resolution of systematic CI failures.

It can happen that a CI system has an outage. For problems related to Zuul, please reach out to the team at #rhos-ops internal IRC channel.