ropensci-org / badges Goto Github PK

Ping @yabellini @noamross @maelle

Current script recently started failing, requiring fix in 8952f9c. The failure notifications were only delivered to me as author of the issue. https://docs.github.com/en/actions/monitoring-and-troubleshooting-workflows/notifications-for-workflow-runs:

Notifications for scheduled workflows are sent to the user who initially created the workflow. If a different user updates the cron syntax in the workflow file, subsequent notifications will be sent to that user instead. If a scheduled workflow is disabled and then re-enabled, notifications will be sent to the user who re-enabled the workflow rather than the user who last modified the cron syntax.

There is currently no direct way for others to receive workflow failure notifications. The easiest viable work-around would seem to be to add an additional workflow step with if: ${{ failure() }} to then send an email to additional people. @maelle Would that be okay for you? The failures are quite loud here, as the workflow runs four times a day

• but failures are pretty critical, so important that somebody else gets notified.

@jeroen The ruby script currently creates this 'onboarded.json' file. Can you please confirm that those json data are not used anywhere anymore, and so both that file and the part of the script that creates it can be deleted? Thanks!

@sckott @noamross Starting this in a separate issue to #2 to help contain this discussion in one place. The following is a suggestion for a viable workflow to capture versioning directly from the actual badges, which is a general approach that @noamross and i both agree is likely to be both more direct, and most clearly indicative of versioning.

1. The bot replies to editor command @ropensci-review-bot approve silver (for example) with approved silver v0.0.x, and labels the issue with the appropriately versioned badge which would all be of the form approved-bronze-v0.0.x
2. This requires the bot to somehow grab the current latest standard versions (@xuanxu can work out the best approach for that), in order to assign the appropriately versioned label.
3. update_badges.rb is triggered by that same bot response (as well as cron-ed on daily basis), and dumps the correspondingly graded and versioned svgs into the gh-pages branch.

The third step will require these modifications to update_badges.rb:

1. The script will have to match both colors as in current #3, as well as versions.
2. That svg_map will slowly have to be expanded to map on to all versioned forms of all badges.

The singular advantage of handling everything here through direct labelling of badges is that it enables the management of versioning to be handled entirely independent of this repo and allows update_badges.rb to be easily expanded to new versions though inserting additional single lines only in svg_map. (And FYI version alignment will always be handled in original review issues, including via yet-to-be-implemented bot commands like @ropensci-review-bot upgrade to gold @ropensci-review-bot downgrade to bronze. Those commands will trigger changes to the badges in the original issue, so those badges will always record current alignment in an ongoing manner.)

@sckott This is intended only as a suggested workflow, so please suggest any other/better approaches you might think of, or clarify anything i may have misunderstood here. Thanks!

@sckott We would now like to incorporate the new stats peer review badges here. There are 3 new badges:

Your current code here simply associates the standard "Approved" label with the standard "Peer-reviewed" svg. We have not to date considered different versions of the "Approved" label itself; only that approval will be triggered by different commands of @ropensci-review-bot approve bronze/silver/gold. I guess the easiest thing would be to translate these into distinct issue labels, so they could then be directly used in the update_badges.rb script? That could all then simply extend the current "status" variable, so it could take these three new values as extensions of "approved". The JSON format would then not have to change. Does that sound workable?

A second problem we would then have would be the versioning. As standards develop, numbers of badges will increase, retaining one set of 3 badges for each preceding iteration of standards. We're still working out the best way to track and record this. The version data should be automatically generated in response to the final approve command, so the bot will declare the current version of standards for which the software is approved. I guess we could do that in a standard way so it could be grepped in update_badges to extract the associated version? There'll also be possibilities for authors to update their software to align with more recent standards, so the command to approve standards alignment will then be repeated and will generate a second version of the same statement.

One direct way to achieve this would be to also have versioned forms of the issue labels themselves, so there would be approved-stats-bronze-0.0.1 and so on. Bot commands could be used to directly update the issue labels, and so update_badges would then only need to find an approved label and then map the full name of the label on to the name of associated svgs. Only problem there would be proliferation of issue labels, so would definitely like your feedback there before thinking any further. Ping @noamross

See https://github.com/ropensci-org/badges/blob/gh-pages/json/onboarded.json

For example cffr is listed twice.

The onboarding.json contains several entries with "pkgname": null which causes issues like ropensci-org/rotemplate#104

  {
    "pkgname": null,
    "submitted": "maelle",
    "iss_no": 24,
    "status": "reviewed",
    "version": "0.1",
    "stats_version": null
  },

The script should ideally have tests. Some ideas:

• use vcr for fixtures for github api data (via octokit) - so that we're not doing real http requests for tests
• use some smaller subset of data, don't need all issues data
• test that
  • badge file names are correct
  • svg content in badges is correct
  • making badges correctly for regular review vs. stats review
  • check that json data is formatted correctly and of write types, etc

Issue to post notifications of workflow failures. Can be closed any time if desired; only used to dump notifications from this workflow step:

For example cffr and melt: https://github.com/ropensci-org/badges/blob/gh-pages/json/onboarded.json

It seems that the corresponding badge images are actually correct:

(cffr) https://github.com/ropensci-org/badges/blob/gh-pages/463_status.svg
(melt) https://github.com/ropensci-org/badges/blob/gh-pages/550_status.svg

This repo generally doesn't need any modifications, but GitHub now disables actions on 60 days of inactivity, which means the badge server automatically stops updating, leading to things like ropensci/software-review#486 (comment). We definitely need a way to prevent this particular workflow from being auto-disabled. @jeroen Any brilliant suggestions here?

@yabellini The new champions badge (#10) won't be allocated until we update the script which generates the "onboarded.json" file. This whole workflow can only assign one badge to a review thread/package, so champions will have the new champions badge but not a "normal" peer-reviewed badge. Just so you know!

Have any champions packages passed through review yet? How quickly do we need the badges assigned? (All packages still under review just get the generic https://github.com/ropensci-org/badges/blob/main/svgs/pending.svg badge.)