dupuy / reliabot Goto Github PK

Provide as much context as you can for your question.

• Reliabot version 0.2.x GitHub Actions workflow for:
  • Building and testing Reliabot across a range of Python releases
  • Tagging and releasing on GitHub
  • Publishing Reliabot to (Test)PyPI

Explain what you're trying to do, and what you don't understand.

I want a build/release/publish process that works like this:

• Single source of version truth in pyproject.yaml – version = ...
  • Git tags generated when version bump PR is merged to main
  • Releases get annotated tags, pre-releases get lightweight ones
• Build and Publish jobs are separate
  • Build and Test jobs run on pull_request activity
  • GitHub Release and (Test)PyPI Publish jobs run after PR merge (push) events
• No use of secret tokens
  • (Test)PyPI uses "Trusted Publisher" (OpenID Connect | OIDC) access
  • Default GITHUB_TOKEN used for GitHub operations
• Least Privilege model
• Best Security practices
• Minimize duplication of effort consistent with the above

There are a number of problems that I'm facing that have a
Catch-22 feeling:

• Trusted Publisher access to TestPyPI fails in pull_request workflow:

  Error: Trusted publishing exchange failure: 
  Token request failed: the server refused the request for the following reasons:
  
  * `invalid-publisher`: valid token, but no corresponding publisher (All lookup strategies exhausted)
  

  Claims provided:

  • sub: repo:dupuy/reliabot:environment:test-pypi
  • repository: dupuy/reliabot
  • repository_owner: dupuy
  • repository_owner_id: 33216
  • job_workflow_ref: dupuy/reliabot/.github/workflows/python-app.yaml@refs/pull/100/merge
  • ref: refs/pull/100/merge

• It seems impossible to get build artifacts from pull_request to push workflows without creating other problems

  • Test-publish and draft-release jobs in pull_request workflow can download workflow artifacts uploaded from build or test jobs in same workflow run, but default GITHUB_TOKEN is scoped to same-workflow access:

    You can only download artifacts in a workflow that were uploaded during the same workflow run.

  • Using release assets works between workflows, but only for published releases

    Downloading release assets from draft workflows used to be possible with the gh CLI command, but was broken multiple times as documented in GitHub CLI issue #3037 and has not been working since January 2024.

    (From comments on the issue, it might be possible to do this using the REST API directly with curl.)

    • Publishing a release for which the tag does not exist automatically creates a lightweight tag.
    • Creating an annotated tag in the push workflow (necessary to get the merged main branch commit) with the release notes from the build job can;t be done until the release is published. Catch-22!

1. Probably the best solution is to run the build job in both pull_request and push workflows, generate the annotated tag and release notes in the push build and publish the release there. This duplicates effort, but that's the least of the desiderata.

2. Using the REST API to retrieve the draft release assets (as noted above) could be substantial effort, and might not even work without a Personal Access Token.

3. Alternately, a GitHub fine-grained Personal Access Token stored as a GitHub secret could be used to retrieve the build artifacts. But this may require regenerating the token periodically, which is a maintenance hassle.

Hopefully the Trusted Publisher (OIDC) (Test)PyPI access works in the push workflow. If that doesn't work, publishing to (Test)PyPI might have to be triggered in a third release event triggered workdlow run.

Describe the solution you want to see

To improve the security of Reliabot and find buggy handling of edge cases,
we should run a fuzzer in a GitHub Actions workflow.

Outline alternatives you've considered

There are a few different fuzzers available for Python

• Atheris 2.3.0 –
  from Google, requires clang to run local on macOS
• Frelatage 0.1.7 –
  "alpha" status AFL lookalike
• Pythonfuzz –
  GitLab-hosted (and maintained), GitHub repo is archived

Of these, Atheris seems the most promising alternative.

While it would be useful to run a fuzzer locally, it's more certain if runs
automatically (on a weekly basis?) in a GitHub Actions workflow.

Given issues with clang that make it harder to run on macOS, it may
make sense to run this in a Docker container for use on Mac systems.

Provide a rationale

Fuzzing the code improves security as it can expose edge cases that Reliabot
doesn't handle correctly (and potentially, memory-unsafe native code).
It also improves reliability by finding potential exception cases,
which can be addressed with fixes to the code.

Also, it would improve the Open SSF Scorecard results.

Reliabot 0.1.1 and earlier generate Dependabot configuration that doesn't specify a schedule interval, which is a required part of configuration.

This causes Dependabot errors like the following:

The property '#/updates/0' did not contain a required property of 'schedule'
The property '#/updates/1' did not contain a required property of 'schedule'

At a minimum, Reliabot should provide a schedule interval for the update entries it creates.

Adding a missing schedule interval for existing update entries (or commenting them out?) might be desirable, but perhaps oversteps the proper scope for Reliabot activity.

Configuring the interval that is used for new update checks (and possibly for update checks that lack a schedule interval) should be a separate feature request. For now, a fixed value (probably "monthly") is sufficient.

The fallback warning for lack of re2 is even worse than the ruamel.yaml failure in #15.

reliabot$ ./reliabot/reliabot.py .
/Users/alexdupuy/Work/reliabot/./reliabot/reliabot.py:55: RuntimeWarning: Cannot import re2, falling back to re
  warnings.warn("Cannot import re2, falling back to re", RuntimeWarning)

A better version would be more informative and not redundant. An option to suppress the warning would be nice.

reliabot$ ./reliabot/reliabot.py .
RuntimeWarning: Cannot import 're2', falling back to 're'

Reliabot works better with the 're2' regular expression module.
See https://github.com/dupuy/reliabot/#installation for installation instructions,
or use the '--re' option to prevent use of 're2' and suppress the warning.

When you run Reliabot in a Python environment that does not have ruamel.yaml you get this error:

reliabot$ ./reliabot/reliabot.py .
Traceback (most recent call last):
  File "/Users/alexdupuy/Work/reliabot/./reliabot/reliabot.py", line 44, in <module>
    from ruamel.yaml import YAML  # ruamel.yaml preserves comments, PyYAML doesn't.
ModuleNotFoundError: No module named 'ruamel'

The reliabot.py script is pretty self contained and doesn't have any other required dependencies (re2 is optional). It would nice to print something more helpful in this case, like this:

reliabot$ ./reliabot/reliabot.py .
ModuleNotFoundError: No module named 'ruamel'

Reliabot requires the ruamel.yaml module to preserve comments in dependabot.yml files
See https://github.com/dupuy/reliabot/#installation for installation instructions,

Describe the problem

Briefly describe the expected behavior and Reliabot's actual behavior.

The dependabot update #83 for the CodeQL version failed the CodeQL check itself:

Setup CodeQL tools
  Did not find CodeQL tools version 2.16.4 in the toolcache.
  Downloading CodeQL tools from https://github.com/github/codeql-action/releases/download/codeql-bundle-v2.16.4/codeql-bundle-linux64.tar.gz . This may take a while.
  connect ECONNREFUSED 54.185.253.63:443
  Waiting 10 seconds before trying again
  connect ECONNREFUSED 54.185.253.63:443
  Waiting [16](https://github.com/dupuy/reliabot/actions/runs/8316590657/job/22756242528#step:5:17) seconds before trying again
  Error: Unable to download and extract CodeQL CLI: connect ECONNREFUSED 54.185.253.63:443

The problem is that the hardened runner is unable to download the update, which has not yet been incorporated into the runner build.

Describe the solution you want to see

If you want to run Reliabot on a macOS system outside of pre-commit, but don't
want to have to worry about virtualenvs or the dependency on the RE2 C++ library
and its Python wrapper, it would be nice to just run brew install reliabot
and have it available.

Outline alternatives you've considered

You can install Reliabot with Pip (once it gets uploaded to PyPI), but RE2
support is tricky and building the RE2 C++ library is a heavy lift for a casual
user.

The pyre2-updated package makes this somewhat easier, but that has no
permanent maintainer, and it took several months for a Python 3.12 version to
become available.

Some of this could also be simplified by enhancing the development Makefile to
install reliabot itself with pipx, but that would still depend on
pyre2-updated support.

Provide a rationale

While there aren't many people who would run reliabot on the command line
outside of pre-commit, and even fewer using macOS, having it available in
Homebrew would make it easier to "try out" before making a big commitment to
pre-commit.

Describe the solution you want to see

Explain how you want Reliabot to work, providing as many details as possible.

When reliabot creates a new directory entry in dependabot.yml, it uses minimal
built-in configuration:

  - directory: /
    package-ecosystem: github-actions
    schedule:
      interval: daily

However, you might prefer a longer interval, and for certain frequently updated
packages (such as AWS SDKs like boto*) you might want to add an ignore
section to skip patch updates for version and not security:

  - package-ecosystem: npm
    directory: /
    schedule:
      interval: weekly
    ignore:
        # For AWS SDK, ignore patch version (not security) updates
      - dependency-name: "aws-sdk"
        update-types: ["version-update:semver-patch"]

Outline alternatives you've considered

A command-line option --default-interval and a simple reliabot-comment
default-interval setting for the dependabot.yml file itself would handle the
most basic case. This would be quite a good first issue to implement.

For settings specific to package ecosystem (whether interval or ignore or
something else, it may be possible to use non-comment configuration for a dummy
.reliabot-defaults directory (which might not even need to exist, if
Dependabot doesn't complain about that).

An advantage of using "live" configuration is that Dependabot parses and
validates it, so that typos or syntax errors can be caught immediately. Default
configuration in comments would pass no matter how egregious the errors, and
Dependabot semantic errors in other YAML configuration files would not be
checked until and unless copied to dependabot.yml. (Using other files would
also split the configuration into multiple locations.)

Provide a rationale

Explain why this enhancement would be useful to many reliabot users.

Being able to configure defaults for newly added code locations can eliminate
any need to edit them when they are created, further automating the process.

Running reliabot 0.1.1 or earlier on a repository whose dependabot.yml file has no comments causes this error:

Traceback (most recent call last):
  File "/Users/alexdupuy/Work/reliabot/./reliabot/reliabot.py", line 1186, in <module>
    sys.exit(main(sys.argv))
  File "/Users/alexdupuy/Work/reliabot/./reliabot/reliabot.py", line 291, in main
    settings = extract_settings(conf, {**EMITTER_SETTINGS, **EXCLUSIONS})
  File "/Users/alexdupuy/Work/reliabot/./reliabot/reliabot.py", line 385, in extract_settings
    comments = [comment.value for comment in config.ca.comment[1]]
TypeError: 'NoneType' object is not subscriptable