dupuy / reliabot Goto Github PK

Explain how you want Reliabot to work, providing as many details as possible.

To improve the security of Reliabot and find buggy handling of edge cases,
we should run a fuzzer in a GitHub Actions workflow.

Outline alternatives you've considered

Summarize any alternatives, and explain why they aren't as good.

There are a few different fuzzers available for Python

• Atheris 2.3.0 –
  from Google, requires clang to run local on macOS
• Frelatage 0.1.7 –
  "alpha" status AFL lookalike
• Pythonfuzz –
  GitLab-hosted (and maintained), GitHub repo is archived

Of these, Atheris seems the most promising alternative.

While it would be useful to run a fuzzer locally, it's more certain if runs
automatically (on a weekly basis?) in a GitHub Actions workflow.

Given issues with clang that make it harder to run on macOS, it may
make sense to run this in a Docker container for use on Mac systems.

Provide a rationale.

Explain why this enhancement would be useful to many Reliabot users.

Fuzzing the code improves security as it can expose edge cases that Reliabot
doesn't handle correctly (and potentially, memory-unsafe native code).
It also improves reliability by finding potential exception cases,
which can be addressed with fixes to the code.

Also, it would improve the Open SSF Scorecard results.

When you run Reliabot in a Python environment that does not have ruamel.yaml you get this error:

reliabot$ ./reliabot/reliabot.py .
Traceback (most recent call last):
  File "/Users/alexdupuy/Work/reliabot/./reliabot/reliabot.py", line 44, in <module>
    from ruamel.yaml import YAML  # ruamel.yaml preserves comments, PyYAML doesn't.
ModuleNotFoundError: No module named 'ruamel'

The reliabot.py script is pretty self contained and doesn't have any other required dependencies (re2 is optional). It would nice to print something more helpful in this case, like this:

reliabot$ ./reliabot/reliabot.py .
ModuleNotFoundError: No module named 'ruamel'

Reliabot requires the ruamel.yaml module to preserve comments in dependabot.yml files
See https://github.com/dupuy/reliabot/#installation for installation instructions,

The fallback warning for lack of re2 is even worse than the ruamel.yaml failure in #15.

reliabot$ ./reliabot/reliabot.py .
/Users/alexdupuy/Work/reliabot/./reliabot/reliabot.py:55: RuntimeWarning: Cannot import re2, falling back to re
  warnings.warn("Cannot import re2, falling back to re", RuntimeWarning)

A better version would be more informative and not redundant. An option to suppress the warning would be nice.

reliabot$ ./reliabot/reliabot.py .
RuntimeWarning: Cannot import 're2', falling back to 're'

Reliabot works better with the 're2' regular expression module.
See https://github.com/dupuy/reliabot/#installation for installation instructions,
or use the '--re' option to prevent use of 're2' and suppress the warning.

Reliabot 0.1.1 and earlier generate Dependabot configuration that doesn't specify a schedule interval, which is a required part of configuration.

This causes Dependabot errors like the following:

The property '#/updates/0' did not contain a required property of 'schedule'
The property '#/updates/1' did not contain a required property of 'schedule'

At a minimum, Reliabot should provide a schedule interval for the update entries it creates.

Adding a missing schedule interval for existing update entries (or commenting them out?) might be desirable, but perhaps oversteps the proper scope for Reliabot activity.

Configuring the interval that is used for new update checks (and possibly for update checks that lack a schedule interval) should be a separate feature request. For now, a fixed value (probably "monthly") is sufficient.

Running reliabot 0.1.1 or earlier on a repository whose dependabot.yml file has no comments causes this error:

Traceback (most recent call last):
  File "/Users/alexdupuy/Work/reliabot/./reliabot/reliabot.py", line 1186, in <module>
    sys.exit(main(sys.argv))
  File "/Users/alexdupuy/Work/reliabot/./reliabot/reliabot.py", line 291, in main
    settings = extract_settings(conf, {**EMITTER_SETTINGS, **EXCLUSIONS})
  File "/Users/alexdupuy/Work/reliabot/./reliabot/reliabot.py", line 385, in extract_settings
    comments = [comment.value for comment in config.ca.comment[1]]
TypeError: 'NoneType' object is not subscriptable