buildkite-plugins / buildkite-plugin-linter Goto Github PK

If the tags v1 and v1.0.1 actually target the same commit I get an error Readme version numbers out of date. Latest is v1.0.1
This is because const tags = await git.listTags({ fs, dir: pluginPath }) have no concept of multiple tags being the "same" version.
Is this the intended behaviour?

Somewhat related to #485

I have a plugin where the linter doesn't find the example in the readme because it's not inside steps: (https://github.com/bugcrowd/test-summary-buildkite-plugin). The linter used to just let that pass with a log message but now it fails. So I guess there's two issues here:

1. The breaking change of requiring a plugin example in the readme without a version bump
2. Not detecting the plugin example

Fixing either of them will solve my problem.

Array syntax is mandatory in example

e.g. the following is rejected

steps:
  - plugins:
      my-org/my-plugin#v1.0.0:

in favour of

steps:
  - plugins:
      - my-org/my-plugin#v1.0.0:

Tt should be unnecessary to configure the linter with --name just so it can search the readme for plugins: ${name}#v.* code blocks, because it's already set in the plugin.yml file. It'll need to handle the org scoping tho, which is currently done via the --name linter argument.

Running the plugin with the --id option causes a warning/error if the provided id ends with buildkite-plugin. That makes sense on traditional plugins, but vendored ones shouldn't have that restriction as the folder name can be whatever is wanted/necessary

The plugin-linter should fail any readme examples that are missing the correct GitHub username/org prefix in the usage examples.

See thedyrt/skip-checkout-buildkite-plugin#1 for an example.

Hi! 👋 In the documentation for my Buildkite plugin i'd like to be able to tell users that they can use the BUILDKITE_PLUGINS_ALWAYS_CLONE_FRESH feature as described here to get rapid development feedback while making changes to a Buildkite plugin. I want to provide an example in the docs, something like this:

steps:
  - name: ":hiking_boot: Kick off plugin"
    env:
      BUILDKITE_PLUGINS_ALWAYS_CLONE_FRESH: "true"
    plugins:
      - ssh://[email protected]/org/bar-buildkite-plugin.git#your-branch-name:
          some_option: foo

However, the linter is complaining that your-branch-name isn't a valid version tag. I mean, that's correct. But it's also pretty annoying. I wonder if folks have a workaround? I could also attempt to implement, e.g., a whitelist of "always valid" version tags, but that is rather a yak shave for what i want to get done right now.

The logic in readme-version-number-linter.js is to take the highest version numbered tag, when sorted semantically:

However, this means it often picks non-stable version numbers, like release candidates and alpha/beta versions:

not ok 3 - Readme version numbers out of date. Latest is v3.2.0-alpha.1
 invalid version numbers:
 - v3.1.2

It also means that if the documentation has been prepared for release, and has a version number greater than the latest one, we'll still error (whereas we could check that the version numbers simply aren't older than the latest release to allow this case)

not ok 3 - Readme version numbers out of date. Latest is v3.2.0-alpha.1
 invalid version numbers:
 - v3.2.0

(Note: 3.2.0 is semantically newer than 3.2.0-alpha.1, so this lint also shouldn't say "out of date")

This package implements the sort of logic we could use for this lint

I wrote https://github.com/joscha/sauce-connect-buildkite-plugin

and the output:

steps:
  - command: 'yarn && yarn saucelabs-based-tests'
    plugins:
      sauce-connect#v1.0.0: ~

makes the linter happy, but technically it needs to be:

steps:
  - command: 'yarn && yarn saucelabs-based-tests'
    plugins:
      joscha/sauce-connect#v1.0.0: ~

as this plugin is not part of the buildkite-plugins repo.

In my private repo, I have a README.md with this YAML example block in it.

steps:
  - command: echo foo
    plugins:
      - ssh://[email protected]/improbable/toolshare-buildkite-plugin.git#v0.0.1: {}

It fails linting:

$ docker-compose run --rm lint
Creating toolshare-buildkite-plugin_lint_run ... done
TAP version 13
ok 1 - plugin.yml is valid
not ok 2 - No readme config examples found with plugin id 'improbable/toolshare'. Have you configured the linter with the correct plugin id?
ok 3 - Readme version numbers are up-to-date (v0.0.1)
1..3
# failed 1 of 3 tests
# time=85.235ms

I think this is because the regexes (https://github.com/buildkite-plugins/buildkite-plugin-linter/blob/master/lib/linters/example-linter.js#L80) are not supporting my I-think-legit example...?

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• Update dependency ajv to v8.11.2
• Update dependency chai to v4.3.7
• Update buildkite plugin docker-compose to v3.13.0
• Update Node.js to v19
• Update buildkite plugin docker-compose to v4
• Update dependency fs-extra to v11
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency minimatch to 3.0.5 [SECURITY]
• Update Node.js to a136ed7
• Update dependency compare-versions to v4.1.4
• Update dependency mocha to v9.2.2
• Update dependency isomorphic-git to v1.21.0
• Update dependency compare-versions to v5
• Update dependency eslint-plugin-promise to v6
• Update dependency mocha to v10
• Update dependency standard to v17
• Update dependency tap to v16
• Update dependency yargs to v17
• Click on this checkbox to rebase all open PRs at once

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

• Update dependency lodash to 4.17.21 [SECURITY]
• Update dependency js-yaml to v4

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

The linter does not recognize plugin examples that use the full git path instead of the ID.

Would be nice if we could pass the runtime configuration yaml and the plugin schema yaml to the linter in say a hook to validate if the correct configuration schema was passed. We currently have to validate this in the plugin itself. Or is there already a solution for this?

The README version checking algorithm creates a chicken-and-egg situation that may be problematic for some build processes. In particular, because it requires that the version is exactly the latest semver tag, unless the master build process (after PR merge) produces a new version and updates the README, or each commit within a PR is given the release semver, then either the PR build will fail or the master build will fail.

A flag should be provided to allow PRs to use a newer-than-latest semver in the README examples (changing the check to greater-or-equals instead of just equals).

For one of our plugin, we don't use the v prefix for our tags and versioning (i.e. we use tag named 1.2.3, not v1.2.3)

This breaks the plugin-linter when it tries to find an example in our README.md referencing the plugin. This is because the code of the linter explicitly expects the refMatch to start with #v, so our README example, while using the correct tag (the one really used in our repo, without the v), doesn't get detected by the linter.

For example, when the example in our README looks like this:

steps:
  - command: install_gems
    plugins:
      - automattic/a8c-ci-toolkit#2.16.0:
          bucket: a8c-ci-cache # optional

Then the linter fails with the following:

ok 1 - plugin.yml configuration is valid
not ok 2 - No readme config examples found with plugin id 'automattic/a8c-ci-toolkit'. Have you configured the linter with the correct plugin id?
ok 3 - Readme version numbers are up-to-date (2.16.0)

If I replace #2.16.0 with #v2.16.0 in the README example, then the linter becomes happy and goes all green… but that now makes our example incorrect, as our tag is 2.16.0, not v2.16.0.

We have our plugin example that pulls from SSH instead of HTTPS and the linter currently doesn't like this example in the README

steps:
  - plugins:
      - ssh://[email protected]/hoteltonight/jenkins-trigger-buildkite-plugin#v0.2:
          job_name: hotelstonight-docker-build
          job_token: hackme
    branches: master

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: Preset name not found within published preset config (monorepo:babel6). Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

E.g. a readme with:

```yml
steps:
  - command: test.sh
    plugins:
      - ./path/to/plugin: ~
```

and:

version: "3.4"
services:
  lint:
    image: buildkite/plugin-linter
    command: ['--id', './path/to/plugin']
    volumes:
      - ".:/plugin:ro"

does result in a:

  | ok 1 - plugin.yml is valid
  | not ok 2 - No readme config examples found with plugin id './path/to/plugin'. Have you configured the linter with the correct plugin id?
  | ok 3 - Readme version numbers are up-to-date (undefined)
  | 1..3

or am I holding it wrong?

If you specify plugin examples using the new array format, the linter fails to find them.