mschuett / yaml-shellcheck

Wrapper script to run shellcheck on YAML CI-config files

yaml-shellcheck

Wrapper script to run shellcheck on YAML CI-config files. Currently supported formats are Bitbucket Pipelines, GitLab CI, GitHub Actions, Drone CI, CircleCI, and (very limited) Ansible

Usage

Shell

Needs Python 3 with library ruamel.yaml, and shellcheck.

$ ./yaml_shellcheck.py -h
usage: yaml_shellcheck.py [-h] [-o OUTDIR] [-k] [-d] [-s SHELL] [-c COMMAND] files [files ...]

run shellcheck on script blocks from .gitlab-ci.yml or bitbucket-pipelines.yml

positional arguments:
  files                 YAML files to read

optional arguments:
  -h, --help            show this help message and exit
  -o OUTDIR, --outdir OUTDIR
                        output directory (default: create temporary directory)
  -k, --keep            keep (do not delete) output directory
  -d, --debug           debug output
  -s SHELL, --shell SHELL
                        default shebang line to add to shell script snippets (default: '#!/bin/sh -e')
  -c COMMAND, --command COMMAND
                        shellcheck command to run (default: shellcheck)

Exit Status/Return Values

The tool exits with the exit code from shellcheck, from their man page:

ShellCheck uses the follow exit codes:
• 0: All files successfully scanned with no issues.
• 1: All files successfully scanned with some issues.
• 2: Some files could not be processed (e.g. file not found).
• 3: ShellCheck was invoked with bad syntax (e.g. unknown flag).
• 4: ShellCheck was invoked with bad options (e.g. unknown formatter).

Docker

# build image
$ docker build . -t yaml_shellcheck:latest
# run image
$ docker run -v `pwd`:/app yaml_shellcheck app/*.yaml

GitLab CI

lint_yaml_shellcheck:
  image:
    name: quay.io/mschuette/yaml-shellcheck:latest
    entrypoint: [""]
  script:
    - find . -name \*.yaml -or -name \*.yml | xargs python3 yaml_shellcheck.py

File Formats

The main function of this tool is to encode which elements inside the YAML data contain shell scripts. So far three formats are supported.

Bitbucket Pipelines

Handling Bitbucket files is very simple. A file with a pipelines object is read as a Bitbucket Pipeline file, and every script attribute inside is considered a shell script.

GitHub Actions & Forgejo Actions

GitHub Actions are similar to Bitbucket. A file with a jobs object is read as a GitHub Actions file, and every run attribute inside is considered a shell script.

As far as I can tell Forgejo Actions (as used e.g. by https://codeberg.org/) intentionally use the same structure as GitHub Actions, so these are covered here as well.

• shell attributes are not supported
  this is a todo, it should be simple enough to only check sh and bash scripts with right shebang line
• expressions get replaced with a simple variable before running shellcheck

Drone CI

Drone CI has a very simple file structure. As far as I can tell it only has conditions, but no deep nesting or includes. All command lists are concatenated and checked as a shell script.

CircleCI

CircleCI has many options, but fortunately the nesting of its jobs.*.steps.run script elements is straightforward. All command lists are concatenated and checked as a shell script.

• shell attributes are supported
• pipeline values and parameters are ignored and replaced with placeholder shell variables

GitLab CI Pipelines

GitLab CI files have more structure, and we try to support more of it.

• script, before_script, after_script: GitLab has not one, but three different shell script attributes which are read as three independent scripts. In GitLab before_script and script get concatenated and executed in a single shell process, whereas after_script always starts a new shell process. This has some implications for variable visibility etc. and is ignored in this tool.
• include is not supported, every YAML file is parsed on its own. For large pipelines with many includes you may want to use other tools to resolve all include and then run yaml_shellcheck on the merged YAML file.
• !reference is semi-supported, we only read the tag and insert a placeholder in order not to break the YAML parsing (todo, should be simple to improve).
• decided to not support variables checking. shellcheck checks lower-case variables for assignments before usage, but it assumes all upper-case variable names are externally provided. -- IMHO GitLab CI jobs should follow that convention and use upper-case variable names for CI/CD variables.

Ansible

Ansible support is limited. This tool reads Ansible playbook or task files with YAML lists.

So far it recognizes three kinds of list elements:

• Tasks using the shell (or ansible.builtin.shell) module.
• Playbook elements, containing a tasks attribute.
• Blocks (block) for nested task lists.
• Jinja expressions ({{ ... }}) are ignored and replaced with placeholder shell variables.

Common

Files are read with a YAML parser, so all YAML anchors are resolved. There is no additional check on data types or structure.

Following the Bitbucket/GitLab usage a script block may contain a string or an array of strings.

yaml-shellcheck's Issues

Feature request: check Github Action files too (not just Github Workflow files)

https://docs.github.com/en/actions/sharing-automations/creating-actions/creating-a-composite-action#creating-an-action-metadata-file

Naive implementation: lukaw3d@48e1ccc
I don't know if this conflicts with any other format; and it checks if inputs field exists, even tho it is optional

Quay.io issues, missing 1.2.0 image

Somehow the quay.io integration and build pipeline was is broken and the v1.2.0 release did not even start an image build pipeline.

I "tried turning it off and on again", and now it should be working again.
New images visible at https://quay.io/repository/mschuette/yaml-shellcheck?tab=tags, among them quay.io/mschuette/yaml-shellcheck:v1.2.0 which is now the new latest.

Bug: using absolute paths throws FileExistsError

$ docker run -v `pwd`:/app2 --rm quay.io/mschuette/yaml-shellcheck /app2/test-input/github-workflow.yml

2024-08-14 02:18:05,957 - root - INFO - read /app2/test-input/github-workflow.yml as GitHub Actions config...
Traceback (most recent call last):
  File "/app/yaml_shellcheck.py", line 459, in <module>
    main()
  File "/app/yaml_shellcheck.py", line 448, in main
    filenames.extend(write_tmp_files(args, result))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/yaml_shellcheck.py", line 402, in write_tmp_files
    subdir.mkdir(exist_ok=True, parents=True)
  File "/usr/local/lib/python3.12/pathlib.py", line 1311, in mkdir
    os.mkdir(self, mode)
FileExistsError: [Errno 17] File exists: '/app2/test-input/github-workflow.yml'

or

$ docker run -v `pwd`:`pwd` --rm quay.io/mschuette/yaml-shellcheck  `pwd`/**/*.y*ml
2024-08-14 02:32:29,902 - root - INFO - read /home/luka/Downloads/yaml-shellcheck/test-input/ansible-playbook.yml as Ansible file...
Traceback (most recent call last):
  File "/app/yaml_shellcheck.py", line 459, in <module>
    main()
  File "/app/yaml_shellcheck.py", line 448, in main
    filenames.extend(write_tmp_files(args, result))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/yaml_shellcheck.py", line 402, in write_tmp_files
    subdir.mkdir(exist_ok=True, parents=True)
  File "/usr/local/lib/python3.12/pathlib.py", line 1311, in mkdir
    os.mkdir(self, mode)
FileExistsError: [Errno 17] File exists: '/home/luka/Downloads/yaml-shellcheck/test-input/ansible-playbook.yml'

This happens because

yaml-shellcheck/yaml_shellcheck.py

Line 397 in 2608670

subdir = outdir / filename

is just subdir = filename for absolute paths.

Naive fix

lukaw3d@8df5d64

$ docker run -v `pwd`:/app2 --rm ghcr.io/lukaw3d/yaml-shellcheck:main /app2/test-input/github-workflow.yml
2024-08-14 02:29:29,192 - root - INFO - read /app2/test-input/github-workflow.yml as GitHub Actions config...

In a/app2/test-input/github-workflow.yml/jobs/hello_world_job/steps/2/run line 2:
echo random-number $ACTION_EXPRESSION
                   ^----------------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
echo random-number "$ACTION_EXPRESSION"

For more information:
  https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...

Gitlab spec:inputs are breaking the script

I am using this for linting in some Gitlab pipelines and just noticed that on newer jobs which use spec:inputs it's breaking since it's technically two YAML files in one.

More information about spec:inputs:
https://docs.gitlab.com/ee/ci/yaml/inputs.html

Error:

Traceback (most recent call last):
  File "/root/yaml-shellcheck/yaml_shellcheck.py", line 440, in <module>
    main()
  File "/root/yaml-shellcheck/yaml_shellcheck.py", line 427, in main
    result = {filename: read_yaml_file(filename)}
  File "/root/yaml-shellcheck/yaml_shellcheck.py", line 368, in read_yaml_file
    data = yaml.load(f)
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/main.py", line 451, in load
    return constructor.get_single_data()
  File "/usr/local/lib/python3.9/site-packages/ruamel/yaml/constructor.py", line 114, in get_single_data
    node = self.composer.get_single_node()
  File "_ruamel_yaml.pyx", line 718, in ruamel.yaml.clib._ruamel_yaml.CParser.get_single_node
ruamel.yaml.composer.ComposerError: expected a single document in the stream
  in "./Jobs/Trigger.gitlab-ci.yml", line 13, column 1
but found another document
  in "./Jobs/Trigger.gitlab-ci.yml", line 52, column 1

Very cut down example pipeline:

spec:
  inputs:
    target:
      type: string
      description: The Target URL
---
trigger:
  stage: .post
  image: $CI_REGISTRY/pipeline-components/helperimages:alpine-utils
  script:
    - |
      curl --location $[[ inputs.target ]]