markdumay / dbm

Simplify the Development and Testing of Docker Images (work in progress)

Hi there! I'm Mark Dumay

Mark Dumay is managing partner and owner of Squadra Analytics, a boutique consulting firm based in the Netherlands. With nearly 20 years of experience, Mark has helped numerous clients in Utilities, Telecom, and Wholesale to get more value from data and analytics.

Mark holds a Master in Computer Science from Delft University of Technology and is a certified Lean Six Sigma Black Belt. He maintains several open-source repositories on GitHub focused on Hugo, Synology, and Docker.

dbm's Issues

Support custom/additional registries

Currently only two registry providers are supported. Additional registries should be supported via flexible plugins.

Support configurable templates

Adds support to define additional templates and to identify which template(s) to use in the current repository.

Add Docker Hub integration test

Docker - work in progress:

• spec/lib_docker_spec.sh: build_cross_platform_image() builds a cross-platform development image FAILED
• spec/lib_docker_spec.sh: push_image() pushes a specific image WARNED

Expose image digest as environment variable

Track SHA digests for each version

The check command currently identifies if upgrades are available for tracked dependencies. However, DBM is not aware of any changes made to an existing (version) tag. Tracking SHA digests and comparing them with the latest available SHA would enable identification of any changes.

Docker Hub
The method to retrieve image digests differs for multi-architecture images and regular images. Use the following code snippet to retrieve the repository digest, which equals the image digest of a multi-architecture image. The endpoint requires an authorization token.

readonly DOCKER_AUTH='https://auth.docker.io'
readonly DOCKER_API='https://hub.docker.com/v2'
readonly DOCKER_REGISTRY_DOMAIN='registry.docker.io'
readonly DOCKER_MANIFEST_HEADER='application/vnd.docker.distribution.manifest.list.v2+json'

# Retrieve authorization token for targeted repository
token=$(curl -sSL "${DOCKER_AUTH}/token?service=${DOCKER_REGISTRY_DOMAIN}&scope=repository:${owner}/${repository}:pull" \
    | jq --raw-output .token)

# Request a "fat manifest" by default, HEAD only
response=$(curl --HEAD -sH "Authorization: Bearer ${token}" \
    -H "Accept: ${DOCKER_MANIFEST_HEADER}" "https://registry-1.docker.io/v2/${owner}/${repository}/manifests/${tag}")

Use below code to retrieve the digest of a regular image.

digest=$(curl -s "${DOCKER_API}/repositories/${owner}/${repository}/tags/${tag}" | jq -r '.images[0].digest')

GitHub
The API endpoint for GitHub is https://api.github.com/repos/{owner}/{repo}/tags. Match the tag name with the desired version and extract commit.sha to get the SHA of the release. As an example, calling https://api.github.com/repos/fsnotify/fsnotify/tags returns the following data:

[
  {
    "name": "v1.4.9",
    "zipball_url": "https://api.github.com/repos/fsnotify/fsnotify/zipball/v1.4.9",
    "tarball_url": "https://api.github.com/repos/fsnotify/fsnotify/tarball/v1.4.9",
    "commit": {
      "sha": "45d7d09e39ef4ac08d493309fa031790c15bfe8a",
      "url": "https://api.github.com/repos/fsnotify/fsnotify/commits/45d7d09e39ef4ac08d493309fa031790c15bfe8a"
    },
    "node_id": "MDM6UmVmMjEzMDY5NjE6cmVmcy90YWdzL3YxLjQuOQ=="
  },
]

Add support for semver ranges and comparators

See for example: https://github.com/qzb/sh-semver

Fix tag override from command line

Support multi-architecture builds

Support local registries

Currently only public Docker registries are supported. Local registries, including credentials if needed, should be supported too.

Add init command

The init command is to initialize a new local repository using provided templates. Init is to support a service flag, so placeholders are generated for each service. The init command fails if the local repository is not empty, or if the specified services are already defined.

Enable signing of image builds

Add scan command

The scan command conducts a vulnerability scan of the defined images.

Support authentication to increase API rate limits

Both Docker Hub and GitHub enforce rate limits when using their API. To increase the rate limit, DBM should support authentication for all API requests. Next to that, an option to check the current rate limit status would allow for more informative error messages. An item should be added to the FAQ to explain the setup.

Type	Anonymous	Authenticated	Remarks
Docker pull	100 pulls/6 hours	200 pulls/6 hours	Use a cache such as mirror.gcr.io to reduce traffic
GitHub request	60 requests/hour	5.000 requests/hour	Use GITHUB_TOKEN in GitHub Actions to enable 1.000 requests/hour per repository

Add unit tests

Status:

• 303 unit tests defined
• 5 unit tests pending

Pending unit tests:

• lib_docker_spec.sh: sign_image()
• lib_trust_spec.sh: add_repository_signer()
• lib_trust_spec.sh: import_delegation_key()
• lib_trust_spec.sh: init_notary_config()
• lib_trust_spec.sh: remove_repository_signer()

Compatibility:

• Ubuntu 16.04, 18.04, 20.04 (Xenial Xerus, Bionic Beaver, Focal Fossa)
  • dash
  • bash
  • busybox ash
  • ksh
  • mksh
  • posh
  • yash
  • zsh
• macOS 10.15 (Catalina)
  • sh
  • dash
  • bash
  • ksh
  • zsh
• macOS 10.15 (Catalina / Homebrew)
  • dash
  • bash
  • ksh
  • mksh
  • yash
  • zsh

Warn for undefined dependencies

Running check against a dbm.ini without any dependencies results in an error [] Malformed, skipping. It should be replaced with WARN No dependencies found.

Add push command

The push command pushes an image to the Docker registry, including correct tags.

Warn for missing variables