kubernetes-sigs / promo-tools

Container and file artifact promotion tooling for the Kubernetes project

Artifact Promotion Tooling

This repository contains a suite of tools responsible for artifact promotion in the Kubernetes project.

DISCLAIMER

Before getting started, it's important to note that the tooling stored here is an aggregate of several efforts to improve artifact promotion, across multiple SIGs, subprojects, and contributors.

We call that out to set the expectation that:

• you may see duplicated code within the codebase
• you may encounter multiple techniques/tools for accomplishing the same thing
• you will encounter several TODOs
• you will see gaps in documentation including:
  • missing documentation
  • example commands that may not work
  • broken links

This list is far from exhaustive.

If you encounter issues, please search for existing issues/PRs in the repository and join the conversation.

If you cannot find an existing issue, please file a detailed report, so that maintainers can work on it.

• DISCLAIMER
• kpromo
  • Installation
    • User
    • Developer
  • Usage
  • Image promotion
  • File promotion
  • GitHub promotion

kpromo

kpromo, or the Kubernetes Promotion Tool, is the canonical tool for promoting Kubernetes project artifacts.

It wraps and unifies the functionality of multiple tools that have existed in the past:

• cip
• cip-mm
• cip-auditor
• gh2gcs
• krel promote-images
• promobot-files

Installation

Requirements:

• Docker
• Go

User

If you're interested in installing kpromo from a tag:

go install sigs.k8s.io/promo-tools/v4/cmd/kpromo@<tag>
$(go env GOPATH)/bin/kpromo <subcommand>

Developer

If you're interested in actively contributing to kpromo or testing functionality which may not yet be in a tagged release, first fork/clone the repo and then run:

go install ./cmd/kpromo/...
$(go env GOPATH)/bin/kpromo <subcommand>

Usage

Usage:
  kpromo [command]

Available Commands:
  cip         Promote images from a staging registry to production
  completion  generate the autocompletion script for the specified shell
  gh          Uploads GitHub releases to Google Cloud Storage
  help        Help about any command
  manifest    Generate/modify a manifest for artifact promotion
  pr          Starts an image promotion for a given image tag
  run         Run artifact promotion
  version     output version information

Image promotion

For background on the image promotion process, see here.

To create an image promotion PR via kpromo pr, see here.

File promotion

See here.

GitHub promotion

See here.

promo-tools's Issues

add a -create-manifest flag

If we want to use the promoter against a prod registry that already has a bunch of images, we have to manually insert each image into the manifest. This would be extremely time consuming. So, we should add a -create-manifest flag that populates a manifest's images: .. key with all the images found at a dest registry (if there are multiple, then maybe the union of all of them).

We already have the functions that fetch all images and digests+tags from an entire GCR registry, so this
feature would just unmarshal that data and then print it out as YAML to a file. The other fields like registries: src: ... would be filled with dummy data, to be filled out manually by a human.

make -garbage-collect aware of manifest lists (aka "fat" manifests)

Currently the garbage collection logic just checks if a digest has no associated tags, and if so, deletes them. This doesn't account for manifest lists.

Thankfully, the gcloud call we use currently in version 234.0.0 crashes if we attempt to delete a digest that is referenced by a manifest list, so nothing happens (and we can use -delete-extra-tags safely).

The solution is to just use the Docker Registry API for this. Let's use the example of gcr.io/cip-demo-prod/coredns. The first call we can use is:

 $ curl -H "Authorization: Bearer $(gcloud auth print-access-token)" https://gcr.io/v2/cip-demo-prod/coredns/tags/list | jq
{
  "child": [],
  "manifest": {
    "sha256:02382353821b12c21b062c59184e227e001079bb13ebd01f9d3270ba0fcbf1e4": {
      "imageSizeBytes": "0",
      "layerId": "",
      "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
      "tag": [
        "1.3.1"
      ],
      "timeCreatedMs": "0",
      "timeUploadedMs": "1550296103137"
    },
    "sha256:613916bb25abd7c28d807e0695eed8c18b1b91555c9071bd742f5d3ba4b05dbe": {
      "imageSizeBytes": "11932168",
      "layerId": "",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "tag": [],
      "timeCreatedMs": "1547391218295",
      "timeUploadedMs": "1550296102546"
    },
    "sha256:638adb0319813f2479ba3642bbe37136db8cf363b48fb3eb7dc8db634d8d5a5b": {
      "imageSizeBytes": "12300583",
      "layerId": "",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "tag": [],
      "timeCreatedMs": "1547391212560",
      "timeUploadedMs": "1550296077349"
    },
    "sha256:7ce7f621749df5cd5f32b6daef6c192c66b067d4fde863f1d7004eeb8e148cae": {
      "imageSizeBytes": "10932753",
      "layerId": "",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "tag": [],
      "timeCreatedMs": "1547391216855",
      "timeUploadedMs": "1550296096170"
    },
    "sha256:d8d1f57070089244091c79002cf486b5108393062a0d7e6b43c4abfb77da7adb": {
      "imageSizeBytes": "11514698",
      "layerId": "",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "tag": [],
      "timeCreatedMs": "1547391214033",
      "timeUploadedMs": "1550296083921"
    },
    "sha256:eec40e97de27c11bf6840ece1a9a829e37e7a37ecae535e1907ec490aa829539": {
      "imageSizeBytes": "11136918",
      "layerId": "",
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "tag": [],
      "timeCreatedMs": "1547391215381",
      "timeUploadedMs": "1550296090208"
    }
  },
  "name": "cip-demo-prod/coredns",
  "tags": [
    "1.3.1"
  ]
}

to get all digests/tag information for an image path. Then we can identify if a digest is actually a manifest list (and not a regular digest) if it has "imageSizeBytes": "0". Then for each manifest list, we can figure out the child manifests that it references by doing a 2nd call:

 $ curl -H "Authorization: Bearer $(gcloud auth print-access-token)" https://gcr.io/v2/cip-demo-prod/coredns/manifests/sha256:02382353821b12c21b062c59184e227e001079bb13ebd01f9d3270ba0fcbf1e4    
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:638adb0319813f2479ba3642bbe37136db8cf363b48fb3eb7dc8db634d8d5a5b",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:d8d1f57070089244091c79002cf486b5108393062a0d7e6b43c4abfb77da7adb",
         "platform": {
            "architecture": "arm",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:eec40e97de27c11bf6840ece1a9a829e37e7a37ecae535e1907ec490aa829539",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:7ce7f621749df5cd5f32b6daef6c192c66b067d4fde863f1d7004eeb8e148cae",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 739,
         "digest": "sha256:613916bb25abd7c28d807e0695eed8c18b1b91555c9071bd742f5d3ba4b05dbe",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

The annoying/ugly part is that we need to migrate away from our simple dependence of gcloud, but that is probably inevitable anyway. AFAIK gcloud does not support querying about manifest lists (to figure out what images a manifest list references). The docker CLI does support this operation, but depending on the docker CLI is not desirable (when a library will do). Thankfully it appears that there is already a Go library that implements the Docker registry API here https://github.com/google/go-containerregistry/blob/52aa2729ea7597c8d5f7d6a977fa5cab9fc3a77a/pkg/v1/google/list.go#L69, so we could use that instead (hmm, but it doesn't appear to support distinguishing between a plain manifest and manifest list (see https://github.com/google/go-containerregistry/blob/52aa2729ea7597c8d5f7d6a977fa5cab9fc3a77a/pkg/gcrane/copy.go#L196 for the only usage of a DockerManifestList type in that repo).

We could just curl out instead --- it's the dumbest thing that will work.

k8s.io disaster recovery plan

Broad issue to track what our disaster recovery plan is if k8s.io registry somehow gets deleted.

One suggestion was creating a backup registry that snapshots k8s.io registry.

Move official home of promoter docker images to k8s.gcr.io or somewhere else more official than gcr.io/cip-demo-staging

Eventually when the promoter is hosted on k8s.gcr.io, we could have a promoter manifest for the promoter itself (i.e., build the promoter to the old gcr.io/cip-demo-staging location, but modify a promoter manifest to promote it into k8s.gcr.io).

This is more of a housekeeping thing and is not super urgent, mainly because the promoter Docker images are all built deterministically by Bazel (you could rebuild the promoter with the same Docker image digest at any time).

fix linter prow job

Currently the linter is failing because (probably) the golangci upstream image changed how it reads the ignore settings.

make the warnings about extra tags found in dest registries optional

See #81 (comment)

DockerHub as a src registry

After doing some research about how time consuming would be to implement DockerHub as a src registry I think there are three paths we can take.

1. Current implementation use gcloud CLI tool for work related registries, so we could create a simple, similar CLI tool with the same API related to commands we need (I think it's the quickest way)
2. Create an abstraction just for DockerHub without using any CLI tool, where it would coexist with current implementation for GCR (GCR would use gcloud tool and DockerHub would send direct requests to its REST API) [quite time consuming].
3. Create an abstraction from the ground up which will be communicating with any Docker Container Registry via REST API (it's the most time-consuming option, because we have to figure out how to handle different ways of auth and I assume your decision of use gcloud instead of just calling API was due of that)

I looked into: https://github.com/google/go-containerregistry which would be a nice tool to use, but there are no features related to registry like listing of repositories etc., so there would be need to implement them.

It would be good to talk about it, because maybe there are other ways I couldn't see. Irrespective of decision I would love to implement the changes. :-)

Bart

use spf13/cobra for multi-subcommand CLI option handling

At the very least, we can have 2 subcommands:

• promote
• snapshot

because they are independent of each other and do completely separate things. It would simplify some of the spaghettiness in cip.go.

NOTE: It is OK to make breaking changes with option handling because the Prow jobs that consume the cip tool only consume pre-built Docker images that reference a particular build. The only caveat here is that we won't be able to simply upgrade the Docker image version next time for these jobs as the calling convention will have to change (the same applies to util/multirun.sh).

make the promoter print out a JSON dump (summary) of everything that happened

This paves the way for other tooling to integrate with the promoter, and also simplifies what to look at when we parse the logs as humans after a promotion run (in Prow).

The logs are cumbersome to parse by human eyes, so at least printing out a (prettified) JSON at the very end of a promotion run could make it easier to see what went OK and what went wrong.

track golangci-lint in Bazel

For that matter, we should try to track all dependencies with Bazel.

Document service accounts and registries

It is unclear what permissions and purpose each service account and registry has. We should document this.

add a script to invoke the promoter multiple times against different manifests

Each time it is invoked, it should set activate the right service account.

Use klog everywhere for logging

Delete the (basic) homegrown stuff in https://github.com/kubernetes-sigs/k8s-container-image-promoter/blob/5fb3b400034911b811880310b2b45257874df6a6/lib/dockerregistry/inventory.go#L72

Makefile: use .bazelrc instead of BAZEL_BUILD_OPTS

Per @BenTheElder's comment in #1, instead of using BAZEL_BUILD_OPTS and passing it in, we can just use a .bazelrc and invoke bazel directly w/o any wrappers.

https://github.com/kubernetes/test-infra/blob/9a497cc7808fbb19cd85eb8caa1453732272745a/.bazelrc#L2

add API version info to promoter manifest

This way, the promoter can reject old manifests that are no longer compatible (in case we introduce any backward-incompatible changes to the promoter binary, and a human updates a Prow job that still uses an old manifest to be run against a newer (incompatible) promoter binary). We need to exit gracefully in such cases instead of possibly crashing.

So the Manifest YAML could have a new key like

apiVersion: v1

or something to that effect (similar to how Kubernetes uses API versioning itself for its Kubernetes objects).

add --delta flag to only act on the delta of 2 manifest files

Instead of giving the entire promoter manifest to the promoter for each run, we could give it 2 manifests: the version at HEAD and the one at a prior commit (or in the context of a PR, HEAD in master branch vs the version in the PR).

There are a couple advantages to this:
(1) There is no need to look at all images in the manifest (it's much faster, and also there's less room for error, because promoting only the handful of images in the PR is a lot easier than making sure every single image in the manifest is OK).
(2) Since we are just comparing the delta of 2 versions of the manifest, we can later more easily introduce the idea of 'demotions'. That is, you could just create a PR that is a git revert of an earlier PR that promoted an image, and the promoter should now be able to un-promote (demote) images. Currently demotion is done by combining promotion with garbage collection, but having knowledge of a delta allows the promoter to perform demotions without relying on garbage collection.

run s/registry/repository find-and-replace across the codebase and README

In practice, we only really deal with repositories, not registries (registries are unique top-level endpoints --- so in the case of GCR, the regional registries). The constant back/and/forth between these two terms in the codebase is rather confusing.

Manifest: do not force all entries in `registries` field to have a `service-account` field

Some registries are public so it does not make any sense to force them to have service-account fields.

Related logs (where we are trying to auth as [email protected]): https://prow.k8s.io/log?job=post-k8sio-cip&id=1114233301771489280

move CI logic into separate scripts

I.e., strip out steps from cloudbuild.yaml and elsewhere into standalone scripts.

GitHub Package Registry as src registry (staging registry)

The https://github.com/containers/skopeo project apparently can perform image copies across registries already. We should consider just using that binary to perform copies instead of using gcloud.

Related: #9

make the -garbage-collect flag optionally promote to a trash repo

Instead of simply deleting images outright for garbage collection (deletion of unreferenced images), we could also (in addition) promote the images to a trash repo. This way, we can feel a lot safer about deleting images. Another benefit is that we could priodically delete all images in the trash registry, if they are older than X days (super simple script --- just list all images by timestamps and then delete if older than X day --- this script could be run periodically by Prow).

The manifest YAML would need to have an extra field for describing the trash registry, probably like this:

registries:
  src: ...
  dest: ...
  trash: ... <--- new field

This would play out as follows:

• image foo:1.0 gets added into the promoter manifest
• promoter runs, and it gets added into the dest registry (or registries, as per #13)
• (some time passes, and foo:1.0 becomes deprecated)
• image foo:1.0 is removed from the manifest
• promoter runs, and now treats all instances of foo:1.0 as an extra tag to be deleted --- with the -delete-extra-tags flag on, it deletes them from all dest registries
• NEW: at this point, the promoter also promotes foo:1.0 into the new trash registry

support 'foreign' layers in manifest lists

https://github.com/docker/distribution/blob/master/docs/spec/manifest-v2-2.md#manifest-list

Currently this boils down to whether gcloud supports foreign layers because we have chosen to delegate all Docker manipulations to that tool. And because gcloud only works on GCR, it's a question of whether GCR supports foreign layers.

At the very least we can create an image with a foreign layer and try to test it being promoted in an e2e integration test.

allow tag-less promotions (promote images without tagging them)

Today the promoter will only promote those images that have tags in them. This feature will allow images to be promoted even if they don't have tags.

We could easily shell out to https://github.com/google/go-containerregistry/tree/master/cmd/gcrane#cp instead of the usual "gcloud container add-tag" command we use; the only annoying part here is that we would have to ship the gcrane binaries from the offiical gcr.io/go-containerregistry/gcrane path, but it should be doable. Alternatively, we could drop our reliance on gcloud container add-tag altogether and just lift the code already in gcrane to use the underlying API calls to GCR (we already import the go-containerregistry library that gcrane uses for other parts of our codebase). This latter approach is the better but obviously it might take more engineering effort.

This issue is a blocker for creating the backfilled promoter manifest for gcr.io/google-containers. That is, we want to have a (large) promoter manifest for all images in gcr.io/google-containers.

We already support the -snapshot flag which allows for snapshotting a registry (all images, including both tagged and tagless images); so this feature would allow for promotions of said snapshots as-is.

add windows support

Make sure that the promoter works from a windows machine. In theory golang is cross-platform (so is our dependency, gcloud), but it has not actually been tested.

Update test registry to use new test service account

See: kubernetes/k8s.io#302 (comment)

make the promoter act recursively for all manifests found under a directory

Instead of telling the promoter "here is the manifest, here is the SA, please do the promotion" (as we do with util/multirun.sh today), we should be able to say "here is a directory full of manifests and SA configuration info --- please perform promotions for all manifests found there". This is purely a convenience function, but very real and necessary.

Practically speaking, once this feature is implemented we could just add new staging repositories in github.com/kubernetes/k8s.io/k8s.gcr.io with a PR and that would be it --- the existing Prow job would just pick it up automatically (no need to update the Prow job to be invoked with the new staging repo manifest). It's a big win for convenience.

add linter as a required presubmit check for all new PRs

We already have the linting rules defined, so it's just a matter of adding another Prow job.

feature request: promote to multiple destination repos (aka "mirroring")

This way, we can simulate a "mirroring" operation. That is, copy to multiple destination registries instead of just 1.

For the manifest, instead of just accepting a YAML string for registries: dest: ..., it could accept an array of strings. And then we would run the copy (promotion) operations in parallel for each destination registry.

rename Go imports to use kubernetes-sigs GH org

Also update README.md instructions to reflect the new home.

e2e tests: dynamically fetch the image digests

Instead of hardcoding the digests in the test-e2e directory, dynamically calculate it on-the-fly, as a human would, for authoring the manifests and promoting images with them.

This is necessary because the image digests for the golden images depend on the gzipping mechanism that lies on the client machine (not server-side). So technically it is possible that the gzip library on the developer workstation (where the tests originated) differ from the gzip library where the e2e tests are executed later on, resulting in completely different digests (which would make every new PR fail).

This isn't a huge deal but it is a known issue.

The solution would be to fetch/calculate the image digests for all golden images used in the e2e tests after they are pushed up, and then to populate the yaml content with these values, before running them. Some amount of templating might be involved here as well.

TL;DR: remove hardcoded values like 2af5205553239e0eb17c544bc1e90c2d6173012a191f0a44131aa2f9c44bb511 from the e2e tests and instead generate them dynamically.

make json decoding boundaries more robust

We hit a transient error earlier today when trying to promote this manifest:

https://github.com/kubernetes/k8s.io/blob/1d14e8ec918ec2c71c30b6b0a06ee3ea9f724d01/k8s.gcr.io/k8s-staging-cluster-api/manifest.yaml

The error was:

https://storage.googleapis.com/kubernetes-jenkins/logs/post-k8sio-cip/1149377755071123456/build-log.txt

MULTIRUN: container image promoter version:
Built:   2019-04-23 03:13:59+00:00
Version: v2.0.1-0-g4db722e
Commit:  4db722e801f62b206f983d0a7adfad0fee99d463
MULTIRUN: gcloud version:
Google Cloud SDK 241.0.0
alpha 2019.04.02
app-engine-go 
app-engine-java 1.9.73
app-engine-python 1.9.85
app-engine-python-extras 1.9.85
beta 2019.04.02
bigtable 
bq 2.0.43
cbt 
cloud-datastore-emulator 2.1.0
core 2019.04.02
datalab 20190116
gsutil 4.38
kubectl 2019.04.02
pubsub-emulator 2019.04.02

MULTIRUN: running against k8s.gcr.io/k8s-staging-cluster-api/manifest.yaml
MULTIRUN: activating service account /etc/k8s-artifacts-prod-service-account/service-account.json
Activated service account credentials for: [[e-mail]]
********** START: k8s.gcr.io/k8s-staging-cluster-api/manifest.yaml **********
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x61750d]

goroutine 10 [running]:
encoding/json.(*Decoder).refill(0xc0004242c0, 0xc000056b01, 0x100c0001aa060)
	GOROOT/src/encoding/json/stream.go:159 +0xcd
encoding/json.(*Decoder).readValue(0xc0004242c0, 0x0, 0x0, 0x6f14e0)
	GOROOT/src/encoding/json/stream.go:134 +0x222
encoding/json.(*Decoder).Decode(0xc0004242c0, 0x68ff80, 0xc000574000, 0xc0005c0020, 0x950268)
	GOROOT/src/encoding/json/stream.go:63 +0x78
github.com/kubernetes-sigs/k8s-container-image-promoter/lib/dockerregistry.extractRegistryTags(0x0, 0x0, 0x0, 0x0, 0x1)
	lib/dockerregistry/inventory.go:744 +0x9f
github.com/kubernetes-sigs/k8s-container-image-promoter/lib/dockerregistry.getRegistryTagsFrom(0x6de800, 0xc00000ee80, 0x759860, 0xc000057cc0, 0x0, 0x0, 0x0, 0x0)
	lib/dockerregistry/inventory.go:433 +0x110
github.com/kubernetes-sigs/k8s-container-image-promoter/lib/dockerregistry.(*SyncContext).ReadRepository.func2(0xc000060900, 0xc000061200, 0xc00006a360, 0xc00012b710, 0xc00012b708)
	lib/dockerregistry/inventory.go:589 +0x793
created by github.com/kubernetes-sigs/k8s-container-image-promoter/lib/dockerregistry.(*SyncContext).ExecRequests
	lib/dockerregistry/inventory.go:713 +0x144
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x61750d]

goroutine 15 [running]:
encoding/json.(*Decoder).refill(0xc000214420, 0xc000056b01, 0x100c0001a80c0)
	GOROOT/src/encoding/json/stream.go:159 +0xcd
encoding/json.(*Decoder).readValue(0xc000214420, 0x0, 0x0, 0x6f14e0)
	GOROOT/src/encoding/json/stream.go:134 +0x222
encoding/json.(*Decoder).Decode(0xc000214420, 0x68ff80, 0xc000558000, 0xc000498020, 0x950268)
	GOROOT/src/encoding/json/stream.go:63 +0x78
github.com/kubernetes-sigs/k8s-container-image-promoter/lib/dockerregistry.extractRegistryTags(0x0, 0x0, 0x0, 0x0, 0x1)
	lib/dockerregistry/inventory.go:744 +0x9f
github.com/kubernetes-sigs/k8s-container-image-promoter/lib/dockerregistry.getRegistryTagsFrom(0x6de800, 0xc00000ee40, 0x759860, 0xc000057c80, 0xc0000a2858, 0xc00008ab28, 0x4121d3, 0x6af540)
	lib/dockerregistry/inventory.go:433 +0x110
github.com/kubernetes-sigs/k8s-container-image-promoter/lib/dockerregistry.(*SyncContext).ReadRepository.func2(0xc000060900, 0xc000061200, 0xc00006a360, 0xc00012b710, 0xc00012b708)
	lib/dockerregistry/inventory.go:589 +0x793
created by github.com/kubernetes-sigs/k8s-container-image-promoter/lib/dockerregistry.(*SyncContext).ExecRequests
	lib/dockerregistry/inventory.go:713 +0x144

Where the Decoder (not our code) choked on the io.Reader handle. I don't know how to read Golang traces but it appears that when we called decoder.Decode(&tags), that function paniced with a nil pointer dereference. It could have been that the io.Reader we passed along was for some reason invalid/empty. The line

github.com/kubernetes-sigs/k8s-container-image-promoter/lib/dockerregistry.getRegistryTagsFrom(0x6de800, 0xc00000ee80, 0x759860, 0xc000057cc0, 0x0, 0x0, 0x0, 0x0)

looks suspicious to me because the last 4 arguments are zeroed out (compared to the call found in goroutine 15 which has the last 4 addresses as non-zero values).

Anyway, we should add robustness checks so that we don't choke like this. One option (hacky) would be to read the io.Reader handle ourselves and write it to a buffer, then (optionally) log it, and then finally use this buffer directly in extractRegistryTags(). Then at least we would gain more interesting error logs the next time there is a nil pointer dereference in the JSON library code that we use. The disadvantage is that we lose the elegance of passing around file handles, but it's probably worth it.

@justinsb wdyt?

Also, pasting my slack comments below for reference:

listx  [13 minutes ago]
ok so rerunning it made it succeed. so either (1) there is a race condition in the ReadRepository code (probably not likely) or (2) there was an error with the stdout stream from GCR which made the json decoder choke (because that's what the trace said). i wonder if we can log the stdout if possible before we send it off to the json decoder (it's a library function). will add as an issue

listx  [12 minutes ago]
at least if we log the stdout before processing it, we'll be able to debug it better in the future

renames: ensure that rename destinations don't overlap across rename entries

A user could mistakenly do this in the manifest:

renames:
- ["gcr.io/foo/a", "gcr.io/bar/subdir/a"]
- ["gcr.io/foo/b", "gcr.io/bar/subdir/a"]

where both a and b will get renamed to subdir/a. We need to catch this as an error during the Manifest parsing phase.

guarantee that deletions never happen by splitting off tag deletion logic (*removal* of existing tags) to a separate tool

We can still keep around the tag deletion code, but we shouldn't do it as part of the same cip tool. This means retiring the -delete-extra-tags and -garbage-collect flags from the tool.

update documentation around *everything* related to the promoter (add playbook)

Describe how to fix common issues

• how to troubleshoot Prow job failures
• how to update Go packages
• how to update the Prow jobs (update Promoter docker image)

/assign @listx

validate digests and tags

If today we have a typo in the sha256 digest or Docker tag field, giving an invalid string is not caught during parsing. We just expect them to be stringlike things.

During manifest parsing, add a check to make sure sha256 digests are actually hex characters only, and that Docker tags meet the requirements of Docker tags described in https://docs.docker.com/engine/reference/commandline/tag/.

Do not make -delete-extra-tags imply -garbage-collect

This is because we want to separate promoter runs from garbage collection runs.

Separation of concerns ftw!

Build fails on macOS : date: illegal option -- -

With bazel 0.29, make build fails on master (93aef5c) on macOS with:

[...]
go: finding github.com/cpuguy83/go-md2man v1.0.4
go: error loading module requirements
DEBUG: /private/var/tmp/_bazel_gnormington/8ca8190ddc76e058023fc72217a03eec/external/bazel_gazelle/internal/go_repository.bzl:147:13: fetch_repo: go: finding golang.org/x/net v0.0.0-20190620200207-3b0461eec859
DEBUG: /private/var/tmp/_bazel_gnormington/8ca8190ddc76e058023fc72217a03eec/external/bazel_gazelle/internal/go_repository.bzl:147:13: fetch_repo: go: finding cloud.google.com/go v0.44.0
INFO: Analyzed target //:cip (51 packages loaded, 6762 targets configured).
INFO: Found 1 target...
ERROR: Process exited with status 1
date: illegal option -- -
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
            [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
Target //:cip failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 260.607s, Critical Path: 0.11s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
make: *** [build] Error 1

Create test Google service account

From: #62

@listx 
We should give [e-mail]
"Storage Admin" and "Storage Object Admin" perms for the test-only GCRs:
- gcr.io/k8s-staging-cip-test
- gcr.io/k8s-cip-test-prod

add script to dynamically generate multiple manifests on the fly

This is a short-term hack until #13 gets resolved in the Go code directly. Once #23 gets merged (serializing multiple runs of the promoter in 1 script), this script can reuse it.

The real-world use case for this script is that we want to be able to push to multiple regional GCRs.

Mac OS X is not supporting static binaries - it doesn't build on Mac

It seems like Mac OS X doesn't support static binaries: https://developer.apple.com/library/content/qa/qa1118/_index.html
so Bazel is failing when trying to build cip on MacOS

I think that we could implement simple fix for that, like kubectl: kubernetes/kubernetes@46e894b#diff-f5ebbdbdf8b4060ed24bc6ac0dcbad75

Let me create a simple PR for that, and if you won't agree with me, we can just close it. :-)

readme inconsistency related to manifests structure

Hi, in documentation there is example manifest:

src: gcr.io/myproject-staging-area
registries:
- name: gcr.io/myproject-staging-area
  service-account: [email protected]
- name: gcr.io/myproject-production
  service-account: [email protected]
images:
- name: apple
  dmap:
    "sha256:e8ca4f9ff069d6a35f444832097e6650f6594b3ec0de129109d53a1b760884e9": ["1.1", "latest"]
- name: banana
  dmap:
    "sha256:c3d310f4741b3642497da8826e0986db5e02afc9777a2b8e668c8e41034128c1": ["1.0"]
- name: cherry
  dmap:
    "sha256:ec22e8de4b8d40252518147adfb76877cb5e1fa10293e52db26a9623c6a4e92b": ["1.0"]
    "sha256:06fdf10aae2eeeac5a82c213e4693f82ab05b3b09b820fce95a7cac0bbdad534": ["1.2", "latest"]

and in 9514603 there was change of src to src-registry

create `--add-image` flag to modify an existing promoter manifest

If we want to edit the manifest today to include a new image (to get it promoted), we have to hand-edit the YAML. This is error-prone. We should create a new flag that can make edits to the manifest in a more automated way.

E.g., something like cip --add-image=gcr.io/foo/bar/blah:1.3.1 -manifest=foo.yaml should fetch the digest of that image, and add an entry in the promoter manifest foo.yaml like this:

images:
  - name: bar/blah
    dmap:
      ...
      <digest of 1.3.1>: ["1.3.1"]

This --add-image flag will lay the foundation for automatically creating promoter manifest PRs once images get built some registry gcr.io/foo. E.g., it could be integrated into any build system that pushes images (cloudbuild.yaml, Makefiles, etc).

Protecting promotion credentials

Broad issue to track how we're going to make sure the promotion credentials to the production registries will be secured against bad actors.

use Prow for CI

#1 adds support for using GCB, but Prow is much more powerful.

Then we can enable /lgtm commands and such. This also requires setting up an OWNERS file. See https://github.com/kubernetes/test-infra/tree/master/prow/plugins/approve/approvers

Create e2e test for use in CI (to actually promote/demote images in Docker Registries)

The idea is, we can create a test (as a Prow Job) to populate 2 test GCRs --- a fake "staging" and a fake "production" GCR. Then we would rotate through 1 or more different promoter manifests, with various flags passed to the promoter, and expect the target registry to look a certain way after the promoter finishes.

We could do the above multiple times in the test. This would be great to test things like garbage collection and whatnot, and other more advanced features of the promoter that might be implemented in the future (e.g., multi-stage promotions, replication, etc.).

The basic ingredients are:

• Have some logic to create real Docker images deterministically (bazel already does this...)
• Populate a staging GCR with these images
• Run a just-compiled promoter (as a presubmit) and run it against a battery of various promoter manifest.yamls.

All of the above could run in a Prow step. It would simply fetch an image with gcloud + bazel, then compile the promoter and then run it.

The integration test would enable developers (who might not have a Docker Registry to play with or have GCP accounts set up) to make changes (such as refactorings) without fear of modifying the behavior of the promoter.

The original idea of the various components of how an integration test might work is documented here.

add --(re)strict-namespace flag to force images to be promoted within a particular namespace

E.g., images coming from gcr.io/foo-project/... should only be able to be promoted as k8s.gcr.io/foo-project/.... That is, the top-level folder name should match 1:1.

This check would kick in after we do the rename logic (it's still possible to rename images for production); it's just that with the flag turned on we would perform an extra check prior to promotion.

This feature could also be enforced as a field in the manifest YAML, but it's better to keep it as a flag-only thing because we don't want individual promoter manifest owners to change this setting (the binary invocation of the promoter will sit in a tightly-controlled Prow job config).

allow DRY-er renames

During promotion, we might need to rename images to multiple repositories, but the current renames: syntax in the Promoter Manifest forces us to specify each repository (any unlisted repository will not use the renamed image path).

E.g. if we have the {eu,asia,us}.gcr.io regional destinations, then the renames would have to look like this in the manifest (as it is implemented today):

renames:
- ["gcr.io/foo/a", "eu.gcr.io/foo/subdir/a", "us.gcr.io/foo/subdir/a", "asia.gcr.io/foo/subdir/a"]

which is explicit but could be seen as verbose. Instead we could support either globbing or brace expansion to take care of such scenarios, like this:

renames:
- ["gcr.io/foo/a", "*/subdir/a"]

where the * means "all destination registries", or

renames:
- ["gcr.io/foo/a", "{eu,us,asia}.gcr.io/foo/subdir/a"]

The globbing part could alternatively be done inside an image's entry instead of the toplevel renames: field. E.g., we could have

images:
- name: "a"
  dmap: ...
  rename: "subdir/a"

and that would be functionally equivalent to having a toplevel ["gcr.io/foo/a", "*/subdir/a"].

allow remapping paths during promotion (do not force src registry to share same namespace as dest registry)

Currently we expect the source image to have the exact same path as the destination path. This limits flexibility. We want to be able to optionally specify the target path in the destination, to allow it to be different than the source path.

This will require a slight tweak to the promoter manifest YAML.

update promoter docker image version to master HEAD

Also, update the Prow jobs that reference the promoter to use the new version.

known issue for k8s.io/kubernetes/pkg/credentialprovider

It's probably due to our reliance on go-containerregistry, but currently we get this gazelle error during a fresh build of make for the k8s.io/kubernetes/pkg/credentialprovider import:

...
go: k8s.io/[email protected]: unknown revision v0.0.0
go: error loading module requirements
gazelle: finding module path for import k8s.io/kubernetes/pkg/credentialprovider: exit status 1: go: finding k8s.io/kubernetes/pkg/credentialprovider latest
...

It's probably related to google/go-containerregistry#496

This is just a tracking issue for the upstream error.