ImageAutomation: Custom TLS certs for container registries about image-reflector-controller

[squaremo]

Quick design:

apiVersion: image.toolkit.fluxcd.io
kind: ImageRepository
spec:
  image: registry:8080/app-image
  interval: 5m
  certificateSecretRef:
    name: registry-certs
---
apiVersion: v1
kind: Secret
type: kubernetes.io/tls
stringData:
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    MIICojCCAYoCCQC2/92/4xw9wjANBgkqhkiG9w0BAQsFADATMREwDwYDVQQDDAhy
    ...
    k2vUMz8m
    -----END CERTIFICATE-----
  tls.crt: |
    -----BEGIN CERTIFICATE-----
    MIICmjCCAYICCQCaeupWLclBOzANBgkqhkiG9w0BAQsFADAPMQ0wCwYDVQQDDARm
    ...
    tNVsKCye54yh1rzTuadBa/I+myVpq+ynPp39k+F0fjVodOwyiTrzpHejyxHi9g==
    -----END CERTIFICATE-----
  tls.key: |
    -----BEGIN RSA PRIVATE KEY-----
    MIIEogIBAAKCAQEAxx3+wgas5lkzRilKR/4IG98FNGjQNJHD1KlVYQH8+NCN1veh
    ...
    5tQ7MDVezip26NpQgZ07oZ9O8o28nONy9AgNmAFifeMIqZT085o=
    -----END RSA PRIVATE KEY-----

An alternative would be to look for the certs in the secret referred to with secretRef; however, those are often generated with kubectl create secret docker-regsitry ..., and with automation, and either of those makes it inconvenient to include certificates. You can always name the same secret, if you put the certificates there.

When a certificate secret is included, the controller will look for those two fields, parse and use which of them is present for HTTPS requests. So you could include a CA certificate if you want to use TLS but don't need to authenticate with a client certificate; or, you could use a client certificate if you are authenticating with the certificate, but your server has a fancy CA-signed certificate itself.

The format is the one produced by kubectl create secret tls, except for the additional ca.crt. The controller will not require the type to be set.

EDIT: I just noticed kubectl create secret tls.

[squaremo]

@bjornmagnusson flux v1's --registry-insecure-hosts would try HTTPS with TLS_INSECURE_SKIP_VERIFY, and failing that, use HTTP. I guess you used it for that first part, because you were using your own CA cert?
Supplying a custom client- and CA cert would be strictly better, because they would not bring TLS_INSECURE_SKIP_VERIFY along with them -- so I want to make sure that you don't need the other bits of --registry-insecure-hosts.

[stefanprodan]

We could use a shorter field name like certSecretRef.

[stefanprodan]

@squaremo I would consider using the same key names we currently use for HelmRepositories https://toolkit.fluxcd.io/components/source/helmrepositories/#spec-examples

apiVersion: v1
kind: Secret
type: Opaque
data:
  certFile: <BASE64>
  keyFile:  <BASE64>
  caFile:   <BASE64>

[squaremo]

I would consider using the same key names we currently use for HelmRepositories

Does HelmRepository eschew the format used by kubectl create secret tls for the same reason I initially did? (to wit: I didn't know about it)

[stefanprodan]

Those keys names are part of the Helm API used for defining the repositories.yaml, I think this API predates kubectl create secret tls.

[squaremo]

Those keys names are part of the Helm API used for defining the repositories.yaml, I think this API predates kubectl create secret tls.

Got it, they are from a parallel tradition :-) I guess I could look for either set of keys; I don't expect people to be creating secrets for this in the same way they'd create them for Helm; but it might avoid some surprises, so why not.

[stefanprodan]

As with Git and Helm we will provide a flux create secret tls command that will generate a secret with certFile, keyFile and caFile keys. We already do that for Helm so adding the tls command should take 5 minutes of copy/paste :))

[stefanprodan]

@squaremo if we standardise around kubernetes.io/tls for ImageRepository then how should we go about GitRepository fluxcd/source-controller#93? Should we introduce a separate certSecretRef field there too?

[squaremo]

There's problems either way around:

• a) secrets created with kubectl create secret tls don't have a ca cert -- you would have to add that yourself if you need it
• b) the secrets for Helm etc. include username/password, which isn't necessary here since those are expected to come from an imagePullSecret

I think it makes more sense for Git and Helm to use their own format. Image repos are a special case because they use imagePullSecrets -- but because of a) above, I'm going to expect the Git/Helm format for certs, and support TLS secrets as a convenience only.

[stefanprodan]

You're right, both Helm and Git-over-https include username/password.

I'm going to expect the Git/Helm format for certs, and support TLS secrets as a convenience only.

Sounds great, thank you.

[bjornmagnusson]

Looks promising.
I wonder if there is any possibility that this kind of configuration can be done on a global level for the entire cluster? (as it applies to all automations in that cluster).

[bjornmagnusson]

Let me know when there is an version of image-reflector-controller to try out, and I will be happy to do it in our context.

[squaremo]

@bjornmagnusson This issue got automagically closed because I mentioned it in the pull request; the TLS certs bit of API should make its way into the next release, which will surely be soon!

[squaremo]

I'm going to expect the Git/Helm format for certs, and support TLS secrets as a convenience only.

In the end I just implemented secrets with the flux create secret data keys (and see fluxcd/flux2#788).

I could add kubectl create secret tls support in another PR. What do you think @bjornmagnusson, would you expect to be able to use kubectl create secret tls?