get image manifest from emvedded registry about k3s

[brandond]

The embedded registry mirror functions only as a mirror. Your webhook would need to support mirrors, and in particular be able to pass the original registry namespace as a parameter to the registry API so that it knows what upstream registry (docker.io, ghcr.io, etc) to provide images for. Containerd and other packages that have first-class support for mirrors all do this.

As described at https://docs.k3s.io/installation/registry-mirror#authentication the embedded registry mirror also requires client certificate authentication. Your webhook would need a client certificate signed by the the cluster client CA in order to connect.

I took a quick look at the vault docs and I couldn't find any references as to what they're using to inspect the images, or whether or not it supports mirrors. I suspect it does not currently support what you're trying to do. You might open an issue against their repo asking for:

• registry mirror support
• registry client certificate authentication support

[johnny990]

The embedded registry mirror functions only as a mirror. Your webhook would need to support mirrors, and in particular be able to pass the original registry namespace as a parameter to the registry API so that it knows what upstream registry (docker.io, ghcr.io, etc) to provide images for. Containerd and other packages that have first-class support for mirrors all do this.

I'm not sure, I got your point about "support mirrors"? This webhook just lookups at the 'image' in k8s object definition and tries to get metadata for it. The ratio of doing it that it substitutes entrypoint and/or command defined in the image by prefixing its own command. Let's say, the mutation de facto is command: "echo hello" -> command: "/vault/vault-env echo hello".
So, to achieve this it calls /v2/* endpoint of registry to get manifest to extract command or entrypoint

As described at https://docs.k3s.io/installation/registry-mirror#authentication the embedded registry mirror also requires client certificate authentication. Your webhook would need a client certificate signed by the the cluster client CA in order to connect.

So, did I read you correctly that embedded mirror doesn't provide any kind of api and just provides image as it is stored locally (pushed via ctr image load for example)?

I took a quick look at the vault docs and I couldn't find any references as to what they're using to inspect the images, or whether or not it supports mirrors. I suspect it does not currently support what you're trying to do. You might open an issue against their repo asking for:
* registry mirror support
* registry client certificate authentication support

I've already put some details how it works above and actually some time ago I've opened ticket there with suggestions how to make it more flexible in terms of handling registries.

But actually, I've found the workaround here. As I said the only reason this webhook wants manifest from registry is to figure out command/entrypoint to mutate it, so my workaround is to provide this data to it by adjusting deployment templates explicitly adding command there. And it worked :)

So, I guess this issue could be closed and thank you for looking at it!

[brandond]

I'm not sure, I got your point about "support mirrors"? This webhook just lookups at the 'image' in k8s object definition and tries to get metadata for it.
So, to achieve this it calls /v2/* endpoint of registry to get manifest to extract command or entrypoint

Yes, exactly. You would need to be able to tell it to use one or more mirror endpoints. So for example, instead of making a request to https://docker.io/v2/..., for an image on docker.io, it should try the mirror endpoint of https://mirror.example.com/v2/...?ns=docker.io. This is how mirror support works in containerd, docker, and other products that support container image registry mirroring.

If you have the ability to change all your pods so that it doesn't need to inspect the image, that sounds like a good workaround until mirror support is added.

[johnny990]

Yes, exactly. You would need to be able to tell it to use one or more mirror endpoints. So for example, instead of making a request to https://docker.io/v2/..., for an image on docker.io, it should try the mirror endpoint of https://mirror.example.com/v2/...?ns=docker.io. This is how mirror support works in containerd, docker, and other products that support container image registry mirroring.

I guess, since it is air-gapped environment there's no point to add support for mirrors for this use case, because it should either have an ability to work with image files (unpack it, call some external command like docker inspect, dunno what else) or just have an option to get this metadata in some offline way (env/settings/you name it).