Container registry auth not found in secret about image-reflector-controller

[from-nibly]

Oh wow. I don't know how I didn't see that! Everything is working after re-bootstrapping.

Thank you for your help!

[somtochiama]

It means you dont have the right Docker server(ghcr.io) in the secret.
Can you paste the command you use to create the secret?

[asosso]

I've the same issue using ECR repository.
Probably the registry is not matched by the controller if there is https://in the secret.

Message: auth for "XXXXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com" not found in secret mynamespace/ecr-docker-login
In the same namespace there is the ecr-docker-login secret. The same secret it works to pull the container.

My .dockerconfigjson base64 decoded starts with {"auths":{"https://XXXXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com":{"auth": ...

Following this guide https://toolkit.fluxcd.io/guides/image-update/#using-access-token-short-lived there isn't https:// in the generated secret: {"auths":{"XXXXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com":{"username":"AWS" ...

I think is related to ref.Context().RegistryStr() parameter: https://github.com/fluxcd/image-reflector-controller/blob/main/controllers/imagerepository_controller.go#L191

[from-nibly]

here is the base64 decoded version of the secret (redacted of course)

{"auths":{"https://ghcr.io":{"username":"xxxxxxx","password":"xxxxxx","auth":"xxxxxxx"}}}

and here is the entire command I ran (also redacted)

kubectl create secret -o yaml -n flux-system docker-registry github --dry-run --docker-server=https://ghcr.io --docker-username=xxxxxx --docker-password=xxxxxxx |  kubeseal --format=yaml --controller-name=kubeseal-sealed-secrets --controller-namespace=kubeseal --context=xxxx > ./clusters/local/flux-system/docker-secret.yaml

I'm realizing I might need to try without the schema in the server? for some reason this works just fine to help kubernetes download docker containers.

[from-nibly]

OK I think that got me farther. It looks like the issue now is this error on the image-automation-controller. unknown error: ERROR: The key you are authenticating with has been marked as read only. I believe that is github responding to a request that the pod is making. Is it using the ssh key from bootstrapping? How would it get access to the repo through a read only key when the github PAT used to bootstrap had full repo permissions? It's possbile that I initially bootstraped the cluster with a read only PAT but I've since rebootstrapped with a new PAT that has full repo permissions.

[relu]

@from-nibly did you happen to bootstrap flux using --token-auth? Otherwise, flux will create a deploy key in your git repository settings which is read-only by default. You can check this under https://github.com/<org>/<repo>/settings/keys. You can read about that in the docs. As you can see in the guide, the example there uses --token-auth.

If you do use a deploy key and it is read-only you will not be able to convert it to a read-write key, instead, you will have to delete and re-create the key, the value of which you can get from the secret flux created during bootstrap.

[squaremo]

Sounds like you are good to go @from-nibly -- I'll close this issue. For posterity, here's some detailed instructions on how to replace the read-only git secret at https://toolkit.fluxcd.io/guides/flux-v1-automation-migration/#replacing-the-git-credentials-secret.