[Bug]: kubernetes authentication method failing on AWS EKS about flipt

[GeorgeMac]

Thanks for raising this @tstraley ! It has been a while since I implemented this, so taking me a hot minute to rebuild my context 😂 bare with me on this one.

Just adding a bunch of context off the top of my head:

If I remember correctly, the bit we have to workaround (the issuer mismatch) is simply that we instruct the go-oidc library to not return an error when the issuer described by the discovery endpoint does not match the URL we used to request that document. This is where the oidc.InsecureIssuerURLContext comes in:
https://pkg.go.dev/github.com/coreos/go-oidc/v3/oidc#InsecureIssuerURLContext

The go-oidc library will return an error after it gets the discovery well-known endpoint if the host used to fetch that does not match the issuer URL in the response. We use the local k8s DNS address to get the discovery document, but it returns a JWKS URL and issuer that does not match that same local k8s DNS name.

However, the go-oidc library will still use JWKs, which is what we're seeing here I believe.

I think this is a form of this issue: aws/containers-roadmap#2234
And it seems related to how EKS it set up to distribute service accounts for IAM roles via its own OIDC provider:
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html#irsa-oidc-background

This issue linked suggests a work-around for EKS is to actually start the discovery journey using the custom EKS OIDC address (e.g. https://oidc.eks.<region>.amazonaws.com/id/<cluster-id>/.well-known/openid-configuration).

You can currently change this in your Flipt configuration like so:

authentication:
  methods:
    kubernetes:
      enabled: true
      discovery_url: "https://oidc.eks.<region>.amazonaws.com/id/<cluster-id>"
  required: true

Could you give this a try for us 🙏

If this works we can make a docs update to explain this edge case a bit better 💯

[GeorgeMac]

This could be related to aws/containers-roadmap#2234

Haha great timing 🙌

[GeorgeMac]

That's amazing, thanks for raising this and working through it!

I will open a docs issue before closing this, so we make sure to get these details in there for future folks.

[tstraley]

Thanks @GeorgeMac -- this makes a lot of sense.

I tried out your suggestion. First attempt was causing pod to crash on startup, but was eventually able to get relevant error (there were some red-herring "context closed" errors on a couple pod restarts masking this one):

Error: configuring kubernetes authentication: fetching OIDC configuration: Get "https://oidc.eks.us-east-1.amazonaws.com/id/<our cluster id>/.well-known/openid-configuration": tls: failed to verify certificate: x509: certificate signed by unknown authority

This endpoint is one of AWS's public endpoints and uses a cert signed by CN=Amazon RSA 2048 M02. I could probably get this specific CA, put it in a k8s config map, and mount the volume; but to get by for now I added /etc/ssl/certs/ca-certificates.crt as the ca path, since this OS CA bundle is in the flipt container image.

authentication:
  methods:
    kubernetes:
      enabled: true
      discovery_url: "https://oidc.eks.<region>.amazonaws.com/id/<cluster-id>"
      ca_path: "/etc/ssl/certs/ca-certificates.crt"
  required: true

This started up fine, and now appears to be working properly!

Please feel free to resolve and update docs as you see fit. Thanks for the help!