Allow OCI credentials expiration/refresh about flipt

[thepabloaguilar]

Hey guys, just a heads up! It worked like a charm, thanks a lot @erka!

[thepabloaguilar]

If we do add an expiration, the Flipt container will still not be able to resolve a new auth correct?

Hey @markphelps, that's my point, when the expiration is reached we reload the configuration!

And what I thought was to keep the current behavior, like, in this case I'm having when the token expire what will be the Flipt behavior? I guess it'll break

But this expiration is just an idea, another will be letting the user set an auth_file:

storage:
  type: "oci"
  oci:
    authentication:
      file: "/example/file"

The behavior will be: Flipt starts and check if file exists and if the credentials are working, every time later Flipt will read the file again and again.
And I'd say this option seems more elegant and can be easy achieved by creating our own credentials provider for oras because it's just a auth.CredentialFunc

And the file could be something like this two lines:

username
password

WDYT?

[erka]

All of those options are possible but I think the native support will be much better. There could be an auth function like

awsCredentialFunc = func(ctx context.Context, hostport string) (auth.Credential, error) {
	var client *ecr.Client
	response, err := client.GetAuthorizationToken(ctx, &ecr.GetAuthorizationTokenInput{})
	if err != nil {
		return auth.EmptyCredential, err
	}
	token := response.AuthorizationData[0].AuthorizationToken
	output, err := base64.StdEncoding.DecodeString(*token)
	if err != nil {
		return auth.EmptyCredential, err
	}
	split := strings.SplitN(string(output), ":", 2)
	if len(split) != 2 {
		return auth.EmptyCredential, err
	}
	return auth.Credential{
		Username: split[0],
		Password: split[1],
	}, nil
}

and oras should handle the refresh in background (https://github.com/oras-project/oras-go/blob/9b6f32158776a699b23edb3db86d053623619b60/registry/remote/auth/client.go#L203-L226)

Configuration could have extra type with static as a default value for auth.StaticCredential, aws for awsCredentialFunc, etc...

wdyt?

[thepabloaguilar]

Thanks @GeorgeMac and @erka! I think what you both suggested is a good option

But @erka I just like to suggest one change in your approach, the awsCredentialFunc should also deal/track an expiration time but this time without exposing to the user and the reason is as you mentioned oras will deal with the authentication BUT it will call that function everytime we make an action against ECR! If you pay attention it always make a first request and only if the returned status code is different form 200 OK it'll deal with auth calling the function we've passed

So to avoid everytime calling an external service (ECR) to get de credentials will be good

Btw, I can implement it!

[erka]

@thepabloaguilar Yep, we could adjust it like here

[erka]

Hey @thepabloaguilar.

Could you please tell us more about your AWS configuration. ECS, EKS, EC2, Fargate? Ideally this should be done by attaching role with permissions to pull from ECR on AWS.

[thepabloaguilar]

Sure @erka, that's true! When you attach a role to a Container/POD we have the permissions but the permissions is not granted automatically when not using the AWS CLI o SDK like this case because Flipt uses "oras" which is not related to AWS stuffs so it doesn't auto resolve the credentials!

Essentially I'm using IRSA with EKS which exposes the environment variables inside the container: "AWS_ROLE_ARN" and "AWS_WEB_IDENTITY_TOKEN_FILE"! When executing any AWS CLI command like aws ecr get-login-password the CLI itself resolves the auth stuff (assuming the role and generating the required tokens) using those variables

And that's the problem, as Flipt uses Oras for that the auth is not resolved by it (which is the expected behavior) so we're going need to update the password environment variable every 12 hours

[markphelps]

Thanks for the explanation @thepabloaguilar ! If we do add an expiration, the Flipt container will still not be able to resolve a new auth correct? How do you invision that working?

[thepabloaguilar]

Or we can even combine both together:

storage:
  type: "oci"
  oci:
    authentication:
      file: "/example/file"
      file_refresh_rate: "1h"

[thepabloaguilar]

If y'all think/understand it's not a great addition to Flipt I completely understand and I'll try to maybe switch from OCI to S3 because it'll use the AWS SDK which auto resolve the credentials stuff! But AFAIK the other cloud provider also put some expiration time in the tokens

[GeorgeMac]

Great improvement suggestions. I think what @erka suggests here is a good shout and ultimately more portable, without extra steps in your environment. i.e. add a type for OCI authentication and support a native aws (would ecr or aws/ecr be more appropriate type name?) that uses the AWS client libs to do it automatically. We could add more down the line, it seems there is a nice one for Docker config format already in the oras library that might be nice for locally building bundles.

[erka]

@thepabloaguilar From code oras should cache the credential while they are valid if oras client has a cache in place. So it should not be many calls to get credential. This is all my theory and it should be tested and verified with CloudTrail

[thepabloaguilar]

@erka that's true if we're setting the Cache property which we're not doing rn but we could: https://github.com/flipt-io/flipt/blob/main/internal/oci/file.go#L146

Putting the cache in there should be enough!

This is all my theory and it should be tested and verified with CloudTrail

I can test it locally to see what the behavior will be