Http wrapping issues about aiodynamo

[raulolteanu-sudo]

So for EKS deployments which use IAM roles and require Amazon STS, I created this custom Credential Handler to deal with my problem, as none of the other standard handlers worked as expected.

This is just for people looking for a quick solution or a general Ideea on how to adapt aiodynamo for more modern EKS deployments.( this is my quick solution, feel free to use it and improve it)

AWS_STS_API_VERSION = os.getenv("AWS_API_VERSION", '2011-06-15')
AWS_STS_CREDENTIALS_DURATION = int(os.getenv("AWS_CREDENTIALS_DURATION", 3600))
AWS_STS_CREDENTIALS_SESSION_NAME = os.getenv("AWS_CREDENTIALS_SESSION_NAME", 'blast_payment')

@dataclass
class STSCredentials(MetadataCredentials):
    """
    Loads credentials from STS metadata endpoint.
    """
    timeout: Timeout = 2 # noticed that fetch_metadata needed a little bit more time
    max_attempts: int = 2
    base_url: URL = URL("https://sts.amazonaws.com/")

    role_arn: Optional[str] = field(
                        default_factory=lambda: os.getenv('AWS_ROLE_ARN', None))

    web_identity_token: Optional[str] = field(
                        default_factory=lambda: STSCredentials.get_token_from_file())

    @classmethod
    def get_token_from_file(cls):
        file = os.getenv('AWS_WEB_IDENTITY_TOKEN_FILE', None)
        if not file:
            return None

        try:
            with open(file, 'r') as f:
                token = f.read()
            return token
        except FileNotFoundError:
            return None

    def is_disabled(self) -> bool:
        return self.role_arn is None or self.web_identity_token is None

    async def fetch_metadata(self, http: HTTP) -> Metadata:
        credentials_url = self.base_url.with_query(dict(Version=AWS_STS_API_VERSION,
                                                        DurationSeconds=AWS_STS_CREDENTIALS_DURATION,
                                                        RoleArn=self.role_arn,
                                                        RoleSessionName=AWS_STS_CREDENTIALS_SESSION_NAME,
                                                        Action="AssumeRoleWithWebIdentity",
                                                        WebIdentityToken=self.web_identity_token))
        headers = {'Accept': 'application/json'}

        raw_response = await self.fetch_with_retry(
            http=http,
            max_attempts=self.max_attempts,
            url=credentials_url,
            timeout=self.timeout,
            headers=headers
        )

        response = json.loads(raw_response)

        credentials = response['AssumeRoleWithWebIdentityResponse']['AssumeRoleWithWebIdentityResult']['Credentials']

        return Metadata(
            key=Key(
                id=credentials["AccessKeyId"],
                secret=credentials["SecretAccessKey"],
                token=credentials["SessionToken"],
            ),
            expires=datetime.datetime.fromtimestamp(credentials["Expiration"]).replace(tzinfo=datetime.timezone.utc),
        )

This would go here instead of Credentials.auto() .
(or create a chain for local/cluster use ChainCredentials(candidates=[FileCredentials(), STSCredentials()])

    async with ClientSession() as session:
        client = Client(AIOHTTP(session), STSCredentials(), "us-east-1")

[raulolteanu-sudo]

@raulolteanu-sudo thank you for sharing your solution here. Would the changes in #113 make it easier or harder (or have no impact) for you to implement this?

Hey there @ojii, Changes look good and will have no impact on my side (besides just changing fetch_with_retry with the new fetch_with_retry_and_timeout just like it is done in InstanceMetadataCredentials class in your PR).

Thanks for the heads-up !

[ojii]

When I was adding credentials handler to assume role with web identity which we need for our EKS setup I came across few issues why I am for now implementing it as custom code in our own project instead of providing it for the library.

Interesting. Would be cool to see that code, which might also help understand what changes to aiodynamo would help make it easier to implement.

Http wrappers assume response is json, https://github.com/HENNGE/aiodynamo/blob/master/src/aiodynamo/http/base.py but as endpoint response is xml (and at least I can't find any mention of possibility to request it as json, but you know AWS docs 💩 ). This is blocker and thought that perhaps there needs to be some refactoring around http handling which takes too long for us.

It's completely valid to write a Credentials implementation that ignores the HTTP that gets passed in. All you need is to implement the three methods (get_key, invalidate and is_disabled). You can use your own http session/library in there. The reason HTTP is passed in is to re-use the client object so not every call to get_key creates a new client session etc.

1. in MetadataCredentials there is fetch_with_retry which couldn't use because it assumes GET but endpoint for assume role is POST. Was thinking should/could this retry option be in base http classes?

As discussed in #45, if anything I'd like to simplify the HTTP interface rather than making it more complex (and adding retries is definitely more complex).

2. Missing timeout in POST, would prefer to have timeout also for this credentials call. As mentioned in question: connector get/post parity #45

Same as above, the idea is to remove timeouts from HTTP.

3. http.post(..) requires body as bytes even when it can in aiohttp and httpx also be None. https://github.com/HENNGE/aiodynamo/blob/master/src/aiodynamo/http/base.py#L28-L30

That should probably change.

[jarikujansuu]

If it is ok to go through base wrapper and assume that there is either aiohttp or httpx implementation under it then it is quite simple to do. I am in our own codebase implementing it for aiohttp, but trivial to add also httpx there. Just don't want to add hacky solutions to library, for our own use those are ok as we know they are there.

Currently I have just getting required configuration from environment variables but it is also possible to pass these web identity token file locations etc. in config files too. I will send PR when get little bit cleaned up version done.

[uweje]

endpoint response is xml (and at least I can't find any mention of possibility to request it as json, but you know AWS docs 💩 )

It actually returns json if you add the header "Accept": "application/json" but as it is not documented it might break.

We based our implementation on MetadataCredentials but it turned out to specific to our needs to share it

[ojii]

@raulolteanu-sudo thank you for sharing your solution here. Would the changes in #113 make it easier or harder (or have no impact) for you to implement this?