Comments (6)
So for EKS deployments which use IAM roles and require Amazon STS, I created this custom Credential Handler to deal with my problem, as none of the other standard handlers worked as expected.
This is just for people looking for a quick solution or a general Ideea on how to adapt aiodynamo for more modern EKS deployments.( this is my quick solution, feel free to use it and improve it)
AWS_STS_API_VERSION = os.getenv("AWS_API_VERSION", '2011-06-15')
AWS_STS_CREDENTIALS_DURATION = int(os.getenv("AWS_CREDENTIALS_DURATION", 3600))
AWS_STS_CREDENTIALS_SESSION_NAME = os.getenv("AWS_CREDENTIALS_SESSION_NAME", 'blast_payment')
@dataclass
class STSCredentials(MetadataCredentials):
"""
Loads credentials from STS metadata endpoint.
"""
timeout: Timeout = 2 # noticed that fetch_metadata needed a little bit more time
max_attempts: int = 2
base_url: URL = URL("https://sts.amazonaws.com/")
role_arn: Optional[str] = field(
default_factory=lambda: os.getenv('AWS_ROLE_ARN', None))
web_identity_token: Optional[str] = field(
default_factory=lambda: STSCredentials.get_token_from_file())
@classmethod
def get_token_from_file(cls):
file = os.getenv('AWS_WEB_IDENTITY_TOKEN_FILE', None)
if not file:
return None
try:
with open(file, 'r') as f:
token = f.read()
return token
except FileNotFoundError:
return None
def is_disabled(self) -> bool:
return self.role_arn is None or self.web_identity_token is None
async def fetch_metadata(self, http: HTTP) -> Metadata:
credentials_url = self.base_url.with_query(dict(Version=AWS_STS_API_VERSION,
DurationSeconds=AWS_STS_CREDENTIALS_DURATION,
RoleArn=self.role_arn,
RoleSessionName=AWS_STS_CREDENTIALS_SESSION_NAME,
Action="AssumeRoleWithWebIdentity",
WebIdentityToken=self.web_identity_token))
headers = {'Accept': 'application/json'}
raw_response = await self.fetch_with_retry(
http=http,
max_attempts=self.max_attempts,
url=credentials_url,
timeout=self.timeout,
headers=headers
)
response = json.loads(raw_response)
credentials = response['AssumeRoleWithWebIdentityResponse']['AssumeRoleWithWebIdentityResult']['Credentials']
return Metadata(
key=Key(
id=credentials["AccessKeyId"],
secret=credentials["SecretAccessKey"],
token=credentials["SessionToken"],
),
expires=datetime.datetime.fromtimestamp(credentials["Expiration"]).replace(tzinfo=datetime.timezone.utc),
)
This would go here instead of Credentials.auto() .
(or create a chain for local/cluster use ChainCredentials(candidates=[FileCredentials(), STSCredentials()])
async with ClientSession() as session:
client = Client(AIOHTTP(session), STSCredentials(), "us-east-1")
from aiodynamo.
@raulolteanu-sudo thank you for sharing your solution here. Would the changes in #113 make it easier or harder (or have no impact) for you to implement this?
Hey there @ojii, Changes look good and will have no impact on my side (besides just changing fetch_with_retry
with the new fetch_with_retry_and_timeout
just like it is done in InstanceMetadataCredentials
class in your PR).
Thanks for the heads-up !
from aiodynamo.
When I was adding credentials handler to assume role with web identity which we need for our EKS setup I came across few issues why I am for now implementing it as custom code in our own project instead of providing it for the library.
Interesting. Would be cool to see that code, which might also help understand what changes to aiodynamo would help make it easier to implement.
Http wrappers assume response is json, https://github.com/HENNGE/aiodynamo/blob/master/src/aiodynamo/http/base.py but as endpoint response is xml (and at least I can't find any mention of possibility to request it as json, but you know AWS docs ๐ฉ ). This is blocker and thought that perhaps there needs to be some refactoring around http handling which takes too long for us.
It's completely valid to write a Credentials
implementation that ignores the HTTP
that gets passed in. All you need is to implement the three methods (get_key
, invalidate
and is_disabled
). You can use your own http session/library in there. The reason HTTP
is passed in is to re-use the client object so not every call to get_key
creates a new client session etc.
1. in
MetadataCredentials
there is fetch_with_retry which couldn't use because it assumesGET
but endpoint for assume role isPOST
. Was thinking should/could this retry option be in base http classes?
As discussed in #45, if anything I'd like to simplify the HTTP
interface rather than making it more complex (and adding retries is definitely more complex).
2. Missing timeout in
POST
, would prefer to have timeout also for this credentials call. As mentioned in question: connector get/post parityย #45
Same as above, the idea is to remove timeouts from HTTP
.
3. http.post(..) requires body as bytes even when it can in aiohttp and httpx also be None. https://github.com/HENNGE/aiodynamo/blob/master/src/aiodynamo/http/base.py#L28-L30
That should probably change.
from aiodynamo.
If it is ok to go through base wrapper and assume that there is either aiohttp or httpx implementation under it then it is quite simple to do. I am in our own codebase implementing it for aiohttp, but trivial to add also httpx there. Just don't want to add hacky solutions to library, for our own use those are ok as we know they are there.
Currently I have just getting required configuration from environment variables but it is also possible to pass these web identity token file locations etc. in config files too. I will send PR when get little bit cleaned up version done.
from aiodynamo.
endpoint response is xml (and at least I can't find any mention of possibility to request it as json, but you know AWS docs ๐ฉ )
It actually returns json if you add the header "Accept": "application/json"
but as it is not documented it might break.
We based our implementation on MetadataCredentials
but it turned out to specific to our needs to share it
from aiodynamo.
@raulolteanu-sudo thank you for sharing your solution here. Would the changes in #113 make it easier or harder (or have no impact) for you to implement this?
from aiodynamo.
Related Issues (20)
- KeyConditionExpression does not allow IN
- Support for applications running on Kubernetes
- after moving to github actions, integration tests are no longer run
- Add Amazon DynamoDB Accelerator (DAX) support to the library HOT 3
- Surprising behaviour in `Client.table_exists` HOT 1
- Auth Error With No Default Credentials HOT 1
- Unable to use AWS_REGION env var HOT 3
- Support AWS Web Identity Token File authentication HOT 2
- Shall we try goodfirstissue.dev ?
- Empty-set safety HOT 3
- Session instantiation HOT 6
- Get most recently added entry HOT 5
- ECS Fargate Credentials Compatibility HOT 2
- Support for typing-extensions version 4 module HOT 2
- Update maximum transaction operations to 100 HOT 1
- InstanceMetadataCredentials does not work with IMDSv2 HOT 5
- Awkward API for handling `TransactionCanceled` from `transact_write_items` HOT 2
- await table.exists() is stuck/hanging HOT 3
- Add response information to caught exceptions HOT 4
- Documentation in other languages, maybe?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aiodynamo.