Comments (21)
@nickatsegment Just my two cents but I believe we can reduce n
GetSecretValue calls to basically 1
SSM GetParameters call. SSM PS' GetParameters
does pass-through to secrets by prefixing param names with /aws/reference/secretsmanager/
.
from chamber.
You still need to pay for KMS keys to use Secrets Manager; each secret is associated with a KMS key, just like with ParameterStore SecureStrings.
from chamber.
I think one of the best feature of Secret Manager is dynamic generated secret which means you don't actually have to store your secret anywhere except in the secret store itself. That is huge for CI/CD since it solves the issue of how to create and store secret as part of CI/CD by eliminating the need of storing it altogether.
from chamber.
Has there been any more interest in adopting secrets manager as a backend store?
from chamber.
As far as I can tell, secrets-manager is just a much more expensive ($0.4 per secret/month + some per-query charges) version of parameter-store (which is free).
The only new feature is automatic rotation, which works automatically for RDS DB credentials, but it's literally just a cronjob for all other kinds of secrets (ie. you need to write your own lambda, just like you do with parameter store).
There's no reason not to support it as an alternate store backend, but replacing parameter-store seems like a bad idea.
Although I'm sure it's built on-top of parameter-store internally, I just checked and you can't see secrets-manager secrets in parameter-store, so chamber would need to merge the secrets from the two APIs manually at read-time, or force the user to pick a service and only read from that.
from chamber.
When I read the blog post I thought the same, and don't understand why it's so much more expensive than store parameter
from chamber.
While Parameter Store is a free service, they still charge you for KMS keys and other underlying services like CloudWatch. In fact, Secrets Manager might be cheaper than parameter store, depending on how you manage your parameters and keys.
from chamber.
This article argues that rate limiting is an issue with SSM: https://www.stackery.io/blog/serverless-secrets/
from chamber.
@jeznag That's 100% true and we hit it often (see #79). It's the main motivation for the experimental s3 backend
from chamber.
AWS Secrets Manager have 700 request/sec for ReadSecret
https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_limits.html#reference_limits_rates
from chamber.
One downside of Secrets Manager: It doesn't look like there's any way to get a batch of secrets, like you can with Parameter Store paths. You could put all the secrets for a service in one Secret, but you're limited to 4 KiB (same as Parameter Store).
from chamber.
Secrets Manager allows you to use /
in secret names, so it is possible to create a "path" per service.
Example: You can prefix all secrets for myService
with myService/
. Then you can use the ListSecrets
operation to get complete list of all secrets, after that you call GetSecretValue
on the ones that has the correct prefix.
In addition you would create a custom IAM read-only policy for that specific prefix resource: arn:aws:secretsmanager:<region>:<accountId>:secret:myService/*
from chamber.
Right, but you still need to make 1 ListSecrets
call + n GetSecretValue
calls vs SSM's 1 (paginated) GetParametersByPath
call.
from chamber.
Yes, and I think that will suffice in most cases. The n GetSecretValue
calls has a rate limit of 700/s, while GetParametersByPath
has an unknown (?) rate limit, and only 10 parameters max per page.
from chamber.
@mumoshu That's pretty cool. Hard to say which would be more efficient in the general case, but it'd be nice if it were user configurable.
from chamber.
@danielgsims I don't think there's a use case at Segment, so I doubt we'd write one in the foreseeable future. I'd happily accept a nice PR.
from chamber.
I was interested in using Secrets Manager in conjunction with chamber, so I went ahead and wrote an implementation of a new backend yesterday. It's working pretty well now, although I did run into some fundamental issues:
- Secrets Manager doesn't seem to expose last changed user anywhere (can't support user visibility in history or list)
- Versions seem to be automatically generated and alphanumeric, and it doesn't look like you can vary tags by version (can't support chronological version numbers, and version fields had to be refactored from integers to strings)
Any suggestions for the above issues? I'm looking into using the SSM API for fetching secrets, but it doesn't look like it will be able to address the problems above. I'm also worried about adding dependency on the SSM API, given that one major motivation for this backend is to avoid rate throttling.
from chamber.
This issue has been automatically marked stale
because it has not had any activity in the last 60 days. If no further activity occurs within 7 days, it will be closed. Closed does not mean "never", just that it has no momentum to get accomplished any time soon.
See CONTRIBUTING.md for more info.
from chamber.
Closing due to staleness. Closed does not mean "never", just that it has no momentum to get accomplished any time soon.
See CONTRIBUTING.md for more info.
from chamber.
@njflowers i was looking at implementing a secretsmanager store myself and ran into the same problems that you did. Where did you end up? Is your version of it on github? I am thinking about creating a version with only support for secretsmanager and not SSM given that the Store interface doesn't carry over well
from chamber.
@viswaug Yes, this code is available on GitHub. Please feel free to view / use it:
https://github.com/njflowers/chamber/tree/add-secrets-manager-backend
I wasn't able to get around the limitations I mentioned regarding user visibility and versions, although I did get version display working. Since I never got a verdict regarding SSM vs SM APIs, I decided to implement them both - you can configure the behavior used by the Secrets Manager backend with the CHAMBER_SM_USE_SSM
environment variable (setting any value implies true). I believe most behavior is working in that branch, although it has not been rebased for some time.
from chamber.
Related Issues (20)
- `write` allows for invalid shell variable creation HOT 2
- Security Scans flagging go-complier v1.13.15 HOT 3
- Bug Importing UPPER_CASE_KEYs HOT 3
- consider adding semantic version info in "version" subcommand HOT 4
- Issues with chamber and aws sso
- CVE-2023-24538 - Go Lang 1.19.6 Critical Vulnerability HOT 1
- Please add renovatebot or dependabot to keep dependencies updated HOT 1
- Please consider dropping support for older golang versions like go1.15 and go1.16
- Please consider using GetParameter(s) instead of GetParametersByPath
- Logger writes to stdout, messing up output HOT 2
- Chamber is not working with recommended AWS SSO config that uses sso-session HOT 15
- CVE-2023-29404 - Go Lang Critical Vulnerability HOT 1
- Feature request: No clobber of existing environment variables HOT 3
- Artifact upload is failing HOT 1
- CVE-2023-39323 - Go Lang Critical Vulnerability
- Allow using `CHAMBER_BACKEND` env variable to set the backend HOT 2
- Feature request: dry run mode to show full path and result of create/update
- Serialize YAML and JSON into string -> cannot unmarshal !!seq !!map into string HOT 9
- Origin of the name chamber HOT 2
- Add dockerfile golang version to dependabot config
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from chamber.