Allow AWS access key to be specified inline within Jenkins plugin config about aws-secrets-manager-credentials-provider-plugin

[DimaNevelev]

Thank you,
I don't work there anymore and in the new place we don't use Jenkins.
I'll try to find someone relevant for testing this.

[chriskilding]

Just to check I understand the surrounding context right...

• Your company has 1 big AWS account for all teams
• All teams' secrets are in that account's Secrets Manager
• Each team has a K8S-on-EC2 cluster (not EKS?) with Jenkins on it

[chriskilding]

Also, are the teams able to edit resources in the AWS account besides the locked-down K8S clusters (i.e. can they put their own secrets in Secrets Manager), or does some central team or enterprise automation tooling edit the AWS resources for them?

[DimaNevelev]

• Your company has 1 big AWS account for all teams:
  • We have multiple AWS accounts, two of them are dedicated to Secrets manager (Dev and Prod)
• All teams' secrets are in that account's Secrets Manager:
  • Yes, all the secrets are stored in two main aws accounts. There is a segregation between the teams via roles based on the secret's name. For example, team AAA has access only to secrets with the name "/aaa/*".
• Each team has a K8S-on-EC2 cluster (not EKS?) with Jenkins on it
  • Each team has a namespace with Jenknis instance in a dedicated for Jenkins K8s. This K8s runs on ec2 nodes/instances (not EKS) in a separate account from the accounts used for secrets manager.
• are the teams able to edit resources in the AWS account
  • The teams can edit their secrets and change their API keys in the secrets manager account. They can't change the resources of the Jenkins K8s account.

[chriskilding]

Ok, the first thing to find out is whether you can use AWS profiles to let the Jenkins box hop into the alternate account for Secrets Manager access; if this can't be done then we'll have to deal with this first.

This plugin doesn't currently support multiple accounts for lack of a good credential namespacing solution. However I believe AWS lets you use a different account to the implicit one, by specifying an assumeRole ARN if you set up an AWS profile in ~/.aws/config and ~/.aws/credentials. Give that a go, and if that works we can look at improving the way config works.

[DimaNevelev]

We can't / don't want the Jenkins infrastructure to be involved. Since ~/.aws/credentials are defined in the infrastructure, we can't do this...
A great solution for us would be an integration with the aws-credentials plugin. The user will set up aws-credentials and choose the relevant credentials in the plugin configuration. The plugin will use these credentials to pull the secrets from the secrets manager service...
The downside of this solution is the dependency on the aws-credentials plugin.

[chriskilding]

I'm planning to restore a part of the beta clients feature which should do what you need. This would expose a client: option in the config that would allow you to change the following fields about the AWS SDK client, including the AWS credentialsProvider which tells the SDK which strategy to authenticate with. The static strategy would allow an access key pair to be specified inline within Jenkins config, rather than inherited from outside:

<plugin>:
  cache: ...
  client:
    credentialsProvider:
      static:  # options: blank (default), static (specify key pair inline - shown here), profile (read from AWS profiles), assumeRole (IAM role ARNs)
        accessKeyId: "foo"
        secretAccessKey: "bar"
    endpointConfiguration: # or leave blank for default
      serviceEndpoint: ...
      signingRegion: ...
    region: "some-region" # or leave blank for default
  listSecrets:
    filters:
      ...

Let me know if this does what you need (I think it should?)

[DimaNevelev]

This is pretty close to what we need...
Two things I would like to change:

1. Providing the secretAccessKey as plain text in Jenkins configuration is a high-security risk.
   I think a preferred (and the standard) way is to store the accessKeyId and secretAccessKey as credentials in Jenkins.
2. We need a combination of static and role. Since the credentials allow only to assume a role and not direct access to the secrets manager

    credentialsProvider:
      static:  # options: blank (default), static (specify credentials ID - shown here), profile (read from AWS profiles), assumeRole (IAM role ARNs)
        credentialsId: "AWSCredentials" // the id of the credentials in Jenkins 
        role: ... // the role to assume

If you don't have the time for these two requests, let me know... restore the clients, and I will contribute it.

[chriskilding]

The secretAccessKey can be modelled as a Secret rather than a String in the Jenkins model. This ensures that when it's serialised to Jenkins config XML the value will not be stored in plaintext.

If the value is inserted through the Web UI, this is sufficient; the plaintext value won't be stored decrypted when it hits the disk.

If the value is inserted through CasC by pasting the value straight into the casc.yaml, then yes the decrypted secretAccessKey would be present on disk. However, CasC has a mechanism for hiding early-stage bootstrapping secrets like this: secrets interpolation. This is what I'd recommend in that use case.

The other approach you describe - feeding a credentials provider with a credential - has 2 problems:

• We still end up solving the problem with CasC secrets interpolation, but we incur a layer of indirection. If a provider is fed with a credential, we'd have to supply that credential somehow... through another provider. We'd likely put this credential in the default on-disk provider. When we configure that through Jenkins CasC, you would set up that on-disk credential elsewhere in the casc.yml file. But we wouldn't want the secret to be in plain text there either. And we'd have to solve that with... CasC secrets interpolation.
• If, later on, this plugin gains support for serving the same type of credential as it uses for bootstrapping, this could lead to an awkward chicken-and-egg situation where it tries to get the bootstrapping credential from itself, before it has initialised. That'll probably lead to a crash.

[chriskilding]

These difficulties are to be expected though: solving the credential bootstrapping problem in Jenkins is hard! Even CasC secrets interpolation just kicks the problem one step further down the road; the secrets must still be injected from somewhere.

The only proper solution to this if you're running Jenkins inside AWS is... you guessed it, IAM roles with EC2 Instance Profiles (or the EKS equivalent), which allows the connection to Secrets Manager without hardcoding an access key on the box.

[DimaNevelev]

I see your point regarding storing the secretAccessKey as credentials in Jenkins.
What do you think about combining the static credentials with the role?

[chriskilding]

I was about to restore the old code when I realised there would be another problem. This plugin is both a CredentialsProvider and a SecretSource. The SecretSource implementation runs early in Jenkins initialisation; when CasC calls upon it to interpolate secret values in casc.yml. This means that if the plugin's access key pair is supplied in casc.yml, it will not be available when the SecretSource implementation needs it. Indeed this is why I had to invent the AWS_SERVICE_ENDPOINT and AWS_SIGNING_REGION environment variables - so that the endpoint configuration could be set during the integration tests.

So I'm not sure we can do what you're after :/

[chriskilding]

@DimaNevelev If you're still watching this, a few things have changed...

First, the SecretSource implementation was extracted into its own plugin.

Second, a new Client configuration feature has been added to the Credentials Provider, which allows you to configure the AWS SDK authentication strategy.

The strategies currently available are

• Default (the AWS SDK provider chain)
• Profile (specify an AWS profile name)
• Instance Role (specify an IAM instance role ARN)

I'm hoping the Profile option might do what you want, but if not, this mechanism would allow the addition of an access key pair strategy (which would let you set a key pair inline in Jenkins configuration). That would be a simple case of another pull request to add the strategy.

Let me know what you think.

[chriskilding]

@DimaNevelev I have now created the feature you asked for in #236

Would you be able to try out the experimental feature (plugin .hpi file is linked below) and let me know whether it works for you?

https://repo.jenkins-ci.org/incrementals/io/jenkins/plugins/aws-secrets-manager-credentials-provider/1.200.vb_97320e2c8d3/aws-secrets-manager-credentials-provider-1.200.vb_97320e2c8d3.hpi

[chriskilding]

Hi @DimaNevelev , did you manage to find someone who can test this?

[DimaNevelev]

Sorry, everyone I know left.