RFC4716 SSH Key stored in AWS does not work on Windows agent for Git clone about aws-secrets-manager-credentials-provider-plugin

[flichtenheld]

Sigh, after fidling around with the keys in secret manager I can't reproduce the issue currently. And during one of the tests I overrode the original value that caused the problems so I can't go back to try different things.

What I found out is that secretsmanager seems also doing some internal newline conversions when using --secret-string. When uploading a file with \r\n line ending with --secret-string the secret actually comes back with \n line endings. I know that at some point yesterday I managed to create a secret with \r\n line endings (which was not the problem, BTW), however now I'm not even able to reproduce that.

All a very long way of saying that at this point I will take the fact that it seems to work now and will only investigate further if I manage to break it again.

Closing for now.

Thanks for the help and sorry for the noise.

[chriskilding]

Is your Jenkins setup a distributed one (controller on one machine, agents that run the jobs on other machines), or all on one box?

[flichtenheld]

Controller is separate machine. All jobs run on distinct agents (both the Linux agents where the key works and the Windows agents where it doesn't)

[chriskilding]

Hmm. I have previously heard of a bug in distributed setups where, when a plugin consumes a credential (e.g. the git plugin, or slack plugin, or whatever), and the plugin does not snapshot the credential prior to transfer, the secret value ends up null when the job tries to use it. This doesn't happen with the Jenkins pipeline (withCredentials or credentials macros), which do take care to snapshot it beforehand.

If you can use Jenkins Pipeline in your setup, could you try adding

node {
    withCredentials([sshUserPrivateKey(credentialsId: '<your SSH key id>', keyFileVariable: 'KEY', usernameVariable: 'KEY_USR')]) {
        // try to interact with $KEY or $KEY_USR some way, but you can't just print the value because this is masked in the build log
    }
}

before the git clone (or maybe just create a new job for this), and see what the result is?

I'm thinking that if something substantial is wrong, the credential won't bind or its value will effectively be null/empty string

[flichtenheld]

Those tests work reasonably fine. File is written correctly to the disk. But in other tests I just found out that this probably has nothing to do with Windows. I tried to use the key in sshagent (on Linux nodes) and that failed as well:

[ssh-agent] Using credentials git (BitbucketSSH key)
[ssh-agent] Using credentials root (SCT SSH Key)
[ssh-agent] Looking for ssh-agent implementation...
[ssh-agent]   Exec ssh-agent (binary ssh-agent on a remote machine)
$ ssh-agent
SSH_AUTH_SOCK=/tmp/ssh-zmXXY9EUZ1wA/agent.4558
SSH_AGENT_PID=4560
Running ssh-add (command line suppressed)
Error loading key "/home/ubuntu/workspace/d_flichtenheld_core-jenkins-test@tmp/private_key_8107316556344895340.key": invalid format

Will do more tests with different keys and credential providers. Definitely seems to be more complicated than "doesn't work on Windows".

[flichtenheld]

So, some results:

The testcases are sshagent() on Linux and checkout() on Windows.
withCredentials() works consistently. It always produces the original credential, with both providers.

The tests work fine with a PEM style key, i.e. one that starts with -----BEGIN RSA PRIVATE KEY----- or -----BEGIN EC PRIVATE KEY-----.

But they do not work with a RFC4716 style key, i.e. one that starts with -----BEGIN OPENSSH PRIVATE KEY-----.

But only when using the AWS credentials provider, with the default provider everything works fine regardless of key format. Maybe the default provider does some auto-magic key conversions?

So I have a suitable work-around now, just converting the keys to PEM format. But still would be interested to know where the actual problem comes from.

[chriskilding]

The AWS SSH private key class looks like this:

public class AwsSshUserPrivateKey extends BaseStandardCredentials implements SSHUserPrivateKey {
    private static final Secret NO_PASSPHRASE = Secret.fromString("");

    private final Supplier<String> privateKey;
    private final String username;

    public AwsSshUserPrivateKey(String id, String description, Supplier<String> privateKey, String username) {
        super(id, description);
        this.privateKey = privateKey;
        this.username = username;
    }

    @NonNull
    @Deprecated
    @Override
    public String getPrivateKey() {
        return privateKey.get();
    }

    @Override
    public Secret getPassphrase() {
        return NO_PASSPHRASE;
    }

    @NonNull
    @Override
    public List<String> getPrivateKeys() {
        return Collections.singletonList(getPrivateKey());
    }

    @NonNull
    @Override
    public String getUsername() {
        return username;
    }

    // ...
}

And the credentials factory creates it by:

String name;
String description;
String username;
<aws secrets manager client object> client;

new AwsSshUserPrivateKey(name, description, new StringSupplier(client, name), username)

And all the StringSupplier does is to get the secret-string from Secrets Manager.

So it's all pretty literal; whatever comes out of Secrets Manager, gets reflected in the SSH key object.

I'm wondering, however, whether encoding of the secret might make a difference. Perhaps one of the keys you tried had e.g. LF line breaks, while another had e.g. CRLF? And maybe this affects Windows?