Issue with getting Region when federated about aws-encryption-sdk-java

[bdonlan]

Okay, that explains things then. We'll get the fix merged and spin a new version. Thanks for reporting this!

[michaelajr]

And oddly enough adding this...

provider.setRegion(Region.getRegion(Regions.US_EAST_1));

...fixes the issue. Just do not understand it. I have the ~/.aws/config file set with the correct region. The region needs to be determined at runtime.

[mattsb42-aws]

What version are you using? This should have been addressed by the regional client supplier added in 1.3.1.

It's hard to tell without seeing your exact config and how you are using federated sessions, but I suspect that when you load the credentials for the federated session, the credential loader is not looking in your config for a default region.

[michaelajr]

Thanks for the help. Here's my set up...

$ env | grep AWS
AWS_DEFAULT_REGION=us-east-1
AWS_REGION=us-east-1

~/.aws/credentials

[default]
aws_access_key_id=redacted
aws_secret_access_key=redacted
aws_session_token=redacted
aws_security_token= redacted

~/.aws/config

[default]
region=us-east-1

Using 1.3.2 of the SDK now (was at 1.3.1). And now using the KmsMasterKeyProvider builder.

        final KmsMasterKeyProvider provider
                = KmsMasterKeyProvider.builder()
                        .withCredentials(
                                new DefaultAWSCredentialsProviderChain())
                        .build();

Yields:

com.amazonaws.encryptionsdk.exception.AwsCryptoException: Can't use keys from region null

Adding `.withDefaultRegion("us-east-1")

        final KmsMasterKeyProvider provider
                = KmsMasterKeyProvider.builder()
                        .withDefaultRegion("us-east-1")
                        .withCredentials(
                                new DefaultAWSCredentialsProviderChain())
                        .build()

Yields

com.amazonaws.encryptionsdk.exception.AwsCryptoException: Can't use keys from region null

The region is definitely not getting set. Or it is getting overwritten. Seems to find the credentials just fine.

[michaelajr]

And going back the old way (now deprecated):

        final KmsMasterKeyProvider provider
                = new KmsMasterKeyProvider(
                        new DefaultAWSCredentialsProviderChain());

Gives me:

com.amazonaws.encryptionsdk.exception.CannotUnwrapDataKeyException: Unable to decrypt any data keys
	at com.eoniantech.secretslocker.aws.ClassPathSecretsLockerIT_get.testGet(ClassPathSecretsLockerIT_get.java:44)
Caused by: com.amazonaws.encryptionsdk.exception.AwsCryptoException: Can't use keys from region us-east-1

Crazy.

[michaelajr]

So to get this working I have use 1.3.1 - and explicitly set the region. Which is of course not what I want. The region needs to be set at runtime. What on earth can be wrong here?

[mattsb42-aws]

Were you using 1.3.1 when you had to set the region? As of 1.3.1 the KMS master key provider should automatically vend a KMS master key for the right region on decrypt based on the ARN in the header.

EDIT: Sorry, re-read and saw you're using 1.3.2 now.

[michaelajr]

When using 1.3.1 the region was undetermined, so the provider could not vend a key. By explicitly adding the region on the provider - it worked. Key is still undetermined when using 1.3.2, and there is not way to set the region.

[michaelajr]

This has to be an issue with the credentials/profile loader in the SDK. Or I'm just doing something wrong in my credentials/config setup. I do not think this has anything to do with the Encryption SDK.

[bdonlan]

Normally, if you use the builder, the KMSMasterKeyProvider should find the appropriate region by examining the key ID in the ciphertext headers during the decrypt flow. It's strange that this isn't working. How are you generating your ciphertexts?

If it's possible to attach a ciphertext generated using a test key, we could take a look at the headers and see if the region is missing somehow.

[michaelajr]

Sure. I can upload an encrypted file in bit. Thanks for the help. When the region is explicitly set, the provider vends the correct key. So when the region is not set, it seems the "region chain" logic (or whatever determines the region to use) is not working. Like I said, this works when using a IAM user and not a federated user.

[michaelajr]

Here's example ciphertext created by a test key.
example.zip

[michaelajr]

Btw, I appear to have similar issues with the S3 SDK. It does not find the default region when I'm federated. So this must be a local config issue, or a broader SDK issue.

[bdonlan]

The strange thing is your message "Can't use keys from region null" - this shouldn't depend on your configuration at all, but instead the region name should come from a parsed arn (see KmsMasterKeyProvider.java:452).

I just stepped through a decrypt on my end and it did identify the correct region (and then failed because I don't have access to your key). So I'm a bit stumped.

Would it be possible to have you set a breakpoint on KmsMasterKeyProvider.getMasterKey and see what its provider and keyId arguments are?

[michaelajr]

Ok. So I think I found the issue. I was able to fix this in 1.3.1. Federation was a red herring.

If I do not explicitly set a region, the KmsMasterKeyProvider is created using Regions.DEFAULT_REGION which (according to AWS SDK core) is us-west-2.

I only used a single key - in us-east-1 - when I created the encrypted file.

So I added a key in us-west-2 - re-encrypted using both keys, and I was able to decrypt as expected without explicitly setting the region! It worked EXACTLY like before.

And... before... I did encrypt with two keys - one from us-east-1 and one from us-west-2. It was only TODAY that I encrypted with a single key in us-east-1.

So it appears in 1.3.1 you need use a key in us-west-2. I do wonder if all my KMS calls are going to us-west-2 as well. Maybe I'll turn on debug logging and see.

In hopes this was also the issue with 1.3.2, I upgraded and changed the code to the builder(). But I still get

com.amazonaws.encryptionsdk.exception.AwsCryptoException: Can't use keys from region null

Very odd. Hope this helps. Will keep digging. And will try to do the breakpoint you asked for.

[michaelajr]

Ok. My bad. When using 1.3.2 the Can't use keys from region null error is when encrypting. Same code works fine 1.3.1.

[michaelajr]

Will this not work in 1.3.2, where this.keyId is an alias in all the specified regions? This works fine in 1.3.1 - the Key ARNs are in the cipher text. But getting the null error in 1.3.2.

    private MasterKeyProvider<?> masterKeyProvider() {

        AWSCredentialsProvider credentials
                = new DefaultAWSCredentialsProviderChain();

        List<KmsMasterKey> masterKeys
                = new LinkedList<>();

        for (String region : this.regions) {
            KmsMasterKeyProvider provider
                    = new KmsMasterKeyProvider(
                            credentials,
                            Region.getRegion( 
                                    Regions.fromName(region)),
                            new ClientConfiguration(),
                            this.keyId);

            masterKeys.add(
                    provider.getMasterKey(
                            this.keyId));
        }

        return MultipleProviderFactory
                .buildMultiProvider(masterKeys);
    }

[michaelajr]

Moved above that worked in 1.3.1 to builder for 1.3.2. Getting NPE when calling provider.getMasterKey. Is this the right approach?

    private MasterKeyProvider<?> masterKeyProvider() {

        final AWSCredentialsProvider credentials
                = new DefaultAWSCredentialsProviderChain();

        List<KmsMasterKey> masterKeys
                = new LinkedList<>();

        for (String region : this.regions) {
            KmsMasterKeyProvider provider
                    = KmsMasterKeyProvider
                            .builder() 
                            .withDefaultRegion(region)
                            .withCredentials(credentials)
                            .withKeysForEncryption(this.keyId) 
                            .build();

            masterKeys.add(
                    provider.getMasterKey(
                            this.keyId));
        }

        return MultipleProviderFactory
                .buildMultiProvider(
                        masterKeys);
    }

[michaelajr]

I apologize for all the spin. To recap. I assumed if a region was not passed to the master key provider on decryption - the current region would be used to vend the resign-specific key. This is not the case. It simply used us-west-2 if the region was not specified. And that is why when I only had a us-east-1 key, I would get an error saying it could not use that key. Adding a us-west-2 key fixed this issue - but exposed a logic error on my part. I believe I need to build the provider with the current region to get the behavior I want (vending the region-specific key). This was all masked in my previous tests since I used two keys - one being us-west-2. It was only when I move to a single key in us-east-1 where I saw the behavior. I wrongly assumed the issue was with my credentials and local region config.

Moving to 1.3.2 does expose another issue for me. NullPointer when using the builder in the previous comment, where I'm trying to get a masker key using an alias and region. Maybe you could offer some guidance there.

Thank you for your patience. Again, I apologies for the swirl.

[bdonlan]

I'm not sure this is actually resolved. In 1.3.2, normally, you should be able to decrypt using keys from any region, using a single MKP. If it's reporting that it can't use region "null" this suggests to me that the wrong key ID is making it into the ciphertext, but that doesn't appear to be the case in the ciphertext you sent. So I'm still very confused about how this is happening.

Would it be possible to get the value of keyId in getMasterKey as I requested above?
Also, regarding the NPE, please provide a stack trace.

[bdonlan]

Since I haven't seen any full stack traces: Is this on encrypt or decrypt?

[michaelajr]

The null was on encrypt. I missed that in my hurried state yesterday. I then opened #50. I will say this explains why my KMS calls were all going to us-west-2. :)

[michaelajr]

Sure. Sorry again for all the spin.