aws-beam / aws-codegen Goto Github PK

It currently exists for generating Elixir code. It would be nice to have the same for Erlang and include support for a number of AWS APIs currently missing.

Description

When providing multiple querystring parameters (e.g. to aws_s3:list_objects/8) the response from AWS is the error The request signature we calculated does not match the signature you provided. Check your key and signing method..

How to reproduce

Run the following with valid values for bucket and prefix:

aws_s3:list_objects(aws_client:make_client(), <<"bucket">>, <<"/">>, undefined, undefined, undefined, <<"prefix">>, undefined).

The result is an error specifying that The request signature we calculated does not match the signature you provided. Check your key and signing method..

Root cause

The querystring parameters are sorted in reverse order. The request sends ?prefix=prefix&delimiter=%2F and the expected querystring as reported by the error returned from AWS should be ?delimiter=%2F&prefix=prefix.

Add Secrets Manager AWS API

Or alternatively automatically process all directories in models/apis and determine if the protocol they use is implemented. Then proceed to generated the service module when applicable taking into consideration the language (i.e. elixir or erlang) specified.

Currently we handle header and uri parameters but don't include those which have "location":"querystring" in the specification.

For example in the S3 service the Delimiter querystring parameter is being ignored in the generation.

There is a suggestion from @mrallen1 to migrate the Erlang client from using jsx to using jsone. I'm creating this ticket so we can discuss the migration here and close #6.

The idea is to improve the docs we generate for each module and function.

Some improvements are:

• split the first sentence from the rest;
• remove unnecessary HTML from the docs;
• fix broken markdown links;
• use markdown lists instead of HTML lists.

In the URL building logic for (I presume) all services, a prefix is currently being prepended to the endpoint host. I am using Digital Ocean and their Spaces API aims for AWS S3 parity, so I was hoping to use this lib to interface with it. However, their service does not use a prefix like s3 for obvious reasons. Since that is currently being hard coded into the url, I was getting a very cryptic error. I was able to work around the issue by adding an option to the client that instructs the service to skip the prefix in the host: tfwright/aws-elixir@72b45d2

If this is something you'd be open to adding, I can open a PR on this repo. There may be other solutions, like just requiring the user to specify the full url as the endpoint rather than composing it out of options, which might be a simpler and more flexible route to go.

In #67 we introduced a receive_body_as_binary option which is currently only available in the generated Erlang code.
Ensure the option is ported to the generated Elixir code, too.

Hints:

https://github.com/aws-beam/aws-elixir/blob/master/lib/aws/request.ex#L44
https://github.com/aws-beam/aws-elixir/blob/master/lib/aws/request.ex#L104

Some endpoints don't encode the body in the default Content-Type but allow to provide an arbitrary request body. The specification signals this cases by having Body as the payload.

For example in the S3 service the PutObject operation should be able to accept a binary body:

"PutObjectRequest":{
      "type":"structure",
      "required":[
        "Bucket",
        "Key"
      ],
      "members":{
        "ACL":{
          "shape":"ObjectCannedACL",
          "location":"header",
          "locationName":"x-amz-acl"
        },
        "Body":{
          "shape":"Body",
          "streaming":true
        },
        "Bucket":{
          "shape":"BucketName",
          "location":"uri",
          "locationName":"Bucket"
        },
        "CacheControl":{
          "shape":"CacheControl",
          "location":"header",
          "locationName":"Cache-Control"
        },
        "ContentDisposition":{
          "shape":"ContentDisposition",
          "location":"header",
          "locationName":"Content-Disposition"
        },
        "ContentEncoding":{
          "shape":"ContentEncoding",
          "location":"header",
          "locationName":"Content-Encoding"
        },
        "ContentLanguage":{
          "shape":"ContentLanguage",
          "location":"header",
          "locationName":"Content-Language"
        },
        "ContentLength":{
          "shape":"ContentLength",
          "location":"header",
          "locationName":"Content-Length"
        },
        "ContentMD5":{
          "shape":"ContentMD5",
          "location":"header",
          "locationName":"Content-MD5"
        },
        "ContentType":{
          "shape":"ContentType",
          "location":"header",
          "locationName":"Content-Type"
        },
        "Expires":{
          "shape":"Expires",
          "location":"header",
          "locationName":"Expires"
        },
        "GrantFullControl":{
          "shape":"GrantFullControl",
          "location":"header",
          "locationName":"x-amz-grant-full-control"
        },
        "GrantRead":{
          "shape":"GrantRead",
          "location":"header",
          "locationName":"x-amz-grant-read"
        },
        "GrantReadACP":{
          "shape":"GrantReadACP",
          "location":"header",
          "locationName":"x-amz-grant-read-acp"
        },
        "GrantWriteACP":{
          "shape":"GrantWriteACP",
          "location":"header",
          "locationName":"x-amz-grant-write-acp"
        },
        "Key":{
          "shape":"ObjectKey",
          "location":"uri",
          "locationName":"Key"
        },
        "Metadata":{
          "shape":"Metadata",
          "location":"headers",
          "locationName":"x-amz-meta-"
        },
        "ServerSideEncryption":{
          "shape":"ServerSideEncryption",
          "location":"header",
          "locationName":"x-amz-server-side-encryption"
        },
        "StorageClass":{
          "shape":"StorageClass",
          "location":"header",
          "locationName":"x-amz-storage-class"
        },
        "WebsiteRedirectLocation":{
          "shape":"WebsiteRedirectLocation",
          "location":"header",
          "locationName":"x-amz-website-redirect-location"
        },
        "SSECustomerAlgorithm":{
          "shape":"SSECustomerAlgorithm",
          "location":"header",
          "locationName":"x-amz-server-side-encryption-customer-algorithm"
        },
        "SSECustomerKey":{
          "shape":"SSECustomerKey",
          "location":"header",
          "locationName":"x-amz-server-side-encryption-customer-key"
        },
        "SSECustomerKeyMD5":{
          "shape":"SSECustomerKeyMD5",
          "location":"header",
          "locationName":"x-amz-server-side-encryption-customer-key-MD5"
        },
        "SSEKMSKeyId":{
          "shape":"SSEKMSKeyId",
          "location":"header",
          "locationName":"x-amz-server-side-encryption-aws-kms-key-id"
        },
        "SSEKMSEncryptionContext":{
          "shape":"SSEKMSEncryptionContext",
          "location":"header",
          "locationName":"x-amz-server-side-encryption-context"
        },
        "RequestPayer":{
          "shape":"RequestPayer",
          "location":"header",
          "locationName":"x-amz-request-payer"
        },
        "Tagging":{
          "shape":"TaggingHeader",
          "location":"header",
          "locationName":"x-amz-tagging"
        },
        "ObjectLockMode":{
          "shape":"ObjectLockMode",
          "location":"header",
          "locationName":"x-amz-object-lock-mode"
        },
        "ObjectLockRetainUntilDate":{
          "shape":"ObjectLockRetainUntilDate",
          "location":"header",
          "locationName":"x-amz-object-lock-retain-until-date"
        },
        "ObjectLockLegalHoldStatus":{
          "shape":"ObjectLockLegalHoldStatus",
          "location":"header",
          "locationName":"x-amz-object-lock-legal-hold"
        }
      },
      "payload":"Body"
    }

The generated code ends up having functions with too many arguments when the operation includes many parameter arguments. This impacts mostly the Erlang client where we don't have options arguments and reduces the usability of the library considerably.

Whenever a call results in an error the path the error message varies depending on the AWS service and the type of error. The path is case-sensitive and some errors contain the field message instead of Message.

It is therefore a simpler approach to just return the whole (decoded) error information to the caller.

We reached out to AWS regarding the source of truth for the specification used by the different official AWS clients, since we are currently relying on the one present in the go-aws-sdk repository.

After some investigation on AWS, their response was the following:

These JSON file formats are intentionally not publicly defined. These files are generated using an internal process based on service API model definitions. While customers could reverse engineer these files, we don't provide any support for their format nor contents. So many third party SDKs rely on them that they're basically a de-facto standard at this point.

So we then asked if our current approach of using the ones from the aws-sdk-go project is the way to go for us. This was confirmed by the AWS person.

(I'm creating this issue to share and document this conversation for the future)

Opening this issue for transparency and as an announcement to the community that we will continue to support aws-elixir and aws-erlang into the future of AWS SDK v2! 🎉📢

AWS announced that AWS SDK Go (v1) will go into maintenance mode as of July 31, 2024. ⚠️ From that point on, the SDK will not receive API updates for new or existing services, or be updated to support new regions. This is problematic for aws-beam as we depend on the aws-sdk-go 🛠️ to generate the code for aws-elixir and aws-erlang.

This issue will encapsulate the work of migrating to aws-sdk-go-v2 instead. 🔄

The following goals must be upheld: ✔️

• Interfaces to the clients that have integrated with either aws-elixir and aws-erlang MAY NOT change at all. Any changes MUST BE transparent and happen "under the hood" 🕵️
• NO new non-backwards compatible features should be included in order to make the migration transparent to the clients 🚫
• NO unexpected diff may occur in either repo. ❌ Semantically, they should produce the same code with some exceptions:
  • SuccessStatusCode is not always defined yet the new repo ensures a default of 200 whereas now we have undefined. This is broken and we can sneak in the fix in this as part of this work 🛠️
  • Some paths have changed as part of the new specification. It may be that we generated old code erroneously or misunderstood the spec. Either way, the paths will be fixed as part of the v2 API. This should in theory be fine 👍🔧
  • Docs have changed, this is expected and has no impact on compiled code. This is acceptable. 📚

Right now both of the implementations use aws as an application name, which may conflict if there will be need to include both of them in a single project (for example as a transitive dependency).

Possible solutions:

• rename one (or both) applications to non-conflicting names
• make Elixir project thin wrapper over Erlang project and rename Elixir application
• merge both projects, make Elixir code wrapper over Erlang implementation and allow building project using both Rebar3 and Mix (this is IMHO the best way that is used for example by opentelemetry_api)

We should handle the special case of s3-control in the Elixir rest-xml template in the same way it is handled in the Erlang template.

The aws-codegen, aws-erlang and aws-elixir repos are forks of the respective jkakar/* versions.
Nowadays, the jkakarrepos are outdated and they point to aws-beam from their respective READMEs.
The current setup is suboptimal since, whenever a PR is raised in one of the aws-beam repositories, jkakar is marked as the default "base repository":

I contacted GitHub support to check if there is an option to update the base repo, to make contributors life easier. They gave the following two options:

1. We can extract your forks from the current repository network. This will leave the repositories owned by jkakar as root repositories, but the aws-beam repositories and the child repositories under them would be moved to their own network where the aws-beam repositories are the root. I can do this with just your confirming that this is what you want to do.

2. We can update the current repository network so that the aws-beam repositories become the root repository for the network and the ones owned by jkakar would become children of those repos. This would leave all repositories currently under the jkakar repositories in the same fork network, but would require permission from jkakar to make the change.

Of course we also have option 3. which is leaving things as they are.

@jkakar @philss Any opinions?

The Mechanical Turk API is not regionalized but doesn't include information for aws-global, it has an entry for sandbox instead (from endpoints.json):

      "mturk-requester" : {
        "endpoints" : {
          "sandbox" : {
            "hostname" : "mturk-requester-sandbox.us-east-1.amazonaws.com"
          },
          "us-east-1" : { }
        },
        "isRegionalized" : false
      },

This special case needs to be addressed in the generation somehow.

This would enable support for the s3 API.

See aws-beam/aws-erlang#23.

The behaviour will be the same, just to make it easier to spot was is generated and what is not.

There is a big difference between the list of generated APIs for Elixir and Erlang, it would be nice for them to be the same once #13 is merged.

There is currently no test that verifies that the application can successfully generate code.

Adding a test that fetches the aws-go-sdk repository and generates both Erlang and Elixir code by running mix run generate.exs ... is the baseline. Anything more than that would be nice to have.

The rest_xml.[ex|erl].eex and rest_json.[ex|erl].eex are almost the same except for the encoding/decoding of request/response. This results in a lot of code duplication which can be reduced if they are unified.

The same applies to the query.[ex|erl].eex and json.[ex|erl].eex templates.

This would enable support for the ec2 API.

In services such as S3, the body of the response can be in the form of a non-encoded binary. Similarly to the work done in #62, we should detect these cases and skip decoding.

This would enable support fot the SNS and SQS APIs.

The specification for the AWS APIs sometimes provides a smoke.json file with the details on what operation/action can be used to smoke test the client.

This is used by aws-sdk-go as well (see here). For example these are the generated smoke tests for S3 which come from this smoke.json.

This protocol is only used by the EC2 service. It is the only remaining protocol that is not supported.

Add a build pipeline using GitHub actions.

The generated @types names are not properly sanitized. With the type name identifier aws-elixir fails to build for the latest quicksight from aws go sdk. Either it should be added like this, or the other path should be used. It's unclear to me why there are two paths, and what's the best option:
dc18ee4

The {Name+} format for a URI parameter means that there can be one or more values provided separated by /. Therefore each value should be URI encoded separately. We are currently encoding all the values together.

For example in S3 the {Key+} parameter in the PutObject operation.

The approach of official-schema-originated code generation is great.

Is there a plan to provide further support for structured interfaces, particularly structured inputs?
Currently inputs for generated API functions are mere maps,
(example: https://github.com/aws-beam/aws-elixir/blob/master/lib/aws/generated/acm.ex#L55)

  def add_tags_to_certificate(%Client{} = client, input, options \\ []) do
    Request.request_post(client, metadata(), "AddTagsToCertificate", input, options)
  end

but schema JSON provides inputs and their shape definitions.
(https://github.com/aws/aws-sdk-go/blob/main/models/apis/acm/2015-12-08/api-2.json#L16-L32)

    "AddTagsToCertificate":{
      "name":"AddTagsToCertificate",
      "http":{
        "method":"POST",
        "requestUri":"/"
      },
      "input":{"shape":"AddTagsToCertificateRequest"},
      "errors":[
        {"shape":"ResourceNotFoundException"},
        {"shape":"InvalidArnException"},
        {"shape":"InvalidTagException"},
        {"shape":"TooManyTagsException"},
        {"shape":"TagPolicyException"},
        {"shape":"InvalidParameterException"},
        {"shape":"ThrottlingException"}
      ]
    },

(https://github.com/aws/aws-sdk-go/blob/main/models/apis/acm/2015-12-08/api-2.json#L240-L250)

    "AddTagsToCertificateRequest":{
      "type":"structure",
      "required":[
        "CertificateArn",
        "Tags"
      ],
      "members":{
        "CertificateArn":{"shape":"Arn"},
        "Tags":{"shape":"TagList"}
      }
    },

My naive idea is to generate structs for shapse of "type": "structure" inputs (for aws-elixir), and accept them as input arguments of API functions.

Producing edoc fails for a couple of modules. Examples of invalid generated docstrings are:

From aws_chime.erl:

%% == Functionality & restrictions ==

From aws_forecast.erl:

%% `<ExportJobName>_<ExportTimestamp>_<PartNumber>.csv'

From aws_route53.erl:

-%% table. These characters include the following: `! " # $ % & ' ( ) * + , -'

From aws_codestar_connections.erl:

%% A host created through the CLI or the SDK is in `PENDING` status by

These errors are verified on the Erlang side. Not sure if the Elixir side is also affected, too.

Using the S3 client (Erlang or Elixir) in the following way results in a 403:

1> aws_s3:put_object(aws_client:make_client(), <<"bucket-name">>, <<"object/key">>, #{<<"ACL">> => <<"private">>, <<"ServerSideEncryption">> => <<"AES256">>, <<"ContentType">> => <<"text/plain">>}).

{error,#{<<"Error">> => #{
                %% ...
                <<"Message">> =>
                   <<"The request signature we calculated does not match the signature you provided. Check your key and si"...>>,
                %% ...
             }, ... }

This is because the request includes a duplicated content-type header as a result of providing the ContentType header parameter:

PUT
/bucket-name/object/key

content-type:text/plain
content-type:text/xml
host:s3.eu-west-1.amazonaws.com
x-amz-acl:private
x-amz-content-sha256:e3b0c
x-amz-date:20200814T125145Z
x-amz-security-token:Fwo...
x-amz-server-side-encryption:AES256

We need to make sure the header is added only if it does not already exist in the provided headers.