AWS.S3.complete_multipart_upload(client, bucket, key, %{}) returns an error about aws-elixir

[dmorn]

OOOOk fot it @onno-vos-dev. This is all about the input payload as expected. After inspecting the encoded request body a found that a proper elixir input is built like

parts =
  Enum.map(uploads, fn {{:ok, etag}, index} ->
    %{"ETag" => etag, "PartNumber" => index}
  end)

input = %{"CompleteMultipartUpload" => %{"Part" => parts}, "UploadId" => upload_id}

And should produce a payload that looks like

<CompleteMultipartUpload>
   <Part>
      <ETag>a2b0962b15f6d5716b3e9df437a8133b</ETag>
      <PartNumber>1</PartNumber>
   </Part>
   <Part>
   ...
   </Part>
</CompleteMultipartUpload>

So that was it, bad input @gabrielmancini ! I believe this issue can be closed @philss, thank you again @onno-vos-dev! 🙌

[onno-vos-dev]

Now I can put Elixir on my CV 😅 Glad to see this is resolved!

Ok if I close this issue?

[philss]

Hi @gabrielmancini 👋

Just to confirm: is the error occurring inside the stream?

Also, have you tried to use the PartNumber parameter for the upload_part/4 call?

Part numbers can be any number from 1 to 10,000, inclusive. A part number uniquely identifies a part and also defines its position within the object being created. If you upload a new part using the same part number that was used with a previous part, the previously uploaded part is overwritten. Each part must be at least 5 MB in size, except the last part. There is no size limit on the last part of your multipart upload.

https://hexdocs.pm/aws/AWS.S3.html#upload_part/5

[dmorn]

I think your code produces the second payload I posted @onno-vos-dev right?

[gabrielmancini]

hello @philss thanks for the reply ;-)

i change a little bit the code but i had the same issue, pls take a look.

the code:

{:ok,
     %{
       "InitiateMultipartUploadResult" => %{
         "UploadId" => key
       }
     }, _} = AWS.S3.create_multipart_upload(client, bucket, path, %{})

    # the magic "must" happend! "bias detected"
    Stream.concat([head], rest)
    |> Stream.chunk_every(chunk_size)
    |> Stream.with_index(1)
    |> Enum.map(fn {chunk, i} ->
      chunk_s =
        chunk
        |> Enum.join("\r\n")

      size = :erlang.byte_size(chunk_s)
      IO.inspect({i, size / @megabyte})
      AWS.S3.upload_part(client, bucket, path, %{
        "Body" => chunk_s,
        "ContentMD5" => :crypto.hash(:md5, chunk_s) |> Base.encode64(),
        "PartNumber" => i,
        "UploadId" => key
      })
      # every post returns 200 ok
      |> IO.inspect()
    end)

    # https://github.com/aws-beam/aws-elixir/issues/84
    AWS.S3.complete_multipart_upload(client, bucket, path, %{})
    |> IO.inspect()

the error:

{:error,
 {:unexpected_response,
  %{
    body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>RBWVRAPGGPPF746S</RequestId><HostId>z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU=</HostId></Error>",
    headers: [
      {"x-amz-request-id", "RBWVRAPGGPPF746S"},
      {"x-amz-id-2",
       "z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU="},
      {"allow", "HEAD, DELETE, GET, PUT"},
      {"content-type", "application/xml"},
      {"transfer-encoding", "chunked"},
      {"date", "Mon, 02 Aug 2021 15:48:06 GMT"},
      {"server", "AmazonS3"}
    ],
    status_code: 405
  }}}
** (Protocol.UndefinedError) protocol Enumerable not implemented for {:error, {:unexpected_response, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>RBWVRAPGGPPF746S</RequestId><HostId>z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU=</HostId></Error>", headers: [{"x-amz-request-id", "RBWVRAPGGPPF746S"}, {"x-amz-id-2", "z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU="}, {"allow", "HEAD, DELETE, GET, PUT"}, {"content-type", "application/xml"}, {"transfer-encoding", "chunked"}, {"date", "Mon, 02 Aug 2021 15:48:06 GMT"}, {"server", "AmazonS3"}], status_code: 405}}} of type Tuple. This protocol is implemented for the following type(s): Function, MapSet, List, Stream, HashDict, GenEvent.Stream, Map, Date.Range, Range, File.Stream, IO.Stream, HashSet
    (elixir 1.12.0) lib/enum.ex:1: Enumerable.impl_for!/1
    (elixir 1.12.0) lib/enum.ex:141: Enumerable.reduce/3
    (elixir 1.12.0) lib/stream.ex:649: Stream.run/1

all the parts appers to work fine, all of then returns status code 200, like this output sample:

{:ok, nil,
 %{
   body: "",
   headers: [
     {"x-amz-id-2",
      "6LK+sRvE0+hmCpMXepbMJIwN6SLhDntv50CKUNcj/fWqGvvBz38oD2kUhIYvgqs/6ZEw6DKEDNg="},
     {"x-amz-request-id", "RBWYDXCATZ5XVHKC"},
     {"date", "Mon, 02 Aug 2021 15:48:08 GMT"},
     {"etag", "\"94efc8234db38870f667962c3425f5a6\""},
     {"server", "AmazonS3"},
     {"content-length", "0"}
   ],
   status_code: 200
 }}

[dmorn]

Hi there @gabrielmancini and @philss!
I'm currently facing the same issue. One thing that is missing from above is a proper input payload.
I am building it like this

parts =
  Enum.map(uploads, fn {{:ok, etag}, index} ->
    %{"ETag" => etag, "PartNumber" => index}
  end)

input = %{"CompleteMultipartUpload" => parts, "uploadId" => upload_id}
AWS.S3.complete_multipart_upload(client, bucket, key, input)

The error is the same though

"%{body: \"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>C31JVFSFYW0HXKPK</RequestId><HostId>n+/zF/4mGwuKv52tlHEeJbuV1tEn4C8TNhv9T4WXuxTPDecSzxicwKcI8kIngE7bVzgzjB0SbPs=</HostId></Error>\", headers: [{\"x-amz-request-id\", \"C31JVFSFYW0HXKPK\"}, {\"x-amz-id-2\", \"n+/zF/4mGwuKv52tlHEeJbuV1tEn4C8TNhv9T4WXuxTPDecSzxicwKcI8kIngE7bVzgzjB0SbPs=\"}, {\"Allow\", \"HEAD, DELETE, GET, PUT\"}, {\"Content-Type\", \"application/xml\"}, {\"Transfer-Encoding\", \"chunked\"}, {\"Date\", \"Thu, 24 Mar 2022 13:36:15 GMT\"}, {\"Server\", \"AmazonS3\"}, {\"Connection\", \"close\"}], status_code: 405}"

It looks like this endpoint does not allow POST requests (see the Allow response header) 🤔

[dmorn]

This is indeed the culprit! Changing

aws-elixir/lib/aws/generated/s3.ex

Line 212 in a80272a

:post,

to a :put makes it work out 😅 On AWS documentation though they actually say this is supposed to be a POST.

[dmorn]

It is indeed specified as a POST in the spec document https://github.com/aws/aws-sdk-go/blob/a893727e4b5ed70693aa65bd1b081d4cdcd18a4b/models/apis/s3/2006-03-01/api-2.json#L33

[onno-vos-dev]

👋 Not super familiar with the Elixir code but in aws-erlang this is a post and I know for a fact that this seems to work just fine judging by our S3 bucket containing complete gigantic files 🤔

So before going ahead and changing it, I'd like you (or someone else) to really scuba dive and ensure the bug isn't hiding elsewhere.

[dmorn]

Hi @onno-vos-dev, thanks for joining in. I do think that this stinks.

[onno-vos-dev]

@dmorn Oh I'm not arguing with you 😄 For inspiration, this is how our Input looks which judging by yours appears to be slightly different 🤔 Unless I'm more rusty in my Elixir than I think I am 🤔

Input =
    #{<<"UploadId">> => UploadId,
      <<"CompleteMultipartUpload">> =>
        #{<<"Part">> =>
            [#{<<"ETag">> => Etag, <<"PartNumber">> => PartNr} || {PartNr, Etag} <- PartEtags]}},

[dmorn]

Uh uh this is different indeed 😜 My erlang may be rusty now, it this what you're producing here?

%{
  "CompleteMultipartUpload" => [
    %{
      "Part" => %{
        "ETag" => "5592cc66124693a066c16198bc40e330",
        "PartNumber" => 1
      }
    },
    %{
      "Part" => %{
        "ETag" => "75afa18db5dfe9f747e39f03c2152c80",
        "PartNumber" => 2
      }
    },
    %{"Part" => %{"ETag" => "1668a6efbd8003e55abef8d7a48e2851", ...}},
    %{"Part" => %{...}},
    %{...},
    ...
  ],
  "uploadId" => "elided"
}

The uploadId is lowercased here cause it is turned into a query parameter and should not be part of the payload (I double checked both the code and tried it with the uppercased version, I get a malformed input error from AWS)

[dmorn]

Anyway it does not matter if I build it like above or like

%{
  "CompleteMultipartUpload" => %{
    "Part" => [
      %{"ETag" => "5592cc66124693a066c16198bc40e330", "PartNumber" => 1},
      %{"ETag" => "75afa18db5dfe9f747e39f03c2152c80", "PartNumber" => 2},
      %{"ETag" => "1668a6efbd8003e55abef8d7a48e2851", ...},
      %{...},
      ...
    ]
  },
  "uploadId" => "VUxIR9hGxsV0i90Gy8FKPI7lNuSyYWpEVVkbYneA65sRDCfL21iDJkEXgDLWJiTHSv8rGaxVmVivAmMpp09xAQ--"
}

The error persists 😅

[onno-vos-dev]

Anyway it does not matter if I build it like above or like

%{
  "CompleteMultipartUpload" => %{
    "Part" => [
      %{"ETag" => "5592cc66124693a066c16198bc40e330", "PartNumber" => 1},
      %{"ETag" => "75afa18db5dfe9f747e39f03c2152c80", "PartNumber" => 2},
      %{"ETag" => "1668a6efbd8003e55abef8d7a48e2851", ...},
      %{...},
      ...
    ]
  },
  "uploadId" => "VUxIR9hGxsV0i90Gy8FKPI7lNuSyYWpEVVkbYneA65sRDCfL21iDJkEXgDLWJiTHSv8rGaxVmVivAmMpp09xAQ--"
}

The error persists sweat_smile

That looks similar to what I generate:

#{<<"CompleteMultipartUpload">> =>
      #{<<"Part">> =>
            [#{<<"ETag">> => <<"etag1">>,<<"PartNumber">> => 1},
             #{<<"ETag">> => <<"etag2">>,<<"PartNumber">> => 2},
             #{<<"ETag">> => <<"etag3">>,<<"PartNumber">> => 3}]},
  <<"UploadId">> => <<"my-elixir-is-terrible">>}

So the plot thickens 🤔 I need to do some thinking and digging.

[dmorn]

I just double checked how ExAws works as we do multi-part uploads with that lib in another project. The input payload they produce resembles the first one of the last above and they do indeed POST.

[dmorn]

@onno-vos-dev 😂 Yes I'm OK with it!

[ferd]

This is an old issue, but I wanted to leave a code sample for multipart uploading with checksums and hashes included, in case anyone else ever needs it.

As it turns out, you need to upload the checksums of each part you upload, store the ETags given, and then re-send all the sections in the completion step, and the checsum you get back is the checksum of all checksums with a suffix that contains the number of parts involved; any moving of that file after the fact (if under 5GB) re-writes the checksum as well: https://github.com/ferd/ReVault/blob/5fede9f9459eff85db5643b5d5f37e2f3c7a1429/apps/revault/test/s3_integration_SUITE.erl#L195-L283 (do note also that under that mode with checksum, all part uploads need to be sequential)