Comments (17)
OOOOk fot it @onno-vos-dev. This is all about the input payload as expected. After inspecting the encoded request body a found that a proper elixir input is built like
parts =
Enum.map(uploads, fn {{:ok, etag}, index} ->
%{"ETag" => etag, "PartNumber" => index}
end)
input = %{"CompleteMultipartUpload" => %{"Part" => parts}, "UploadId" => upload_id}
And should produce a payload that looks like
<CompleteMultipartUpload>
<Part>
<ETag>a2b0962b15f6d5716b3e9df437a8133b</ETag>
<PartNumber>1</PartNumber>
</Part>
<Part>
...
</Part>
</CompleteMultipartUpload>
So that was it, bad input @gabrielmancini ! I believe this issue can be closed @philss, thank you again @onno-vos-dev! 🙌
from aws-elixir.
Now I can put Elixir on my CV 😅 Glad to see this is resolved!
Ok if I close this issue?
from aws-elixir.
Hi @gabrielmancini 👋
Just to confirm: is the error occurring inside the stream?
Also, have you tried to use the PartNumber
parameter for the upload_part/4
call?
Part numbers can be any number from 1 to 10,000, inclusive. A part number uniquely identifies a part and also defines its position within the object being created. If you upload a new part using the same part number that was used with a previous part, the previously uploaded part is overwritten. Each part must be at least 5 MB in size, except the last part. There is no size limit on the last part of your multipart upload.
https://hexdocs.pm/aws/AWS.S3.html#upload_part/5
from aws-elixir.
I think your code produces the second payload I posted @onno-vos-dev right?
from aws-elixir.
hello @philss thanks for the reply ;-)
i change a little bit the code but i had the same issue, pls take a look.
the code:
{:ok,
%{
"InitiateMultipartUploadResult" => %{
"UploadId" => key
}
}, _} = AWS.S3.create_multipart_upload(client, bucket, path, %{})
# the magic "must" happend! "bias detected"
Stream.concat([head], rest)
|> Stream.chunk_every(chunk_size)
|> Stream.with_index(1)
|> Enum.map(fn {chunk, i} ->
chunk_s =
chunk
|> Enum.join("\r\n")
size = :erlang.byte_size(chunk_s)
IO.inspect({i, size / @megabyte})
AWS.S3.upload_part(client, bucket, path, %{
"Body" => chunk_s,
"ContentMD5" => :crypto.hash(:md5, chunk_s) |> Base.encode64(),
"PartNumber" => i,
"UploadId" => key
})
# every post returns 200 ok
|> IO.inspect()
end)
# https://github.com/aws-beam/aws-elixir/issues/84
AWS.S3.complete_multipart_upload(client, bucket, path, %{})
|> IO.inspect()
the error:
{:error,
{:unexpected_response,
%{
body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>RBWVRAPGGPPF746S</RequestId><HostId>z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU=</HostId></Error>",
headers: [
{"x-amz-request-id", "RBWVRAPGGPPF746S"},
{"x-amz-id-2",
"z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU="},
{"allow", "HEAD, DELETE, GET, PUT"},
{"content-type", "application/xml"},
{"transfer-encoding", "chunked"},
{"date", "Mon, 02 Aug 2021 15:48:06 GMT"},
{"server", "AmazonS3"}
],
status_code: 405
}}}
** (Protocol.UndefinedError) protocol Enumerable not implemented for {:error, {:unexpected_response, %{body: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>RBWVRAPGGPPF746S</RequestId><HostId>z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU=</HostId></Error>", headers: [{"x-amz-request-id", "RBWVRAPGGPPF746S"}, {"x-amz-id-2", "z6b1vYUzI5NAdvm/yIzpJ0kt+0iKQ0ZSQAHRpzMgs9BJDZ2Kz0QJ9FYjYpzpO3Ko1J498e+JKgU="}, {"allow", "HEAD, DELETE, GET, PUT"}, {"content-type", "application/xml"}, {"transfer-encoding", "chunked"}, {"date", "Mon, 02 Aug 2021 15:48:06 GMT"}, {"server", "AmazonS3"}], status_code: 405}}} of type Tuple. This protocol is implemented for the following type(s): Function, MapSet, List, Stream, HashDict, GenEvent.Stream, Map, Date.Range, Range, File.Stream, IO.Stream, HashSet
(elixir 1.12.0) lib/enum.ex:1: Enumerable.impl_for!/1
(elixir 1.12.0) lib/enum.ex:141: Enumerable.reduce/3
(elixir 1.12.0) lib/stream.ex:649: Stream.run/1
all the parts appers to work fine, all of then returns status code 200, like this output sample:
{:ok, nil,
%{
body: "",
headers: [
{"x-amz-id-2",
"6LK+sRvE0+hmCpMXepbMJIwN6SLhDntv50CKUNcj/fWqGvvBz38oD2kUhIYvgqs/6ZEw6DKEDNg="},
{"x-amz-request-id", "RBWYDXCATZ5XVHKC"},
{"date", "Mon, 02 Aug 2021 15:48:08 GMT"},
{"etag", "\"94efc8234db38870f667962c3425f5a6\""},
{"server", "AmazonS3"},
{"content-length", "0"}
],
status_code: 200
}}
from aws-elixir.
Hi there @gabrielmancini and @philss!
I'm currently facing the same issue. One thing that is missing from above is a proper input payload.
I am building it like this
parts =
Enum.map(uploads, fn {{:ok, etag}, index} ->
%{"ETag" => etag, "PartNumber" => index}
end)
input = %{"CompleteMultipartUpload" => parts, "uploadId" => upload_id}
AWS.S3.complete_multipart_upload(client, bucket, key, input)
The error is the same though
"%{body: \"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<Error><Code>MethodNotAllowed</Code><Message>The specified method is not allowed against this resource.</Message><Method>POST</Method><ResourceType>OBJECT</ResourceType><RequestId>C31JVFSFYW0HXKPK</RequestId><HostId>n+/zF/4mGwuKv52tlHEeJbuV1tEn4C8TNhv9T4WXuxTPDecSzxicwKcI8kIngE7bVzgzjB0SbPs=</HostId></Error>\", headers: [{\"x-amz-request-id\", \"C31JVFSFYW0HXKPK\"}, {\"x-amz-id-2\", \"n+/zF/4mGwuKv52tlHEeJbuV1tEn4C8TNhv9T4WXuxTPDecSzxicwKcI8kIngE7bVzgzjB0SbPs=\"}, {\"Allow\", \"HEAD, DELETE, GET, PUT\"}, {\"Content-Type\", \"application/xml\"}, {\"Transfer-Encoding\", \"chunked\"}, {\"Date\", \"Thu, 24 Mar 2022 13:36:15 GMT\"}, {\"Server\", \"AmazonS3\"}, {\"Connection\", \"close\"}], status_code: 405}"
It looks like this endpoint does not allow POST requests (see the Allow response header) 🤔
from aws-elixir.
This is indeed the culprit! Changing
aws-elixir/lib/aws/generated/s3.ex
Line 212 in a80272a
from aws-elixir.
It is indeed specified as a POST in the spec document https://github.com/aws/aws-sdk-go/blob/a893727e4b5ed70693aa65bd1b081d4cdcd18a4b/models/apis/s3/2006-03-01/api-2.json#L33
from aws-elixir.
👋 Not super familiar with the Elixir code but in aws-erlang this is a post
and I know for a fact that this seems to work just fine judging by our S3 bucket containing complete gigantic files 🤔
So before going ahead and changing it, I'd like you (or someone else) to really scuba dive and ensure the bug isn't hiding elsewhere.
from aws-elixir.
Hi @onno-vos-dev, thanks for joining in. I do think that this stinks.
from aws-elixir.
@dmorn Oh I'm not arguing with you 😄 For inspiration, this is how our Input
looks which judging by yours appears to be slightly different 🤔 Unless I'm more rusty in my Elixir than I think I am 🤔
Input =
#{<<"UploadId">> => UploadId,
<<"CompleteMultipartUpload">> =>
#{<<"Part">> =>
[#{<<"ETag">> => Etag, <<"PartNumber">> => PartNr} || {PartNr, Etag} <- PartEtags]}},
from aws-elixir.
Uh uh this is different indeed 😜 My erlang may be rusty now, it this what you're producing here?
%{
"CompleteMultipartUpload" => [
%{
"Part" => %{
"ETag" => "5592cc66124693a066c16198bc40e330",
"PartNumber" => 1
}
},
%{
"Part" => %{
"ETag" => "75afa18db5dfe9f747e39f03c2152c80",
"PartNumber" => 2
}
},
%{"Part" => %{"ETag" => "1668a6efbd8003e55abef8d7a48e2851", ...}},
%{"Part" => %{...}},
%{...},
...
],
"uploadId" => "elided"
}
The uploadId is lowercased here cause it is turned into a query parameter and should not be part of the payload (I double checked both the code and tried it with the uppercased version, I get a malformed input error from AWS)
from aws-elixir.
Anyway it does not matter if I build it like above or like
%{
"CompleteMultipartUpload" => %{
"Part" => [
%{"ETag" => "5592cc66124693a066c16198bc40e330", "PartNumber" => 1},
%{"ETag" => "75afa18db5dfe9f747e39f03c2152c80", "PartNumber" => 2},
%{"ETag" => "1668a6efbd8003e55abef8d7a48e2851", ...},
%{...},
...
]
},
"uploadId" => "VUxIR9hGxsV0i90Gy8FKPI7lNuSyYWpEVVkbYneA65sRDCfL21iDJkEXgDLWJiTHSv8rGaxVmVivAmMpp09xAQ--"
}
The error persists 😅
from aws-elixir.
Anyway it does not matter if I build it like above or like
%{ "CompleteMultipartUpload" => %{ "Part" => [ %{"ETag" => "5592cc66124693a066c16198bc40e330", "PartNumber" => 1}, %{"ETag" => "75afa18db5dfe9f747e39f03c2152c80", "PartNumber" => 2}, %{"ETag" => "1668a6efbd8003e55abef8d7a48e2851", ...}, %{...}, ... ] }, "uploadId" => "VUxIR9hGxsV0i90Gy8FKPI7lNuSyYWpEVVkbYneA65sRDCfL21iDJkEXgDLWJiTHSv8rGaxVmVivAmMpp09xAQ--" }The error persists sweat_smile
That looks similar to what I generate:
#{<<"CompleteMultipartUpload">> =>
#{<<"Part">> =>
[#{<<"ETag">> => <<"etag1">>,<<"PartNumber">> => 1},
#{<<"ETag">> => <<"etag2">>,<<"PartNumber">> => 2},
#{<<"ETag">> => <<"etag3">>,<<"PartNumber">> => 3}]},
<<"UploadId">> => <<"my-elixir-is-terrible">>}
So the plot thickens 🤔 I need to do some thinking and digging.
from aws-elixir.
I just double checked how ExAws works as we do multi-part uploads with that lib in another project. The input payload they produce resembles the first one of the last above and they do indeed POST.
from aws-elixir.
@onno-vos-dev 😂 Yes I'm OK with it!
from aws-elixir.
This is an old issue, but I wanted to leave a code sample for multipart uploading with checksums and hashes included, in case anyone else ever needs it.
As it turns out, you need to upload the checksums of each part you upload, store the ETags given, and then re-send all the sections in the completion step, and the checsum you get back is the checksum of all checksums with a suffix that contains the number of parts involved; any moving of that file after the fact (if under 5GB) re-writes the checksum as well: https://github.com/ferd/ReVault/blob/5fede9f9459eff85db5643b5d5f37e2f3c7a1429/apps/revault/test/s3_integration_SUITE.erl#L195-L283 (do note also that under that mode with checksum, all part uploads need to be sequential)
from aws-elixir.
Related Issues (20)
- AWS.ApiGatewayManagementApi, URL and headers with stage HOT 1
- S3 put_object returns {:error, :timeout} but uploads file correctly HOT 5
- `AWS.S3.delete_objects` content md5 header HOT 5
- SelectObjectContent response HOT 1
- `AWS.STS.assume_role_with_web_identity` should not rely on access_key_id and secret_access_key
- Error calling textractor HOT 3
- Setting endpoint to `amazonaws.com` results in different behaviour as not setting an endpoint HOT 1
- AWS.S3.get_object always returns {:error, :timeout} with hackney > 1.18.1 HOT 5
- Streaming response handling HOT 10
- `put_bucket_cors` request is incorrect HOT 5
- Setup pipeline to automatically roll out new releases upon tagging
- S3 ListParts returns wrong types HOT 7
- QLDB Commit Transaction HOT 2
- NXDOMAIN when calling CostExplorer.get_cost_and_usage HOT 5
- Cannot get AWS Polly synthesize_speech to work HOT 3
- :nxdomain when calling AWS.CloudFront.list_distributions HOT 5
- Wrong region for Route53 requests HOT 12
- AWS.TimestreamWrite does not work. HOT 5
- Support array member names
- New spec is specific to keys in requests being strings
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aws-elixir.