Multipart Upload Issue(s) about oci-go-sdk

[jasonyin]

Thanks for reporting the issue. Is it for uploadPart API? Are you using a proxy server? Do you have same issue for PutObject API (single upload).

[gcstang]

yes it's for upload part api. put object worked correctly. No, not using a proxy.

…

On Sun, Apr 8, 2018, 23:20 Jason Yin ***@***.***> wrote: Thanks for reporting the issue. Is it for uploadPart API? Are you using a proxy server? Do you have same issue for PutObject API (single upload). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#105 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADBkKcILp9tqUFIk8txHO_aTCUokuuRHks5tmuGQgaJpZM4TK09D> .

[jasonyin]

For your first timeout error, you can either change the clientTimeout, but for larger object or unreliable network which takes un-predictable time to upload (wait for responseHeader come back). You could also set the http.Transport.ResponseHeaderTimeout to 0.

BTW, for "latest version of oci-go-sdk", were you referring to V1.2.0?

Looks like you are not using the default httpClient or http.Transport? If yes, could you pls share the code so I can try to repro the issue you have?

[gcstang]

I'm using the Latest version of Master and whatever httpClient is built into the code pkgs
"github.com/oracle/oci-go-sdk/common"
"github.com/oracle/oci-go-sdk/example/helpers"
"github.com/oracle/oci-go-sdk/objectstorage"

[jasonyin]

I need more information to debug the issue, especially how you split the file into multi-parts. Sharing the code would be helpful, however, with limited information here, I noticed that the http.ContentLength is different than the body length which will cause golang http transport through error. Could be an bug around 'UploadPartBody' for UploadPartRequest. I would try to set it as

UploadPartBody: ioutil.NopCloser(bytes.NewReader(buffer)),

net/http: HTTP/1.x transport connection broken: http: ContentLength=104857600 with Body length 385447926

After Change to smaller split size:
net/http: HTTP/1.x transport connection broken: http: ContentLength=20971520 with Body length 385447926

[gcstang]

@jasonyin I'll put something together and add it to the ticket but this is part of a larger application so it will take me a little while.

Not sure this small part will help, otherwise I'll work on the attachment.

`func (o *ObjStore) UploadPart(bucketName, objectName, uploadId string, partNo, contentLength int, content io.ReadCloser) *string {
namespace := o.GetNamespace()
request := objectstorage.UploadPartRequest{
NamespaceName:&namespace,
BucketName:&bucketName,
ObjectName:&objectName,
UploadId:&uploadId,
UploadPartNum:&partNo,
ContentLength:&contentLength,
UploadPartBody:content,
}

r, err := o.client.UploadPart(*o.context, request)
helpers.LogIfError(err)

return r.ETag

}`

[jasonyin]

Thanks for sharing the info!

There are two key input parameters here, contentLength and content (in your sample code):

1. The contentLength is the length of the part you are uploading (100M or 20M in your error trace) which is correct;

2. Body length (shows in the error trace) is 368M fixed which looks wrong to me, it should be same as the part size you are uploading (i.e. 20M or 100M depends on the size of part), it's not the size of the file you are uploading (368M). The body length in the error trace comes from your content parameter( io.ReaderCloser), I suspect that the content parameter you are using was wrong. Here is a sample code, hoping it helps to debug the issue:

// make a buffer with partSize (i.e. 20M or 100M)
buffer := make([]byte, partSize)

// load content from the file you are uploading into a buffer
_, err := file.ReadAt(buffer, offset)

// try to update your code with following line, I think it most likely would fix the issue
content := ioutil.NopCloser(bytes.NewReader(buffer))

Please let me know if you have any questions/issues, thanks!

---

net/http: HTTP/1.x transport connection broken: http: ContentLength=104857600 with Body length 385447926

After Change to smaller split size:
net/http: HTTP/1.x transport connection broken: http: ContentLength=20971520 with Body length 385447926

[gcstang]

I found the issue with this particular piece.

Thank you for your time.

[jasonyin]

glad you got the issue resolved, could you pls share the root cause for the issue? is it because the upload part stream you were using was entire file not the split part?

[gcstang]

@jasonyin yes I had a typo in there (wrong var)