cloudfoundry / bosh-s3cli Goto Github PK

Hi I'm using this tool to upload to S3 html files, but I need them to be set with metadata Content-Type: text/html. So browsers can display the HTML.

How to do this?

Hi, how can I download an arm64 compatible version of this tool (for Linux)?

FYI https://bosh-cpi.ci.cf-app.com/pipelines/s3cli isn't visible to the public but is mentioned in the readme.

Perhaps it became private by default when upgrading to Concourse Teams/v2.

Hello,

Can you help me, how can I upload any object with public-read ACL? I found this ACL in the go source code, but I do not known how/where can I specify.

Thank you!

This is for the case where a S3 service has a poor certificate - perhaps self-signed, or misconfigured where the common name in the Subject is not also included in the Subject Alternative Name.

Can there be some method to specify that the S3 servers certifcate should not be verified for the 'bosh upload-blobs"

A case example

The "bosh upload-blobs" to the S3 url https://ecslab.example.com fails when the url is not in the certificates Subject Common Name and also not in the Subject Alternative Name (SAN). This is an example where there is a Load Balancer in front of six S3 nodes. The Load Balancer terminates the TLS session and opens a backend connection to one of the S3 nodes.

Blob upload 'my/large_blob' failed

Uploading blobs:

Creating blob for path 'my/large_blob:

Creating blob in inner blobstore:

Generating blobstore ID:

upload failure: RequestError: send request failed

caused by: Post https:/bosh-release-blobstore/93abbe20-335d-4ef8-1234-a0a59d87717a?uploads=: x509: certificate is valid for ecslabn1-node-2.travt.net, ecslabn1-node-1.example.com, ecslabn1-node-6.example.com, ecslabn1-node-5.example.com, ecslabn1-node-4.example.com, ecslabn1-node-3.example.com, not ecslab.example.com

Exit code 1

Bypassing the Load Balancer and directly accessing the backend ecslabn1-node-1.example.com fails because the nodes have a certificate where the Subject Common Name is "localhost" and there is no Subject Alternative Name ...

Blob upload 'my/large_blob' failed

Uploading blobs:

Creating blob for path 'my/large_blob':

Creating blob in inner blobstore:

Generating blobstore ID:

upload failure: RequestError: send request failed

caused by: Post https:/bosh-release-blobstore/bc663d16-5507-47e4-1234-cfc01ebad369?uploads=: x509: certificate is valid for localhost, not ecslabn1-node-1.example.com

I only just learned about s3cli from recent release notes of bosh-cli. The readme infers its primarily a CLI. Perhaps its handy for each github release to include the prebuilt versions from CI - which are a little easier for 3rd party CI systems to discover and download?

Looking through the codebase, there is support for using a Alicloud-centric config when using the s3cli. However, when we look at the pipeline and in the test resources, we see that there are no Alicloud specific integration tests.

@xiaozhu36 could you or your team please write some integration tests that cover general interaction with Alicloud and test any Alicloud specific functionality before we can close this PR?

It would be useful if s3cli could be configured to use server-side encryption, pulling options like ServerSideEncryption, SSECustomerAlgorithm, etc. from the config file and passing them to PutObject. I'm interested in this feature for use with bosh (cloudfoundry/bosh#1278). If you all would be open to it, I'd be happy to submit a pull request.

Sorry for filing yet another "does not install" report, but it's really the case. The README says that the installation command is go get ... however it is officially deprecated and doesn't work:

$ go get github.com/cloudfoundry/bosh-s3cli
go: go.mod file not found in current directory or any parent directory.
	'go get' is no longer supported outside a module.
	To build and install a command, use 'go install' with a version,
	like 'go install example.com/cmd@latest'
	For more information, see https://golang.org/doc/go-get-install-deprecation
	or run 'go help get' or 'go help install'.

go install doesn't work for me either:

$ go install github.com/cloudfoundry/bosh-s3cli@latest
go: downloading github.com/cloudfoundry/bosh-s3cli v0.0.206
go: github.com/cloudfoundry/bosh-s3cli@latest (in github.com/cloudfoundry/[email protected]):
	The go.mod file for the module providing named packages contains one or
	more replace directives. It must not contain directives that would cause
	it to be interpreted differently than if it were the main module.

I am using go version go1.20.3 linux/amd64.

How can I install the app rather than downgrading Go?

(SEE THE SECOND COMMENT FOR A BETTER DESCRIPTION)

At a user level when you try to do a 'bosh upload release' with a Mirantis 9.1 OpenStack cluster (ceph 0.94.6) as your external blob store, it fails with the error:

ERROR -- DirectorJobRunner: Failed to create S3 object, code 1, output: '', error: '2017/02/07 22:11:09 Upload retry limit exceeded: MultipartUpload: upload multipart failed
	upload id: 2~EkVqqp9F03ogG0ICbCIYfYagfmHhI_A
caused by: InvalidArgument:
	status code: 400, request id: tx00000000000000000071f-00589a45fd-68a1d-default
2017/02/07 22:11:09 performing operation put: upload retry limit exceeded: MultipartUpload: upload multipart failed
	upload id: 2~EkVqqp9F03ogG0ICbCIYfYagfmHhI_A
caused by: InvalidArgument:
	status code: 400, request id: tx00000000000000000071f-00589a45fd-68a1d-default
'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/blobstore_client-260.5.0/lib/blobstore_client/s3cli_blobstore_client.rb:115:in `store_in_s3'

This was previously working (and continues to) for the Ops Manager team on their Mirantis 6 OpenStack cluster (ceph 0.80.9).

The store_in_s3 function uses the s3cli command to upload files. We were able to reproduce this problem directly using that command and found that the problem was specific to multi-part uploads. Small uploads work fine.

/:/var/vcap/data/tmp/director# /var/vcap/data/packages/s3cli/8cbc6ee1b5acaac18c63fafc5989bd6911c9be83-f1febb77993dcec28a75c887593e0ad31867096d/bin/s3cli -c s3_blobstore_config-5aeac0fd-0340-48a4-aa18-82186f4a0371 put s3_blobstore_config-5aeac0fd-0340-48a4-aa18-82186f4a0371 foo
2017/02/07 00:01:37 Successfully uploaded file to http:/boshtest/foo

/:/var/vcap/data/tmp/director# /var/vcap/data/packages/s3cli/8cbc6ee1b5acaac18c63fafc5989bd6911c9be83-f1febb77993dcec28a75c887593e0ad31867096d/bin/s3cli -c s3_blobstore_config-5aeac0fd-0340-48a4-aa18-82186f4a0371 put ubuntu-16.04.1-desktop-amd64.iso ubuntu-16.04.1-desktop-amd64.iso
2017/02/07 00:02:14 Upload retry limit exceeded: MultipartUpload: upload multipart failed
    upload id: 2~xCkzsLekMnvMAa_e2VH5cvgqPvsNp3k
caused by: InvalidArgument:
    status code: 400, request id: tx00000000000000002f892-0058990e85-57636-default
2017/02/07 00:02:14 performing operation put: upload retry limit exceeded: MultipartUpload: upload multipart failed
    upload id: 2~xCkzsLekMnvMAa_e2VH5cvgqPvsNp3k
caused by: InvalidArgument:
    status code: 400, request id: tx00000000000000002f892-0058990e85-57636-default

On the OpenStack side we have two sets of logs reporting errors...

The Ceph Rados Gateway logs return a status 400 and the fcgi-pm binary logs a 503.

Toolsmiths team investigated on behalf of the Ops Manager team. Notes in Tracker story:

https://www.pivotaltracker.com/story/show/138956065