Can't put object to AWS via Zenko about zenko

[LaureVergeron]

To add on this: we couldn't replicate to aws either, but writing to local storage worked fine. I've had similar issues last week in Japan.
Samuel's instance, like mine, is managed by Orbit, and thus the aws location definition was probed by Orbit.

[ssalaues]

@LaureVergeron as far as the replication issue goes, there is a current known problem with our backbeat-producer pod where it hangs on startup but functions normally upon restart/delete. There is an open PR into backbeat scality/backbeat#296 that will fix this. In the meantime, it's suggested that after deploying zenko to delete the backbeat-producer pod before attempting replication.

@samuellugassy
Could you post the logs on your cloudserver-front pod?

[samuellugassy]

Here are the cloudserver-front logs:

kubectl logs zenko-cloudserver-front-6b869f97cd-r75nx
Host name has been modified to zenko-cloudserver-front zenko.local
Note: In your /etc/hosts file on Linux, OS X, or Unix with root permissions, make sure to associate 127.0.0.1 with zenko-cloudserver-front zenko.local
Log level has been modified to info

[email protected] start_s3server /usr/src/app
node index.js

{"name":"S3","time":1527496808151,"level":"warn","message":"scality kms unavailable. Using file kms backend unless mem specified.","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":19}
{"name":"S3","time":1527496808628,"workerId":1,"workerPid":25,"level":"info","message":"new worker forked","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":19}
{"name":"S3","time":1527496808804,"level":"info","message":"connected to mongodb","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":19}
{"name":"S3","time":1527496811296,"level":"warn","message":"scality kms unavailable. Using file kms backend unless mem specified.","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","time":1527496811436,"https":false,"level":"info","message":"Http server configuration","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","time":1527496811547,"address":"::","port":8000,"pid":25,"level":"info","message":"server started","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx"}
{"name":"S3","time":1527496811630,"level":"info","message":"connected to mongodb","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","time":1527496816499,"req_id":"5b828ecd1b4f80289983","level":"info","message":"initialized management database","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","time":1527496816739,"req_id":"5b828ecd1b4f80289983","level":"info","message":"returning existing token","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","configurationVersion":8,"time":1527496817627,"req_id":"5b828ecd1b4f80289983","level":"info","message":"applied configuration version","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","time":1527496817629,"level":"info","message":"connecting to push server","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","time":1527496817674,"req_id":"5b828ecd1b4f80289983","level":"info","message":"this deployment's Instance ID is 5eb85832-3c7a-4050-a6b0-4e7723973ea1","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","time":1527496817674,"req_id":"5b828ecd1b4f80289983","elapsed_ms":1234.305799,"level":"info","message":"management init done","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","time":1527496818737,"level":"info","message":"connected to push server","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","time":1527496819036,"req_id":"b3a71e021c4bb771bdc2","level":"info","message":"overlay push processed","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","method":"getCRRStats","time":1527496829322,"req_id":"0aae267cd0409eae0054","level":"debug","message":"getting CRR stats","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","method":"getCRRStats","error":{"code":"ECONNREFUSED","errno":"ECONNREFUSED","syscall":"connect","address":"10.99.132.108","port":8900},"time":1527496829439,"req_id":"0aae267cd0409eae0054","level":"error","message":"failed to get CRR stats","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","method":"getCRRStats","time":1527496839041,"req_id":"23b1d551f043a56a0c5f","level":"debug","message":"getting CRR stats","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","method":"getCRRStats","error":{"code":"ECONNREFUSED","errno":"ECONNREFUSED","syscall":"connect","address":"10.99.132.108","port":8900},"time":1527496839092,"req_id":"23b1d551f043a56a0c5f","level":"error","message":"failed to get CRR stats","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","method":"getCRRStats","time":1527496848895,"req_id":"7d1de8cfaf47a1297d1f","level":"debug","message":"getting CRR stats","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","method":"getCRRStats","error":{"code":"ECONNREFUSED","errno":"ECONNREFUSED","syscall":"connect","address":"10.99.132.108","port":8900},"time":1527496848914,"req_id":"7d1de8cfaf47a1297d1f","level":"error","message":"failed to get CRR stats","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","time":1527496878797,"req_id":"8d6ee028c24caaa7995b","level":"info","message":"overlay push processed","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","time":1527496938870,"req_id":"f2150c8ec743bb73e0be","level":"info","message":"overlay push processed","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","time":1527496998857,"req_id":"6c316e69a541b6412273","level":"info","message":"overlay push processed","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","time":1527497058725,"req_id":"e74b7c1b6549ad791742","level":"info","message":"overlay push processed","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","time":1527497118736,"req_id":"2bda4eb97a4191e7b2e3","level":"info","message":"overlay push processed","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","clientIP":"::ffff:127.0.0.1","clientPort":35640,"httpMethod":"GET","httpURL":"/bucket1/?max-keys=1000&versions&prefix&delimiter=%2F","time":1527497144690,"req_id":"ed1cb138d3479cb5e39a","level":"info","message":"received request","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","bucketName":"bucket1","bytesReceived":0,"bodyLength":0,"bytesSent":667,"clientIP":"::ffff:127.0.0.1","clientPort":35640,"httpMethod":"GET","httpURL":"/bucket1/?max-keys=1000&versions&prefix&delimiter=%2F","httpCode":200,"time":1527497144839,"req_id":"ed1cb138d3479cb5e39a","elapsed_ms":148.781656,"level":"info","message":"responded with XML","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","clientIP":"::ffff:127.0.0.1","clientPort":35640,"httpMethod":"GET","httpURL":"/bucket1/?uploads","time":1527497144843,"req_id":"7aa719f314419de5ed10","level":"info","message":"received request","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","bucketName":"bucket1","bytesReceived":0,"bodyLength":0,"bytesSent":259,"clientIP":"::ffff:127.0.0.1","clientPort":35640,"httpMethod":"GET","httpURL":"/bucket1/?uploads","httpCode":200,"time":1527497144868,"req_id":"7aa719f314419de5ed10","elapsed_ms":24.694262,"level":"info","message":"responded with XML","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","clientIP":"::ffff:127.0.0.1","clientPort":35708,"httpMethod":"GET","httpURL":"/","time":1527497153924,"req_id":"1cf7d0687e42a59b2dea","level":"info","message":"received request","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","bytesReceived":0,"bodyLength":0,"bytesSent":562,"clientIP":"::ffff:127.0.0.1","clientPort":35708,"httpMethod":"GET","httpURL":"/","httpCode":200,"time":1527497153943,"req_id":"1cf7d0687e42a59b2dea","elapsed_ms":18.524242,"level":"info","message":"responded with XML","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","clientIP":"::ffff:127.0.0.1","clientPort":35708,"httpMethod":"GET","httpURL":"/bucket1/?max-keys=1000&versions&prefix&delimiter=%2F","time":1527497153954,"req_id":"00608fed0d43a6ee028c","level":"info","message":"received request","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","bucketName":"bucket1","bytesReceived":0,"bodyLength":0,"bytesSent":667,"clientIP":"::ffff:127.0.0.1","clientPort":35708,"httpMethod":"GET","httpURL":"/bucket1/?max-keys=1000&versions&prefix&delimiter=%2F","httpCode":200,"time":1527497153965,"req_id":"00608fed0d43a6ee028c","elapsed_ms":10.499366,"level":"info","message":"responded with XML","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","clientIP":"::ffff:127.0.0.1","clientPort":35708,"httpMethod":"GET","httpURL":"/bucket1/?uploads","time":1527497153969,"req_id":"8c1149439543b530b85e","level":"info","message":"received request","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","bucketName":"bucket1","bytesReceived":0,"bodyLength":0,"bytesSent":259,"clientIP":"::ffff:127.0.0.1","clientPort":35708,"httpMethod":"GET","httpURL":"/bucket1/?uploads","httpCode":200,"time":1527497153982,"req_id":"8c1149439543b530b85e","elapsed_ms":13.043651,"level":"info","message":"responded with XML","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","clientIP":"::ffff:127.0.0.1","clientPort":35708,"httpMethod":"PUT","httpURL":"/bucket1/gemalto.jpg","time":1527497154025,"req_id":"6534b2e9b443bc6013d5","level":"info","message":"received request","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","bucketName":"bucket1","objectKey":"gemalto.jpg","bytesReceived":189862,"bodyLength":189862,"clientIP":"::ffff:127.0.0.1","clientPort":35708,"httpMethod":"PUT","httpURL":"/bucket1/gemalto.jpg","contentLength":189862,"httpCode":200,"time":1527497154198,"req_id":"6534b2e9b443bc6013d5","elapsed_ms":173.152243,"level":"info","message":"responded to request","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","clientIP":"::ffff:127.0.0.1","clientPort":35640,"httpMethod":"GET","httpURL":"/bucket1/?max-keys=1000&versions&prefix&delimiter=%2F","time":1527497154327,"req_id":"59e14f5d8f439126af45","level":"info","message":"received request","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","bucketName":"bucket1","bytesReceived":0,"bodyLength":0,"bytesSent":1112,"clientIP":"::ffff:127.0.0.1","clientPort":35640,"httpMethod":"GET","httpURL":"/bucket1/?max-keys=1000&versions&prefix&delimiter=%2F","httpCode":200,"time":1527497154350,"req_id":"59e14f5d8f439126af45","elapsed_ms":23.77725,"level":"info","message":"responded with XML","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","clientIP":"::ffff:127.0.0.1","clientPort":35640,"httpMethod":"GET","httpURL":"/bucket1/?uploads","time":1527497154359,"req_id":"10572d66624eac1a77c5","level":"info","message":"received request","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","bucketName":"bucket1","bytesReceived":0,"bodyLength":0,"bytesSent":259,"clientIP":"::ffff:127.0.0.1","clientPort":35640,"httpMethod":"GET","httpURL":"/bucket1/?uploads","httpCode":200,"time":1527497154375,"req_id":"10572d66624eac1a77c5","elapsed_ms":16.123853,"level":"info","message":"responded with XML","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","clientIP":"::ffff:127.0.0.1","clientPort":35640,"httpMethod":"HEAD","httpURL":"/bucket1/gemalto.jpg?versionId=39383437323530323834353832323939393939395247303031202032352e30","time":1527497157937,"req_id":"af23a9519a4fbcef5c11","level":"info","message":"received request","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","bucketName":"bucket1","objectKey":"gemalto.jpg","bytesReceived":0,"bodyLength":0,"clientIP":"::ffff:127.0.0.1","clientPort":35640,"httpMethod":"HEAD","httpURL":"/bucket1/gemalto.jpg?versionId=39383437323530323834353832323939393939395247303031202032352e30","httpCode":200,"time":1527497157953,"req_id":"af23a9519a4fbcef5c11","elapsed_ms":16.306579,"level":"info","message":"responded with content headers","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","clientIP":"::ffff:127.0.0.1","clientPort":35640,"httpMethod":"HEAD","httpURL":"/bucket1/gemalto.jpg?versionId=39383437323530323834353832323939393939395247303031202032352e30","time":1527497164835,"req_id":"7383507cd243b5f4da57","level":"info","message":"received request","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}
{"name":"S3","bucketName":"bucket1","objectKey":"gemalto.jpg","bytesReceived":0,"bodyLength":0,"clientIP":"::ffff:127.0.0.1","clientPort":35640,"httpMethod":"HEAD","httpURL":"/bucket1/gemalto.jpg?versionId=39383437323530323834353832323939393939395247303031202032352e30","httpCode":200,"time":1527497164848,"req_id":"7383507cd243b5f4da57","elapsed_ms":13.314978,"level":"info","message":"responded with content headers","hostname":"zenko-cloudserver-front-6b869f97cd-r75nx","pid":25}

[LaureVergeron]

Hey,

Just started a new minikube VM, still can't replicate to AWS from local; configured my aws region from Orbit, probes went through OK + can access target AWS bucket directly using AWS profile.

From the logs, the issue seems to be that Backbeat (Replication Service Account) uses sigv2 when my AWS region expects sigv4 for auth?
Replication Service Account key is: URXDGO3FY11XGQ62OYZD

What I did is create a bucket s3://local in region us-east-1 defined as local fs storage, and asked to replicate that bucket in region aws, defined as zenkobucket in the ironman AWS account.
I then created an object s3://local/test, and that object was not replicated to region aws.

List using local profile:

$ s3cmd ls s3://local --no-check-certificate
2018-05-28 14:27       245   s3://local/test

List targeting aws directly (empty):

$ aws s3 --profile laure ls s3://zenkobucket
$

See cloudserver front logs:

{"name":"S3","currentTimestamp":1527517697254,"time":1527517697254,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:c1702ebc344b9af74334","level":"trace","message":"current timestamp","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","signatureFromRequest":"FTVIw7YCRMgOyYIWXCD+edI3zaY=","time":1527517697254,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:c1702ebc344b9af74334","level":"trace","message":"signature from request","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","time":1527517697255,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:c1702ebc344b9af74334","level":"trace","message":"constructing string to sign","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","stringToSign":"PUT\n964cdf567b3f54d1ea6f98f125696038\n\n\nx-amz-date:Mon, 28 May 2018 14:28:17 GMT\n/_/backbeat/multiplebackenddata/local/test","time":1527517697255,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:c1702ebc344b9af74334","level":"trace","message":"constructed string to sign","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","algo":"sha1","time":1527517697255,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:c1702ebc344b9af74334","level":"trace","message":"algo for calculating signature","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","time":1527517697255,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:c1702ebc344b9af74334","level":"debug","message":"authenticating V2 request","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","authInfo":{"message":{"body":{"userInfo":{"accountDisplayName":"CustomAccount","canonicalID":"97603964dbecdaadac8a5df443d0523e064cc20b616ec292580b6e4138ee7c0d/replication","arn":"arn:aws:iam::123456789012:root"}}}},"time":1527517697256,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:c1702ebc344b9af74334","level":"debug","message":"received info from Vault","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","implName":"multipleBackends","keyContext":{"bucketName":"local","owner":"d5351803ed7c5f73989dcb52115a24882af2f6700ab505a51089b525a6eb5f0c","namespace":"default","objectKey":"test","metaHeaders":{"x-amz-meta-scal-replication-status":"REPLICA","x-amz-meta-scal-version-id":"39383437323438323332373131353939393939395247303031202032332e30","x-amz-meta-s3cmd-attrs":"uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1526279617/atime:1527497055/md5:964cdf567b3f54d1ea6f98f125696038/ctime:1526279617"},"contentType":"text/plain","cipherBundle":null},"method":"put","time":1527517697256,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:c1702ebc344b9af74334","level":"debug","message":"sending put to datastore","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","error":{"code":503,"description":"The request has failed due to a temporary failure of the server.","ServiceUnavailable":true},"implName":"multipleBackends","time":1527517697344,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:c1702ebc344b9af74334","level":"error","message":"put error from datastore","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","error":{"code":503,"description":"The request has failed due to a temporary failure of the server.","ServiceUnavailable":true},"time":1527517697348,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:c1702ebc344b9af74334","level":"error","message":"error in datastore","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","error":{"code":503,"description":"The request has failed due to a temporary failure of the server.","ServiceUnavailable":true},"method":"putObject","time":1527517697348,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:c1702ebc344b9af74334","level":"error","message":"error putting data","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","bytesSent":199,"clientIP":"::ffff:172.17.0.7","clientPort":43276,"httpMethod":"PUT","httpURL":"/_/backbeat/multiplebackenddata/local/test?operation=putobject","httpCode":503,"time":1527517697350,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:c1702ebc344b9af74334","elapsed_ms":96.545449,"level":"info","message":"responded with error JSON","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","clientIP":"::ffff:172.17.0.7","clientPort":43276,"httpMethod":"GET","httpURL":"/local/test?partNumber=1&versionId=39383437323438323332373131353939393939395247303031202032332e30","time":1527517708443,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:580bfaafff4e97e46122","level":"info","message":"received request","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","bucketName":"local","objectKey":"test","bytesReceived":0,"bodyLength":0,"clientIP":"::ffff:172.17.0.7","clientPort":43276,"httpMethod":"GET","httpURL":"/local/test?partNumber=1&versionId=39383437323438323332373131353939393939395247303031202032332e30","contentLength":245,"httpCode":200,"time":1527517708456,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:580bfaafff4e97e46122","elapsed_ms":13.275751,"level":"info","message":"responded with streamed content","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","clientIP":"::ffff:172.17.0.7","clientPort":43274,"httpMethod":"PUT","httpURL":"/_/backbeat/multiplebackenddata/local/test?operation=putobject","time":1527517708458,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"info","message":"received request","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"MultipleBackendGateway","error":"The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.","errorName":"InvalidRequest","dataStoreName":"aws","backendType":"aws_s3","time":1527517708507,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2:ea32277da643aaf1b825","level":"error","message":"err from data backend","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"MultipleBackendGateway","controllingLocationConstraint":"aws","time":1527517708509,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2:5b8a4a826d45a1bce0e5","level":"debug","message":"put to location","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"MultipleBackendGateway","error":{"code":503,"description":"Error returned from AWS: The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256.","ServiceUnavailable":true},"dataStoreType":"aws_s3","time":1527517708509,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2:5b8a4a826d45a1bce0e5","level":"error","message":"error from datastore","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","clientIP":"::ffff:172.17.0.7","clientPort":43274,"httpMethod":"PUT","httpURL":"/_/backbeat/multiplebackenddata/local/test?operation=putobject","time":1527517708458,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"info","message":"received request","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","method":"routeBackbeat","time":1527517708459,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"debug","message":"routing request","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","method":"Arsenal.auth.server.extractParams","time":1527517708459,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"trace","message":"entered","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","version":"v2","authMethod":"headers","time":1527517708459,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"trace","message":"identified auth method","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","time":1527517708459,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"trace","message":"running header auth check","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","requestTimestamp":1527517708000,"time":1527517708459,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"trace","message":"request timestamp","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","currentTimestamp":1527517708459,"time":1527517708459,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"trace","message":"current timestamp","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","signatureFromRequest":"2fWZvdSMklc4xIWm6ju04wYbkhs=","time":1527517708459,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"trace","message":"signature from request","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","time":1527517708459,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"trace","message":"constructing string to sign","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","stringToSign":"PUT\n964cdf567b3f54d1ea6f98f125696038\n\n\nx-amz-date:Mon, 28 May 2018 14:28:28 GMT\n/_/backbeat/multiplebackenddata/local/test","time":1527517708459,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"trace","message":"constructed string to sign","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","algo":"sha1","time":1527517708459,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"trace","message":"algo for calculating signature","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","time":1527517708459,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"debug","message":"authenticating V2 request","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","authInfo":{"message":{"body":{"userInfo":{"accountDisplayName":"CustomAccount","canonicalID":"97603964dbecdaadac8a5df443d0523e064cc20b616ec292580b6e4138ee7c0d/replication","arn":"arn:aws:iam::123456789012:root"}}}},"time":1527517708459,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"debug","message":"received info from Vault","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","implName":"multipleBackends","keyContext":{"bucketName":"local","owner":"d5351803ed7c5f73989dcb52115a24882af2f6700ab505a51089b525a6eb5f0c","namespace":"default","objectKey":"test","metaHeaders":{"x-amz-meta-scal-replication-status":"REPLICA","x-amz-meta-scal-version-id":"39383437323438323332373131353939393939395247303031202032332e30","x-amz-meta-s3cmd-attrs":"uid:0/gname:root/uname:root/gid:0/mode:33188/mtime:1526279617/atime:1527497055/md5:964cdf567b3f54d1ea6f98f125696038/ctime:1526279617"},"contentType":"text/plain","cipherBundle":null},"method":"put","time":1527517708460,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"debug","message":"sending put to datastore","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","error":{"code":503,"description":"The request has failed due to a temporary failure of the server.","ServiceUnavailable":true},"implName":"multipleBackends","time":1527517708509,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"error","message":"put error from datastore","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","error":{"code":503,"description":"The request has failed due to a temporary failure of the server.","ServiceUnavailable":true},"time":1527517708511,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"error","message":"error in datastore","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","error":{"code":503,"description":"The request has failed due to a temporary failure of the server.","ServiceUnavailable":true},"method":"putObject","time":1527517708512,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","level":"error","message":"error putting data","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23}
{"name":"S3","accessKey":"URXDGO3FY11XGQ62OYZD","bytesSent":199,"clientIP":"::ffff:172.17.0.7","clientPort":43274,"httpMethod":"PUT","httpURL":"/_/backbeat/multiplebackenddata/local/test?operation=putobject","httpCode":503,"time":1527517708512,"req_id":"12d86b94054b8c4e6152:c30128ece3439585a2d2:7c0f215d4547aba5ecd2","elapsed_ms":53.728039,"level":"info","message":"responded with error JSON","hostname":"zenko-cloudserver-front-6bf9b45f8c-m7ztl","pid":23} 

[ssalaues]

@samuellugassy
Are you still receiving the error? From your logs it looks like you were able to successfully upload /bucket1/gemalto.jpg. Have you tried to ls s3://bucket1 to confirm it's there? Otherwise from the logs, it looks like it's functioning normally.

@LaureVergeron from what I could tell, it looks like you minikube VM had issue communicating with Orbit (the service replication account deals directly with Orbit). Also could you create a separate issue for this problem? It's starting to get confusing trying to keep track of both issues in one thread and will only get worse the longer the thread goes.