Comments (9)
Hmm, interesting!
Can you check if rclone does the same thing with incorrect credentials with AWS S3?
I suspect this will need to be fixed in the upstream library.
from rclone.
I don't have an AWS S3 available, but when I replace the "rclone serve s3" server with MinIO and send the same request with a wrong password, I see this response:
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
<Resource>/</Resource>
<RequestId>17C4FD2A69658918</RequestId>
<HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId>
</Error>
... and the rclone s3 client immediately fails with an error:
2024/04/10 20:01:56 ERROR : : error listing: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
status code: 403, request id: 17C4FD2A69658918, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
2024/04/10 20:01:56 Failed to lsf with 2 errors: last error was: error in ListJSON: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
status code: 403, request id: 17C4FD2A69658918, host id: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8
I think it's safe to say that "rclone serve s3" is creating an invalid xml response message.
The rclone client is fine, i think.
from rclone.
Please give this a try
v1.67.0-beta.7876.5c6c39510.fix-7749-s3-serve-error on branch fix-7749-s3-serve-error (uploaded in 15-30 mins)
from rclone.
Sorry, that didn't fix it :( The error is still there, exactly the same.
$ ./rclone --version
rclone v1.67.0-beta.7876.5c6c39510.fix-7749-s3-serve-error
- os/version: debian 12.1 (64 bit)
- os/kernel: 6.1.0-20-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.2
- go/linking: static
- go/tags: none
from rclone.
This fixes the XML for me
Before (the master branch)
2024/04/18 11:59:02 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/04/18 11:59:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/04/18 11:59:03 DEBUG : HTTP REQUEST (req 0xc000b88500)
2024/04/18 11:59:03 DEBUG : GET / HTTP/1.1
Host: localhost:8080
User-Agent: rclone/v1.67.0-beta.7896.2063e017d
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240418T105903Z
Accept-Encoding: gzip
2024/04/18 11:59:03 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/04/18 11:59:03 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/04/18 11:59:03 DEBUG : HTTP RESPONSE (req 0xc000b88500)
2024/04/18 11:59:03 DEBUG : HTTP/1.1 403 Forbidden
Content-Length: 236
Content-Type: text/xml; charset=utf-8
Date: Thu, 18 Apr 2024 10:59:03 GMT
<?xml version="1.0" encoding="UTF-8"?>
<errorResponse><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message></errorResponse>
After (with the fix above)
2024/04/18 12:00:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/04/18 12:00:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/04/18 12:00:20 DEBUG : HTTP REQUEST (req 0xc000d10100)
2024/04/18 12:00:20 DEBUG : GET / HTTP/1.1
Host: localhost:8080
User-Agent: rclone/v1.67.0-beta.7876.5c6c39510.fix-7749-s3-serve-error
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240418T110020Z
Accept-Encoding: gzip
2024/04/18 12:00:20 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/04/18 12:00:20 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/04/18 12:00:20 DEBUG : HTTP RESPONSE (req 0xc000d10100)
2024/04/18 12:00:20 DEBUG : HTTP/1.1 403 Forbidden
Content-Length: 236
Content-Type: text/xml; charset=utf-8
Date: Thu, 18 Apr 2024 11:00:20 GMT
<?xml version="1.0" encoding="UTF-8"?>
<ErrorResponse><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message></ErrorResponse>
I tested this with running this in one terminal
rclone serve -vv --auth-key user,pass s3 /tmp/src/
And this in another (note the incorrect secret_access_key
)
rclone lsf -vv ':s3,provider=Rclone,access_key_id=user,secret_access_key=passX,endpoint="http://localhost:8080":' --dump bodies
What problem are you having and how can I replicate it? Preferably with terminal commands like the above not with containers.
from rclone.
I'm gonna test it again. Maybe I did something wrong.
from rclone.
Sorry, but this really didn't fix the problem. The problem is that the error response has a wrong root xml element. This patch basically changed the wrong element name into another wrong element name.
- was (1.66):
<errorResponse>
- is now (1.67 patch):
<ErrorResponse>
- expected (S3 specification):
<Error>
from rclone.
Ah, sorry I mis-understood the problem.
Can you try this please
v1.67.0-beta.7876.2dc6016de.fix-7749-s3-serve-error on branch fix-7749-s3-serve-error (uploaded in 15-30 mins)
PS Incorrect root XML nodes are very common - I found a bug exactly like that in Google Cloud Storage's S3 implementation recently!
from rclone.
That looks better :)
- I can verify that the xml message in the response now has the correct root xml element.
- rclone (as a client) now fails instantly (before, it was retrying over and over again for about a minute)
2024/04/23 10:09:03 ERROR : : error listing: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method. status code: 403, request id: , host id: 2024/04/23 10:09:03 Failed to lsf with 2 errors: last error was: error in ListJSON: SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method. status code: 403, request id: , host id:
I would say this pretty much fixes the problem.
Thanks you for the patch!
The only thing I just noticed it that the "request id" and "host id" (in the log above) have no values. This is probably because the corresponding <RequestId>
and <HostId>
xml elements are missing in the response. I leave it to you though to decide if that is an issue or not ;)
from rclone.
Related Issues (20)
- rclone config does not use --password-command when encrypting configuration
- Implement chunked and parallel Upload in serve WebDAV .
- Public Link for Swift Backend HOT 1
- Explain dual id behavior when lsf google drive HOT 3
- Proton Drive: error 422 due to authenticating using an empty access token? HOT 11
- Suggestion: add a link to the nfsmount command in the documentation for the mount command HOT 2
- docker rclone volume for cloudflare r2 not working
- S3: if a bucket already exists, bucket creation silently fails HOT 5
- Life changing simple feature for headless Oauth signin HOT 1
- Allow passing config password as command parameter HOT 1
- webdav compressed stream not recognized
- `rclone serve` seems to always request a listing
- `sync` need `--download` flag to ensure `--checksum` get hash and avoid falling back to `--size-only` HOT 2
- rclone lsjson does not implement --include filter HOT 1
- DLNA Browse+BrowseMetadata not to spec, angers pedantic client. HOT 3
- Rclone sync behaving badly, forcibly clearing RAM cache HOT 7
- copy the file from onedrive to local it show the error `unauthenticated` HOT 1
- S3(Alibaba): bucket ACL changed after `rclone mkdir` HOT 4
- serve webdav: htpasswd+bcrypt cause poor performance when listing or writing dirs with many subdirs /files HOT 3
- backend: pcloud: support block level sync (upload only) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rclone.