How to set s3 region? about send

[boomam]

lol, gotta laugh - i solved it - regenerating the key to not have special characters. :-p
Thats super-wierd, as wrapping the string in quotes, or not, regardless didnt work, but having a fully alphanumeric string, worked fine.

[timvisee]

You should be able to set this using these variables:

send/server/config.js

Lines 27 to 41 in b9faece

	s3_bucket: {
	format: String,
	default: '',
	env: 'S3_BUCKET'
	},
	s3_endpoint: {
	format: String,
	default: '',
	env: 'S3_ENDPOINT'
	},
	s3_use_path_style_endpoint: {
	format: Boolean,
	default: false,
	env: 'S3_USE_PATH_STYLE_ENDPOINT'
	},

Does that help?

[deluxghost]

You should be able to set this using these variables:

send/server/config.js

Lines 27 to 41 in b9faece

	s3_bucket: {
	format: String,
	default: '',
	env: 'S3_BUCKET'
	},
	s3_endpoint: {
	format: String,
	default: '',
	env: 'S3_ENDPOINT'
	},
	s3_use_path_style_endpoint: {
	format: Boolean,
	default: false,
	env: 'S3_USE_PATH_STYLE_ENDPOINT'
	},

Does that help?

hmm, i dont see a region config, only endpoint, bucket and path style.

[timvisee]

I must admit that I'm not too familiar with S3. Doesn't setting a custom endpoint allow you to set the region as well (URL prefix)?

If not, I'll probably be able to take a look into this after the weekend.

[deluxghost]

I must admit that I'm not too familiar with S3. Doesn't setting a custom endpoint allow you to set the region as well (URL prefix)?

If not, I'll probably be able to take a look into this after the weekend.

Im not very familiar with that too, looks like there is no standard for s3 url yet

other apps like gitlab support s3 region config. If you are using a 3rdparty lib, it might be easy to implement

[ihatemyisp]

I'll be the first to admit my S3 knowledge is severely lacking but I use Send (docker compose) with Minio. The S3 Region I have set in Minio is not the one Send uses by default, but it works.

Do you have S3_USE_PATH_STYLE_ENDPOINT set to "true"?

I did have to change one section of the IAM policy that @timvisee has on GitLab to:

...
"Statement": [
        {
            "Action": [
                "s3:ListAllMyBuckets"
            ],
            "Resource": [
                "arn:aws:s3:::<s3_bucket_name>"
            ],
            "Effect": "Allow"
        },
...

[NamPNQ]

Use can set environment variable S3_USE_PATH_STYLE_ENDPOINT=true to fix your problems, this is not related region

[boomam]

Having a similar issue, the path style variable however does not solve the issue.
Are there any known workarounds?

[lilas-dev]

Having a similar issue, the path style variable however does not solve the issue. Are there any known workarounds?

I think you have make some mistake on your config, I just try a simple setup, its work like a charm

version: "3"

services:
  send:
    image: registry.gitlab.com/timvisee/send:latest
    ports:
      - '1234:1234'
    environment:
      - NODE_ENV=production
      - BASE_URL=http://localhost:1234
      - PORT=1234
      - REDIS_HOST=redis
      - AWS_ACCESS_KEY_ID=iloveyou
      - AWS_SECRET_ACCESS_KEY=andyoudontloveme
      - S3_BUCKET=send
      - S3_ENDPOINT=http://minio:9000
      - S3_USE_PATH_STYLE_ENDPOINT=true
    depends_on:
      minio:
        condition: service_healthy
  redis:
    image: redis
  minio:
    image: minio/minio
    command: ['server', '/data']
    environment:
      - MINIO_ROOT_USER=iloveyou
      - MINIO_ROOT_PASSWORD=andyoudontloveme
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
  mc:
    image: minio/mc
    depends_on:
      minio:
        condition: service_healthy
    entrypoint: >
      /bin/sh -c "
      /usr/bin/mc alias set myminio http://minio:9000 iloveyou andyoudontloveme;
      /usr/bin/mc mb myminio/send;
      "

[boomam]

Having a similar issue, the path style variable however does not solve the issue. Are there any known workarounds?

I think you have make some mistake on your config, I just try a simple setup, its work like a charm

Can you confirm, what region do you have your Minio set too?
To be clear, the issue is not connecting it to Minio, but connecting to Minio where the region is not the default - as there is no variable in 'send' right now to set that.

[lilas-dev]

@boomam I dont see Minio support region https://min.io/docs/minio/linux/reference/minio-server/minio-server.html#environment-variables

Can you post your config and the log of error? Its hard to understand what happened without this.

[boomam]

@boomam I dont see Minio support region https://min.io/docs/minio/linux/reference/minio-server/minio-server.html#environment-variables

Correct, but you still have to specify it on the connection.
It's the same with other apps that can use S3, like Pydio or NextCloud.

Can you post your config and the log of error? Its hard to understand what happened without this.

I'd have to loop back in a bit to generate the error again, I ended up giving up with Send and deployed something else.

Regardless, other than the URL the config for S3 is the same as yours - just not specifying port 9000 being the only difference, as I have all my traffic go through a load balancer on 443/tcp. So the endpoint instead reads as 'https://minio.domain.com'.

To be clear on that aspect, it works with other apps - where I can specify the region in use.
So its not a LB issue or a Minio config issue. ;-)

Your other config examples, such as how you are deploying Minio, are not valid for me as they are on separate systems.

[boomam]

Had to tidy it up, it dumps it all on one line for some reason in STOUT -

Timestamp=1685717940242000000 
Logger=FirefoxSend 
Type=send.upload.upload 
Severity=2 
Pid=1 
EnvVersion=2.0 
Fields={"error":"UnknownEndpoint: Inaccessible 
host: `' at port `443'. This service may not be available in the `us-east-1' region.","stack":"\n    
at Request.ENOTFOUND_ERROR (/app/node_modules/aws-sdk/lib/event_listeners.js:611:46)\n    
at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n    
at Request.emit (/app/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n    
at Request.emit (/app/node_modules/aws-sdk/lib/request.js:686:14)\n    
at error (/app/node_modules/aws-sdk/lib/event_listeners.js:443:22)\n    
at ClientRequest.<anonymous> (/app/node_modules/aws-sdk/lib/http/node.js:99:9)\n    
at ClientRequest.emit (node:events:390:28)\n    
at ClientRequest.emit (node:domain:475:12)\n    
at TLSSocket.socketErrorListener (node:_http_client:447:9)\n    at TLSSocket.emit (node:events:390:28)"}

Connectivity between container and service is confirmed, it can get to Minio/S3.
It's the region its tripping up on.

[lilas-dev]

@boomam From the logs, I think the problem maybe about your config of ` S3_ENDPOINT` Le ven. 2 juin 2023 à 17:03, boomam ***@***.***> a écrit :

…

Had to tidy it up, it dumps it all on one line for some reason in STOUT - Timestamp=1685717940242000000 Logger=FirefoxSend Type=send.upload.upload Severity=2 Pid=1 EnvVersion=2.0 Fields={"error":"UnknownEndpoint: Inaccessible host: `' at port `443'. This service may not be available in the `us-east-1' region.","stack":"\n at Request.ENOTFOUND_ERROR (/app/node_modules/aws-sdk/lib/event_listeners.js:611:46)\n at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n at Request.emit (/app/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n at Request.emit (/app/node_modules/aws-sdk/lib/request.js:686:14)\n at error (/app/node_modules/aws-sdk/lib/event_listeners.js:443:22)\n at ClientRequest.<anonymous> (/app/node_modules/aws-sdk/lib/http/node.js:99:9)\n at ClientRequest.emit (node:events:390:28)\n at ClientRequest.emit (node:domain:475:12)\n at TLSSocket.socketErrorListener (node:_http_client:447:9)\n at TLSSocket.emit (node:events:390:28)"} Connectivity between container and service is confirmed, it can get to Minio/S3. It's the region its tripping up on. — Reply to this email directly, view it on GitHub <#82 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQ34G6OJGQPMLMXMQVDTQZLXJH6DNANCNFSM5TZ3FNNQ> . You are receiving this because you commented.Message ID: ***@***.***>

[boomam]

@boomam From the logs, I think the problem maybe about your config of S3_ENDPOINT Le ven. 2 juin 2023 à 17:03, boomam @.***> a écrit :

Can you expand on this please, as it looks like your comment may have been cut off?
 
The S3_ENDPOINT config i am using, is in the format of -
- S3_ENDPOINT='https://minio.domain.com'
 

I have also tested other encapsulations, too. Along with specifying and not specifying 443.
The issue persists regardless of format.

[lilas-dev]

@boomam From the logs, I think the problem maybe about your config of S3_ENDPOINT Le ven. 2 juin 2023 à 17:03, boomam @.***> a écrit :

Can you expand on this please, as it looks like your comment may have been cut off?   The S3_ENDPOINT config i am using, is in the format of - - S3_ENDPOINT='https://minio.domain.com'  

I have also tested other encapsulations, too. Along with specifying and not specifying 443. The issue persists regardless of format.

Please try this

 S3_ENDPOINT=minio.domain.com

[boomam]

@boomam From the logs, I think the problem maybe about your config of S3_ENDPOINT Le ven. 2 juin 2023 à 17:03, boomam @.***> a écrit :

Can you expand on this please, as it looks like your comment may have been cut off?   The S3_ENDPOINT config i am using, is in the format of - - S3_ENDPOINT='https://minio.domain.com'  
I have also tested other encapsulations, too. Along with specifying and not specifying 443. The issue persists regardless of format.

Please try this

 S3_ENDPOINT=minio.domain.com

Already have, noted above. ;-)

To re-illiterate in case it was missed, the connection is fine, I can see the connection even in Minio's logs.
Its the region.

[boomam]

For thoroughness, re-tested regardless -
Two errors in the log, one is the same one from above, and the second is this -

Timestamp=1685735679756000000 
Logger=FirefoxSend 
Type=send.upload.upload 
Severity=2 
Pid=1 
EnvVersion=2.0 Fields={"error":"SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.","stack":"\n    
at Request.extractError (/app/node_modules/aws-sdk/lib/services/s3.js:711:35)\n    
at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n    
at Request.emit (/app/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n    
at Request.emit (/app/node_modules/aws-sdk/lib/request.js:686:14)\n    
at Request.transition (/app/node_modules/aws-sdk/lib/request.js:22:10)\n    
at AcceptorStateMachine.runTo (/app/node_modules/aws-sdk/lib/state_machine.js:14:12)\n    
at /app/node_modules/aws-sdk/lib/state_machine.js:26:10\n    
at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:38:9)\n    
at Request.<anonymous> (/app/node_modules/aws-sdk/lib/request.js:688:12)\n    
at Request.callListeners (/app/node_modules/aws-sdk/lib/sequential_executor.js:116:18)"}

[lilas-dev]

Try check your config of creds, its should not wrap the string in quote

[boomam]

Try check your config of creds, its should not wrap the string in quote
To confirm latest tests, variables -

S3_ENDPOINT=minio.domain.com
AWS_SECRET_ACCESS_KEY=1234abc

Same issue.