s3_object_key_format without index overwrites (not append) log files about fluent-plugin-s3

[daipom]

I have moved the issue here because it is about out_s3.

[daipom]

out_s3 plugin does not support append feature.
So, we need to upload files without duplicating file names.

I would like to have single log file on MinIO per day.

You can make out_s3 to upload files once a day.
(if you can tolerate very slow upload frequency...)

[arepusb]

You can make out_s3 to upload files once a day.
(if you can tolerate very slow upload frequency...)

Fluentd is already configured to upload logs to MinIO once per day. I attached config in the description.
Here is part of the config

timekey 1440m  
timekey_wait 10m

Anyway I get more than one file per day often. Their time difference is no more than two minutes (much less than 10m).

[daipom]

Hmm, tag key or chunk_limit_size can be the cause.

<buffer tag,time>

chunk_limit_size 256m

[arepusb]

Size of log files ~ 200-300 bytes. It's much less than 256m.

[daipom]

Then, please try removing tag key

- <buffer tag,time>
+ <buffer time>

[arepusb]

Then, please try removing tag key

- <buffer tag,time>
+ <buffer time>

To be honest I tested both variants before creation this request. Thank you for trying to help!

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 7 days

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 7 days

[daipom]

@arepusb
Sorry for the interval.
Did you find out anything?