Other 's3' plugin already use same buffer_path about fluent-plugin-s3

[repeatedly]

Is this a entire configuration? Could you paste the log of fluentd/td-agent?

[BDuelz]

/var/log/td-agent/td-agent.log

2015-10-13 19:33:17 +0000 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2015-10-13 19:33:17 +0000 [info]: starting fluentd-0.12.12
2015-10-13 19:33:17 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0'
2015-10-13 19:33:17 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2015-10-13 19:33:17 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.0.0'
2015-10-13 19:33:17 +0000 [info]: gem 'fluent-plugin-mongo' version '0.7.10'
2015-10-13 19:33:17 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1'
2015-10-13 19:33:17 +0000 [info]: gem 'fluent-plugin-s3' version '0.5.9'
2015-10-13 19:33:17 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2015-10-13 19:33:17 +0000 [info]: gem 'fluent-plugin-td' version '0.10.27'
2015-10-13 19:33:17 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.1'
2015-10-13 19:33:17 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2015-10-13 19:33:17 +0000 [info]: gem 'fluentd' version '0.12.12'
2015-10-13 19:33:18 +0000 [info]: adding match pattern="web__error" type="copy"
2015-10-13 19:33:18 +0000 [error]: config error file="/etc/td-agent/td-agent.conf" error="Other 's3' plugin already use same buffer_path: type = s3, buffer_path = /tmp/fluent-plugin-s3"
2015-10-13 19:33:18 +0000 [info]: process finished code=256
2015-10-13 19:33:18 +0000 [warn]: process died within 1 second. exit.
2015-10-13 19:33:35 +0000 [error]: dry run failed: Other 's3' plugin already use same buffer_path: type = s3, buffer_path = /tmp/fluent-plugin-s3

[BDuelz]

The error itself is raised in fluentd

[BDuelz]

Also, I want to say, if I don't use the copy output plugin, things work as expected:

<match web__error>
  type s3
  aws_key_id ACC_KEY
  aws_sec_key SEC_KEY
  s3_bucket log-bucket
  path web__error/
  buffer_path /tmp/fluent-plugin-s3
  s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
  time_slice_format %Y-%m-%d/%H
  flush_interval 15s
  utc
</match>

However, I need to be able to send logs to multiple locations.

[repeatedly]

I tested s3 and copy and it worked. In addition, many users use s3 and copy so this combination should not be a problem.

Above configuration is your entire configuration?
If you use same buffer_path configuration across <match>, this error happens. Error configuration example is below.

<source>
  type forward
</source>

<match s3.**>
    type s3

    aws_key_id "#{ENV['AWS_ACCESS_KEY_ID']}"
    aws_sec_key "#{ENV['AWS_SECRET_ACCESS_KEY']}"
    s3_bucket fluent-plugin-s3
    path test1/
    buffer_path /tmp/fluent-plugin-s3
    flush_at_shutdown true
    utc
</match>

<match s3.**>
    type s3

    aws_key_id "#{ENV['AWS_ACCESS_KEY_ID']}"
    aws_sec_key "#{ENV['AWS_SECRET_ACCESS_KEY']}"
    s3_bucket fluent-plugin-s3
    path test2/
    buffer_path /tmp/fluent-plugin-s3
    flush_at_shutdown true
    utc
</match>

[BDuelz]

You're totally right. I was using the same buffer_path in multiple config files. It works now.

Sorry to waste your time.

Cheers