Insert in mongos : operate_invalid_records about fluent-plugin-mongo

[repeatedly]

Hmm... shard key(exclude_broken_fields) seems broken.
The simple solution is ignoring the document when exclude_broken_fields are broken.

What do you think?

[repeatedly]

Another solution is using but maybe this approach needs the handling improvement of fluentd itself.

[Kaimlo]

Hum, what's the best for you? The goal is to not lose insert request into mongo.

[repeatedly]

fluent-plugin-mongo is designed on the assumption that an incoming event is valid as BSON.
It's a MongoDB limitation, so __broken_data approach is the last resort...

The best is secondary mongod or other output plugin is needed.
Because if shard key is broken, out_mongo can't send the request to mongos.
The problem is current Fluentd doesn't support the direct call.

Ignoring document is very simple solution.
In fact, some users use ignore_invalid_document true to ignore.
But it depends on the data type...

[Kaimlo]

Ok so I will just use ignore_invalid_document true for now.