fangli / fluent-plugin-influxdb Goto Github PK
View Code? Open in Web Editor NEWA buffered output plugin for fluentd and InfluxDB
License: MIT License
A buffered output plugin for fluentd and InfluxDB
License: MIT License
This is just a suggestion for discussion/consideration.
Currently the plugin increments sequence_tag
when multiple entries with identical timestamp occur, and resets it to zero when an event with different timestamp is seen. There are two problems with this.
You could end up generating a lot of time series. Suppose, for example, that in one particular second you have an unusual burst of 1000 messages. You'll end up creating 1000 time series, and in future all queries will have to be across all those 1000 series, even though mostly they will be empty.
If the records are being written with timestamps that are not increasingly linearly - e.g. there is some "backfill" going on - then there is a high risk of overwriting previous data, because the sequence tag is reset to zero every time the source timestamp changes. This is mainly a problem when the source timestamp has only 1-second resolution.
I would like to suggest another approach, which is to make use of the nanosecond timestamp precision of influxdb.
There are several approaches to this, but I propose the following one:
This means that the stored timestamps are at worst in error by one millisecond. The chances of conflict are extremely low. If your input records have 1-second resolution then either you would have to have one million events in a single second, or you would have to be backfilling to a previous second and be unlucky enough to hit the same range of timestamps.
Even if your input records have millisecond resolution, as long as they arrive with monotonically increasing timestamps there should not be a problem, although some reordering is possible for records in adjacent milliseconds when the counter wraps. Maybe when the time precision is millisecond or better, the counter should only run between 0 and 999, so the error is no worse than one microsecond. (This upper limit can be made configurable anyway)
I did consider some other options - e.g. generating a random offset between 0 and 999999 for each record, or using a hash of the record. The former has a higher probability of collision (around 50% probability of at least one collision when 1000 records are stored in a single second). The latter will not record repeated records at all, which is undesirable for logs where you may actually want to count multiple instances of the same event.
If time_precision
rounding is required then do it in the plugin and convert back to nanoseconds; but I don't really see why anyone would want to reduce the resolution of their existing timestamps. ↩
I've been trying to troubleshoot an issue where writes into influx seem to come in very slowly over time. My setup is as follows:
influxdb 0.8.8
fluent-plugin-influxdb (0.1.4)
fluentd (0.12.8)
fluentd config:
<match fuse.**>
type influxdb
host ***
port 8086
dbname ***
user ***
password ***
time_precision ms
buffer_type memory
buffer_chunk_limit 524288 # 512 * 1024
buffer_queue_limit 1024
flush_interval 30
retry_limit 17
retry_wait 1.0
I've played around and tried a number of different combinations of the config.
I have 2 fluentd instances dumping into a single influxdb instance. All machines are barely registering any CPU usage when messages are being sent, but it takes 20-30 minutes for the buffers to clear out into influx after all the data has been written to fluentd. This is around 72000 messages total (each message is ~100 bytes) being sent.
Turning debug logging on in influx I see a ton of spam about authenticating the user specified in the fluentd config, 10-20 a second. Any thoughts on what the issue might be?
My temporary workaround is to
gem install influxdb --version 0.2.3
before
fluent-gem install fluent-plugin-influxdb
That may not be the exact version where the issue was introduced, but it works.
Here's a log from a start->finish run with the failure:
2016-05-18 18:59:56 +0000 [info]: reading config file path="/etc/fluentd/fluentd.conf"
2016-05-18 18:59:56 +0000 [info]: starting fluentd-0.12.23
2016-05-18 18:59:56 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.5.0'
2016-05-18 18:59:56 +0000 [info]: gem 'fluent-plugin-exclude-filter' version '0.0.1'
2016-05-18 18:59:56 +0000 [info]: gem 'fluent-plugin-flatten-hash' version '0.2.0'
2016-05-18 18:59:56 +0000 [info]: gem 'fluent-plugin-grep' version '0.3.4'
2016-05-18 18:59:56 +0000 [info]: gem 'fluent-plugin-influxdb' version '0.2.7'
2016-05-18 18:59:56 +0000 [info]: gem 'fluentd' version '0.12.23'
/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- cause (LoadError)
from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
from /var/lib/gems/1.9.1/gems/influxdb-0.3.0/lib/influxdb/client.rb:2:in `<top (required)>'
from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
from /var/lib/gems/1.9.1/gems/influxdb-0.3.0/lib/influxdb.rb:19:in `<top (required)>'
from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:60:in `require'
from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:60:in `rescue in require'
from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:35:in `require'
from /var/lib/gems/1.9.1/gems/fluent-plugin-influxdb-0.2.7/lib/fluent/plugin/out_influxdb.rb:3:in `<top (required)>'
from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/plugin.rb:172:in `block in try_load_plugin'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/plugin.rb:170:in `each'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/plugin.rb:170:in `try_load_plugin'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/plugin.rb:130:in `new_impl'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/plugin.rb:59:in `new_output'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/plugin/out_copy.rb:46:in `block in configure'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/plugin/out_copy.rb:37:in `each'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/plugin/out_copy.rb:37:in `configure'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/agent.rb:132:in `add_match'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/agent.rb:63:in `block in configure'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in `each'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/agent.rb:57:in `configure'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/root_agent.rb:86:in `configure'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/engine.rb:129:in `configure'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/engine.rb:103:in `run_configure'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:483:in `run_configure'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:154:in `block in start'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:360:in `call'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:360:in `main_process'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:333:in `block in supervise'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:332:in `fork'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:332:in `supervise'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/supervisor.rb:150:in `start'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
from /usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
from /var/lib/gems/1.9.1/gems/fluentd-0.12.23/bin/fluentd:5:in `<top (required)>'
from /usr/local/bin/fluentd:23:in `load'
from /usr/local/bin/fluentd:23:in `<main>'
2016-05-18 18:59:56 +0000 [info]: process finished code=256
2016-05-18 18:59:56 +0000 [warn]: process died within 1 second. exit.
I don't know if I do wrong something.
I want to put numeric data on influxdb with fluentd, but all data will save as a string type.
My fluentd conf file is like this:
source:
<source>
@type http
port 12102
format tsv
keys string1.string2,number1,number2
delimiter |
</source>
Match:
<match test>
@type influxdb
dbname test
flush_interval 10s
host localhost
port 8086
</match>
Mark.
Will be done when I back to office.
It looks like this plugin isn't 0.9 compatible. I'm trying to use it against influxdb 0.9 and getting:
2015-06-04 15:07:04 -0700 [warn]: fluent/output.rb:354:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2015-06-04 15:07:05 -0700 error_class="InfluxDB::Error" error="404 page not found\n" plugin_id="object:3fd80a06ce04"
This looks a lot like this: influxdata/influxdb#2032
Assuming the problem is indeed 0.9 compatibility, it would be great to have a 0.9 compatible branch of this repository.
i use fluent-plugin-influxdb:1.0.0.rc3
<filter container.*>
@type record_transformer
remove_keys Name
enable_ruby true
time ${Time.parse(record["time"]).to_i}
test_value testdb
<match container.*>
@type influxdb
host influxdb
port 8086
dbname ${test_value}
user writer
password YWM0ZGY2M2Y4NjY0
use_ssl false
auto_tags true
buffer_chunk_limit 8m
buffer_queue_limit 32
buffer_queue_full_action drop_oldest_chunk
flush_interval 1s
num_threads 4
#0 Connecting to database: ${test_value}, host: influxdb, port: 8086, username: writer, use_ssl = false, verify_ssl = true
Hi,
if I use tag_keys ["vhost_Uid"]
, series are created like something.metrics,vhost_Uid=123
, but selecting from something.metrics
, that vhost_Uid
field is null
. Is it really need to be skipped in values
?
Donatas.
The gemspec for 0.1.8 specifies the dependency of influxdb as ~>0.1.8 which causes it to install 0.2.0 which is not compatible with Influx 0.8. I believe it should be changed to either just 0.1.8 or ~<0.1.8.
We are running influxdb plugin with config:
<match metrics.*>
@type influxdb
host 10.47.246.7
port 8086
dbname test
use_ssl false
time_key time
auto_tags
time_precision ms
retry 3
@type memory
# chunk_limit_size 524288 # 512 * 1024
# chunk_limit_records 1024
flush_interval 10
retry_limit 3
retry_wait 1.0
num_threads 1
json
when we stop influxdb we see that fluentd is leaking memory:
any ideas?
for example
A numeric No like '123456' don't want to mark as tag
Now, insert an integer to a float field will throw an "already exists as type float" error
because there isn't precision float in Javascript, e.x: 1.0, 2.0... 10.0 all will be converted to integer automatically.
And as described in the 3032, now all numbers in influxdb are float, so could this be fixed?
Or could add an auto parse func from String?
Cuz now, the influxdb supports boolean is still very bad,
but most metrics from fluent-bit use boolean value.
Still lacks transform func without using kapacitor
Dear Friends
I want to use your plugin but I get error.
ERROR: Error installing fluent-plugin-influxdb:
influxdb requires Ruby version >= 2.2.0.
I uninstalled my ruby and installed different versions from 2.0.0, 2.2.0, 2.3, 2.4
but the error still exist. would you please help me how to solve this.
Thank you.
It would be useful if this plugin could act as a Fluent formatter plugin for Influx Line Protocol. It would extend its usefulness to do things like putting ILP onto a Kafka topic.
I don't know hot to write a "defalut_retention_policy" or "About default_retention_policy and ".
Can U give an example,
such as keep date for 30 daya , how can I write for "defalut_retention_policy"
Considering that sending an InfluxDB line with empty tag values such as (notice the empty ident tag):
syslog,host=fwtpcore1b,ident= message=\"BLAHBLAHBLAH\" 1449233580
will result in InfluxDB returning a missing tag value error, I would say that the InfluxDB Ruby client should omit tags with empty values in the line altogether. InfluxCommunity/influxdb-ruby#124 has been opened for this purpose.
That being said, ideally the Fluentd InfluxDB plugin should also prevent sending any empty tag to the Ruby client in the first place in order to catch the error as early as possible.
I have come upon this issue in cases where Syslog message were received with no ident field while having Fluentd output to InfluxDB.
I would like to be able to use the fluentd tags for configuring this plugin, using the tag_parts mixin. For example, with this config extract, the database name is determine by the second part of the tag. When I try this, the dbname is kept as "${tag_parts[1]}"
<match influxdb.**>
@type influxdb
dbname ${tag_parts[1]}
host influxdb
</match>
From what google tells me, this mixin would be needed: https://github.com/y-ken/fluent-mixin-rewrite-tag-name.
Do you think this would be possible?
My timestamp is 2015-12-13T07:39:31Z, but when insert to influxdb has only one message in one second. In fact, one second has more than one message.
fluent-plugin-influxdb (0.2.2)
influxdb (0.2.3)
influxdb server : 0.9.6.1
By the way: the config "tag_keys" isn't efficient.
I found an issue where the fluent-plugin-influxdb does not handle fatal influxdb errors well. I was switching from calling influxdb direct and using fluent so we had the time field set, but this plugin tried to set it too. This caused influxdb to throw an error:
2015-06-05 01:21:38 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-06-05 01:21:38 +0000 error_class="InfluxDB::Error" error="Cannot have duplicate field names" plugin_id="object:3fcef1e5c70c"
2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/influxdb-0.1.8/lib/influxdb/client.rb:234:in `block in post'
2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/influxdb-0.1.8/lib/influxdb/client.rb:264:in `call'
2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/influxdb-0.1.8/lib/influxdb/client.rb:264:in `connect_with_retry'
2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/influxdb-0.1.8/lib/influxdb/client.rb:227:in `post'
2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/influxdb-0.1.8/lib/influxdb/client.rb:170:in `_write'
2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/influxdb-0.1.8/lib/influxdb/client.rb:163:in `write_point'
2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-influxdb-0.1.4/lib/fluent/plugin/out_influxdb.rb:52:in `block in write'
2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_memory.rb:61:in `feed_each'
2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_memory.rb:61:in `msgpack_each'
2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-influxdb-0.1.4/lib/fluent/plugin/out_influxdb.rb:49:in `write'
2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/buffer.rb:325:in `write_chunk'
2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/buffer.rb:304:in `pop'
2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:321:in `try_flush'
2015-06-05 01:21:38 +0000 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:140:in `run'
2015-06-05 01:21:39 +0000 [warn]: temporarily failed to flush the buffer. next_retry=2015-06-05 01:21:40 +0000 error_class="InfluxDB::Error" error="Cannot have duplicate field names" plugin_id="object:3fcef1e5c70c"
When this happens, the plugin gets into a retry loop state that prohibits any new messages from being relayed correctly. Instead of trying to resend the buffer, these types of messages should be logged and discarded. This will allow valid messages to continue being logged.
I had to make the following change, otherwise the thread will forever attempt to connect to the database. when it is not online.
def shutdown
@influxdb.stop!
end
config:
<match .influxdb.test.health>
@type influxdb
measurement: health
host 127.0.0.1
port 40002
dbname test
auto_tags true
tag_keys: ["key1", "key2"]
time_key time
time_precision ms
plugin will auto convert js "new Date().getTime()" integer value to a float, I guess it's an issue for ruby integer scope, I convert "new Date().getTime()" to a string as a work around.
comments1:
I think all timestamp must be an integer, so could you add an auto integer convert for timestamp.
Hi all i'm receiving a lot of error for 2 type of tail:
The first:
2016-03-02 19:32:53 +0100 [warn]: temporarily failed to flush the buffer. next_retry=2016-03-02 19:32:54 +0100 error_class="InfluxDB::Error" error="{\"error\":\"partial write:\\nunable to parse 'nginx.access.log,daemon=nginx:,server=10.0.1.164,host=188.95.76.247,date=02/Mar/2016:18:32:40\\\\ +0000,httprequest=GET,request=/voucher/view/123621?pdf=1,protocol=HTTP/1.1,postrequest=-,statuscode=200,pagesize=49480,referrer=-,useragent=Amazon\\\\ CloudFront,renderizepage=0.814\\\\ 0.814\\\\ . htacces1=\\\"-\\\",htacces2=\\\"-\\\" 1456943562': invalid tag format\\nunable to parse 'nginx.access.log,daemon=nginx:,server=10.0.4.140,host=66.249.66.105,date=02/Mar/2016:18:32:41\\\\ +0000,httprequest=GET,request=/it/roma/tour-di-castelli-e-palazzi-c/?_escaped_fragment_=,protocol=HTTP/1.1,postrequest=-,statuscode=200,pagesize=16352,referrer=-,useragent=Amazon\\\\ CloudFront,renderizepage=0.660\\\\ 0.626\\\\ . htacces1=\\\"-\\\",htacces2=\\\"-\\\" 1456943563': invalid tag format\\nunable to parse 'nginx.access.log,daemon=nginx:,server=10.0.4.12,host=79.109.5.14,date=02/Mar/2016:18:32:51\\\\ +0000,httprequest=GET,request=/frontendV3/PROD/fonts/glyphicons-halflings-regular.woff,protocol=HTTP/1.1,postrequest=-,statuscode=200,pagesize=23292,referrer=https://www.musement.com/es/granada/alhambra-v/?gclid=CNC3yrzRossCFfMV0wod9XUIWw,useragent=Amazon\\\\ CloudFront,renderizepage=0.000\\\\ -\\\\ . htacces1=\\\"-\\\",htacces2=\\\"-\\\" 1456943573': invalid tag format\"}\n" plugin_id="object:3fbe5c5716c4"
the second:
2016-03-02 19:32:32 +0100 [warn]: failed to flush the buffer. error_class="InfluxDB::Error" error="{\"error\":\"unable to parse 'track.access.log,host=47.60.45.56,date=2016-03-02\\\\ 19:32:24\\\\ +0100\\\\ ,httprequest=GET,request=track.gif,env=prod\\u0026,url=https%253A%252F%252Fm.musement.com%252Fes%252Fgranada%252Falhambra-entradas-sin-colas-y-visita-guiada-por-la-manana-4072%252Fbooking,pagetype=frontend_mobile_event_booking,referrer=https%253A%252F%252Fm.musement.com%252Fes%252Fgranada%252Falhambra-entradas-sin-colas-y-visita-guiada-por-la-manana-4072%252F,locale=es,hitid=cb0bd867d219e6abd0796f0a6ba69e49-1456943543225,maid=,currency=EUR,session=ad00bc3mlg7a0vvqbh1jcp4tk1,countryId=161,cityId=174,venueId=,eventsId=4072,cart=,transactionId=,transactionTotal=,transactionMrgn=,transactionProducts=,customer=,eventName=event,eventValue=calendar-date,eventAttribute=open,httpprotocol=HTTP/1.1,httpcode=200,size=1443,cityname=Valencia,regionname=Comunidad\\\\ Valenciana,countryname=ES,referer=https://m.musement.com/es/granada/alhambra-entradas-sin-colas-y-visita-guiada-por-la-manana-4072/booking\\\\\\\",useragent=Mozilla/5.0\\\\ (Linux;\\\\ Android\\\\ 5.1.1;\\\\ ALE-L21\\\\ Build/HuaweiALE-L21)\\\\ AppleWebKit/537.36\\\\ (KHTML\\\\,\\\\ like\\\\ Gecko)\\\\ Chrome/48.0.2564.95\\\\ Mobile\\\\ Safari/537.36\\\\\\\" webid=\\\"cb0bd867d219e6abd0796f0a6ba69e49\\\",useragentFALSE=\\\"Mozilla/5.0%20(Linux;%20Android%205.1.1;%20ALE-L21%20Build/HuaweiALE-L21)%20AppleWebKit/537.36%20(KHTML\\\\,%20like%20Gecko)%20Chrome/48.0.2564.95%20Mobile%20Safari/537.36\\\" 1456943544': missing tag value\\nunable to parse 'track.access.log,host=37.14.34.97,date=2016-03-02\\\\ 19:32:25\\\\ +0100\\\\ ,httprequest=GET,request=track.gif,env=prod\\u0026,url=https%253A%252F%252Fwww.musement.com%252Fes%252Froma%252Fmuseos-vaticanos-v%252F%253Fgclid%253DCj0KEQiAu9q2BRDq3MDbvOL1yaYBEiQAD6qoBsu2fRcJDEwRwsNK7qQtYd6tJXSn1PxHDrKayAxLIEgaAh6z8P8HAQ%2526gclsrc%253Daw.ds,pagetype=frontend_venue,referrer=https%253A%252F%252Fwww.google.es%252F,locale=es,hitid=f154c32f575f635202e1d53351eb4fa1-1456943495070,maid=,currency=EUR,session=,countryId=82,cityId=2,venueId=164,eventsId=,cart=,transactionId=,transactionTotal=,transactionMrgn=,transactionProducts=,customer=,eventName=venue,eventValue=calendar-date,eventAttribute=open,httpprotocol=HTTP/1.1,httpcode=200,size=6474,cityname=Mislata,regionname=Comunidad\\\\ Valenciana,countryname=ES, [...]
this is my config for the first error
<source>
type tail
path /log/admin/access.log
pos_file /log/td-agent/admin.access.log.pos
tag nginx.access.log
format /^(?<daemon>[^ ]*)\s*(?<server>[^ ]*)\s*(?<host>[^ ]*)\s*(?<htacces1>[^ ]*)\s*(?<htacces2>[^ ]*)\s*\[(?<date>[^ ]*\s\D[0-9]*)\]\s*\"(?<httprequest>[^ ]*)\s*(?<request>[^ ]*)\s*(?<protocol>[^ ]*)\"\s*\"(?<postrequest>.*?)\"\s*(?<statuscode>[^ ]*)\s*(?<pagesize>[^ ]*)\s*\"(?<referrer>[^ ]*)\"\s*\"(?<useragent>.*?)"\s*(?<renderizepage>.*)$/
time_format %d/%b/%Y:%H:%M:%S %z
</source>
<match nginx.access.log.**>
type influxdb
host pippo
port 8086
dbname access_log
user pippo
password pippo
use_ssl false
time_precision s
tag_keys ["daemon", "server", "host", "htaccess1", "htaccess2", "date", "httprequest", "request", "protocol", "postrequest", "statuscode", "pagesize","referrer", "useragent", "renderizepage"]
sequence_tag _seq
flush_interval 10
retry_limit 3
</match>
and this for the second:
<source>
type tail
path /log/track/access.log
pos_file /log/td-agent/grafanatrack.access_log.pos
tag track.access.log
format /^(?<host>[^ ]*)[\s^-]*\s(?<date>[^ ]*\s[0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2}\s*[\W0-9]{1,5}[^"]*)[\s^\W]*(?<httprequest>[^ ]*)\s*\W(?<request>[a-z]+\.[a-z]+)\?*env=(?<env>.*?)&;&url=(?<url>.*?)&;&page_type=(?<pagetype>.*?)&;&referrer=(?<referrer>.*?)&;&locale=(?<locale>.*?)&;&id=(?<webid>.*?)&;&hitid=(?<hitid>.*?)&;&maid=(?<maid>.*?)&;&agent=(?<useragentFALSE>.*?)&;¤cy=(?<currency>.*?)&;&session=(?<session>.*?)&;&countryId=(?<countryId>.*?)&;&cityId=(?<cityId>.*?)&;&venueId=(?<venueId>.*?)&;&eventsId=(?<eventsId>.*?)&;&cart=(?<cart>.*?)&;&transactionId=(?<transactionId>.*?)&;&transactionTotal=(?<transactionTotal>.*?)&;&transactionMrgn=(?<transactionMrgn>.*?)&;&transactionProducts=(?<transactionProducts>.*?)&;&customer=(?<customer>.*?)&;&eventName=(?<eventName>.*?)&;&eventValue=(?<eventValue>.*?)&;&eventAttribute=(?<eventAttribute>[^ ]*)\s*(?<httpprotocol>[^" ]*)[\s\W]*(?<httpcode>[^ ]*)\s*(?<size>[^" ]*)[\s"]*(?<cityname>.*?)"\s"(?<regionname>.*?)"\s"(?<countryname>.*?)"\s"(?<referer>[^ ]*)\s*[\s^\W]*(?<useragent>.*)$/
time_format %Y-%m-%d %H:%M:%S %z
</source>
<match track.access.log.**>
type influxdb
host pippo
port 8086
dbname tracking
user pippo
password pippo
use_ssl false
time_precision s
tag_keys ["host", "date", "httprequest", "request", "env", "url", "pagetype", "referrer", "locale", "web_id", "hitid", "maid", "currency", "session", "countryId", "cityId", "venueId", "eventsId", "cart", "transactionId", "transactionTotal", "transactionMrgn", "transactionProducts", "customer", "eventName", "eventValue", "eventAttribute", "httpprotocol", "httpcode", "size", "cityname", "regionname", "countryname", "referer", "useragent"]
sequence_tag _seq
flush_interval 10
retry_limit 0
</match>
This is the log received on the first source:
nginx: 10.0.4.140 88.181.150.136 - - [02/Mar/2016:18:41:19 +0000] "GET /favicon.ico?random_1456909201 HTTP/1.1" "-" 200 5430 "https://www.musement.com/fr/barcelone/la-sagrada-familia-v/" "Amazon CloudFront" 0.000 - .
and this is the log received in the second:
62.19.79.55 - - 2016-03-02 19:42:11 +0100 "GET /track.gif?env=prod&&;&url=https%253A%252F%252Fwww.musement.com%252Fit%252Fvienna%252Fbiglietto-combinato-sissi-il-castello-di-schonbrunn-hofburg-e-il-museo-del-mobile-imperiale-2329%252F&;&page_type=frontend_event&;&referrer=https%253A%252F%252Fwww.google.it%252F&;&locale=it&;&id=5ed06fe6feeea5da65ce08f24335f2d7&;&hitid=5ed06fe6feeea5da65ce08f24335f2d7-1456944126582&;&maid=&;&agent=Mozilla/5.0%20(Windows%20NT%206.3;%20WOW64)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/48.0.2564.116%20Safari/537.36&;¤cy=EUR&;&session=&;&countryId=10&;&cityId=82&;&venueId=&;&eventsId=2329&;&cart=&;&transactionId=&;&transactionTotal=&;&transactionMrgn=&;&transactionProducts=&;&customer=&;&eventName=&;&eventValue=&;&eventAttribute= HTTP/1.1" 200 6469 "-" "-" "IT" "https://www.musement.com/it/vienna/biglietto-combinato-sissi-il-castello-di-schonbrunn-hofburg-e-il-museo-del-mobile-imperiale-2329/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36"
why?
hi,
i'm trying to ingest telemetry data from my juniper router into my influx db. i have a few fieldKeys that are of type string. i want to convert these fieldKeys to influx tags by using the auto_tag option, but this is not working.
if i use the tag_keys option then i have the same key saved as a field in influx and as a tag.
example this is what the field keys looks like in my influx db when i ingest the data:
> show field keys from interfaces
name: interfaces
fieldKey fieldType
-------- ---------
device string
egress_queue integer
interface string
interface_parent string
type string
value integer
if i use the tag_key option in my fluent.conf file as listed below, i then end up with a key being a fieldKey and tagKey in my influx db. this gives me problem when i use grafana. i hope i stated that clearly.
i have the following fluent.conf file configured:
#############
## INPUT ##
#############
<source>
@type udp
tag jnpr.jvision
#tag interfaces
format juniper_jti
output_format structured
port 50051
bind 0.0.0.0
</source>
##############
## OUTPUT ##
##############
<match jnpr.**>
#<match interfaces>
@type copy
<store>
type influxdb
host localhost
port 8086
dbname testfluent
measurement interfaces
#user juniper
#password juniper
#value_keys ["value"]
#field_keys ["value"]
#tag_keys ["interface_parent","device","type"]
auto_tags enable
buffer_type memory
# buffer_chunk_limit 524288 # 512 * 1024
# buffer_queue_limit 1024
flush_interval 2
# retry_limit 17
# retry_wait 1.0
# num_threads 1
</store>
</match>
## match tag=debug.** and dump to console
<match jnpr.**>
#<match interfaces>
@type stdout
@id stdout_output
</match>
is there a way to convert a fieldKey to a tagKey and have influx db store it as such without the same key being a field and a tag in influx? thanks!
I wanted to reach out and figure out if you would accept a PR to support the latest InfluxDB 2.0 apis?
We have a plugin ready to go, I think it would just need to be pr'd and reviewed here: https://github.com/influxdata/influxdb-plugin-fluent/blob/master/lib/fluent/plugin/out_influxdb2.rb
I'd love to figure out how to combine these two plugins.
Eventhough fluent/fluent-plugin-mongo#79 was fixed, the error still occurs in OUTPUT influxdb plugin.
2016-08-18 18:49:51 +0000 [warn]: /usr/lib/ruby/gems/2.3.0/gems/fluent-plugin-influxdb-0.2.8/lib/fluent/plugin/out_influxdb.rb:95:in `to_msgpack'
When trying to run the fluent-plugin in 1.0.0-rc1 i get the following error:
Error log:
2017-02-04 21:12:14 +0000 [warn]: emit transaction failed: error_class=NoMethodError error="undefined method `nsec' for 1486242734:Fixnum\nDid you mean? inspect" tag="metric.docker.upbeat_allen"
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluent-plugin-influxdb-1.0.0.rc1/lib/fluent/plugin/out_influxdb.rb:217:in `precision_time'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluent-plugin-influxdb-1.0.0.rc1/lib/fluent/plugin/out_influxdb.rb:108:in `format'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/output.rb:876:in `block in handle_stream_simple'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event.rb:193:in `block in each'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event.rb:192:in `each'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event.rb:192:in `each'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/output.rb:875:in `handle_stream_simple'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/output.rb:770:in `execute_chunking'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/output.rb:707:in `emit_buffered'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event_router.rb:153:in `emit_events'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event_router.rb:90:in `emit_stream'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event_router.rb:81:in `emit'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluent-plugin-grep-0.3.4/lib/fluent/plugin/out_grep.rb:77:in `block (2 levels) in emit'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluent-plugin-grep-0.3.4/lib/fluent/plugin/out_grep.rb:70:in `catch'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluent-plugin-grep-0.3.4/lib/fluent/plugin/out_grep.rb:70:in `block in emit'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event.rb:107:in `each'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluent-plugin-grep-0.3.4/lib/fluent/plugin/out_grep.rb:69:in `emit'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/compat/output.rb:162:in `process'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/output.rb:696:in `emit_sync'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event_router.rb:90:in `emit_stream'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/event_router.rb:81:in `emit'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/in_forward.rb:377:in `on_message'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/in_forward.rb:231:in `block in handle_connection'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/in_forward.rb:272:in `block (3 levels) in read_messages'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/in_forward.rb:271:in `feed_each'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/in_forward.rb:271:in `block (2 levels) in read_messages'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/in_forward.rb:280:in `block in read_messages'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/in_forward.rb:560:in `on_read'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/cool.io-1.4.6/lib/cool.io/io.rb:123:in `on_readable'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/cool.io-1.4.6/lib/cool.io/io.rb:186:in `on_readable'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run_once'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run'
2017-02-04 21:12:14 +0000 [warn]: /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluentd-0.14.8/lib/fluent/plugin/in_forward.rb:190:in `run'
2017-02-04 21:12:14 +0000 [warn]: out_grep: NoMethodError undefined method `nsec' for 1486242734:Fixnum
Did you mean? inspect /opt/rh/rh-ruby23/root/usr/local/share/gems/gems/fluent-plugin-influxdb-1.0.0.rc1/lib/fluent/plugin/out_influxdb.rb:217:in `precision_time'
Fluentd Dockerfile
FROM centos:7
MAINTAINER Jimmi Dyson <[email protected]>
ENTRYPOINT ["/start-fluentd"]
ENV LD_LIBRARY_PATH /opt/rh/rh-ruby23/root/usr/lib64
ENV FLUENTD_VERSION 0.14.8
ENV ELASTICSEARCH_HOST es-logging.default.svc
RUN yum update -y && \
yum install -y centos-release-scl-rh && \
yum install -y scl-utils make gcc bzip2 rh-ruby23 rh-ruby23-ruby-devel && \
yum clean all
RUN scl enable rh-ruby23 'gem update --system --no-document' && \
scl enable rh-ruby23 'gem install --no-document json_pure jemalloc' && \
scl enable rh-ruby23 "gem install --no-document fluentd -v ${FLUENTD_VERSION}" && \
ln -s /opt/rh/rh-ruby23/root/usr/local/bin/* /usr/bin
ENV ELASTICSEARCH_HOST es-logging.default.svc
RUN touch /var/lib/rpm/* && yum install -y gcc-c++ && yum clean all
RUN scl enable rh-ruby23 'gem install --no-document fluent-plugin-elasticsearch -v 1.9.2' && \
scl enable rh-ruby23 'gem install fluent-plugin-influxdb -v 1.0.0.rc1 --no-document' && \
scl enable rh-ruby23 'gem install fluent-plugin-parser' && \
scl enable rh-ruby23 'gem install fluent-plugin-grep' && \
scl enable rh-ruby23 'gem cleanup fluentd'
ADD start-fluentd /start-fluentd
Fluentd config:
<source>
type forward
port 24224
bind 0.0.0.0
</source>
<match docker.**>
@type grep
regexp1 log resource_request
add_tag_prefix metric
</match>
<match metric.docker.**>
@type influxdb
measurement measurement
host 10.93.1.21
port 8086
dbname db1
user root
password somepassword
use_ssl false
time_precision s
tag_keys ["key1", "key2"]
sequence_tag _seq
</match>
<match metric.docker.**>
@type stdout
</match># input plugin that exports metrics
Stdout:
{"measurement":"resource_request","tags":{"name":"asd","value":"asd","key1":"test"},"values":{"response_time":1,"status":200}}
I use fluent-logger nodejs.
var config = {
host: 'localhost',
port: 24224,
timeout: 3.0,
milliseconds: 1
};
when I write log to influxdb through fluent(support milliseconds? support nodejs?)
error came:
2017-02-28 10:16:37 +0000 [error]: #0 unexpected error on reading data host="172.17.0.1" port=33606 error_class=NoMethodError error="undefined method `nsec' for 1488276994397.0:Float\nDid you mean? inspect"
2017-02-28 10:16:37 +0000 [error]: #0 suppressed same stacktrace
49131-08-17 16:48:06.000000000 +0000 test: {"level":"info","message":"newdada:"}
2017-02-28 10:18:49 +0000 [warn]: #0 emit transaction failed: error_class=NoMethodError error="undefined method `nsec' for 1488277126086.0:Float\nDid you mean? inspect" tag="test"
Hi,
I try to use this plugin to convert nignx access log into metrics and push it into influxdb.
The nginx access log format likes the following:
[26/Oct/2019:17:21:40 +0000] 302 "[email protected]" "GET / HTTP/1.0" 4801 "-" "-" 0.631 0.631 .
I expect the status code(302), request time(0.631) and response time(0.631) should be saved into influxdb AS number/float data type. But actually all of them are stored as string.
> select * from fe_portal
name: fe_portal
time agent code hostname method path project referer region remote request_time response_time size tailed_path timestamp user version
---- ----- ---- -------- ------ ---- ------- ------- ------ ------ ------------ ------------- ---- ----------- --------- ---- -------
1572110977000000000 GoogleHC/1.0 302 wf20-td-agent-test-1-0 GET / prod-test - asia-southeast1 127.0.0.1 0.630 0.630 4801 /var/log/pan/my-access-test.log 26/Oct/2019:17:29:21 +0000 - HTTP/1.0
> select request_time::float from fe_portal
> select request_time::string from fe_portal
name: fe_portal
time request_time
---- ------------
1572110977000000000 0.630
>
My fluentd config likes the following:
<source>
@type tail
pos_file /var/log/td-agent/nginx-influx.pos
path_key tailed_path
@id input_tail
path /var/log/pan/*access*.log
tag fe_portal
<parse>
@type regexp
expression /^\[(?<timestamp>[^\]]*)\] (?<code>[^ ]*) "(?<user>[^ ]*)@(?<remote>[^ ]*)" "(?<method>\S+)(?: +(?<path>[^\ "]*?)(?: +\S*)?) (?<version>\S+/\S+)" (?<size>[\d.]+) "(?<referer>[^ ]*)" "(?<agent>[^\"]*)" (?<request_time>[\d.-]+) (?<response_time>[\d.-]+).*$/
time_format %d/%b/%Y:%H:%M:%S %z
</parse>
</source>
<filter fe_portal>
@type record_transformer
auto_typecast true
<record>
hostname ${hostname}
project prod-test
region asia-southeast1
</record>
</filter>
<match fe_portal>
type influxdb
num_threads 8
dbname test
host influxdb
port 8086
flush_interval 10s
cast_number_to_float true
@id influxdb
tag_keys ["method","path", "code", "region", "hostname"]
</match>
if record contains a NULL as value, no Error, has POST log, no records, just gone
timeStamp type conversion Error, should be INTEGER or a DATE? but FLOAT with an Error now
auto_tags almost not work, in most time, cause fail.
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/config/basic_parser.rb:88:in `parse_error!': got incomplete JSON array configuration at mqtt.conf line 18,9 (Fluent::ConfigParseError)
17: tag_keys ['key1', 'key2']
18: </match>
---------^
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/config/literal_parser.rb:236:in `scan_json'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/config/literal_parser.rb:57:in `parse_literal'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/config/v1_parser.rb:128:in `parse_element'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/config/v1_parser.rb:93:in `parse_element'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/config/v1_parser.rb:41:in `parse!'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/config/v1_parser.rb:31:in `parse'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/config.rb:29:in `parse'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/supervisor.rb:412:in `read_config'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/supervisor.rb:127:in `start'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/command/fluentd.rb:167:in `<top (required)>'
from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
from /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:73:in `require'
from /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/bin/fluentd:6:in `<top (required)>'
from /opt/td-agent/embedded/bin/fluentd:23:in `load'
from /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
from /usr/sbin/td-agent:7:in `load'
from /usr/sbin/td-agent:7:in `<main>'
Im seeing the following error when trying to send dstat data through fleuntd to influxdb using this plugin.
Versions:
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-dstat-0.2.5
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-influxdb-0.1.2
influxDB ( Tested version 0.6.0 & 0.8.2 )
The error is :
2014-09-14 16:17:28 -0400 [warn]: fluent/output.rb:344:rescue in try_flush: temporarily failed to flush the buffer. next_retry=2014-09-14 16:17:28 -0400 error_class="Net::HTTPServerException" error="400 "Bad Request"" instance=-608640758
2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http/response.rb:119:in error!' 2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/net/http/response.rb:128:in
value'
2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-influxdb-0.1.2/lib/fluent/plugin/out_influxdb.rb:50:in write' 2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.53/lib/fluent/buffer.rb:296:in
write_chunk'
2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.53/lib/fluent/buffer.rb:276:in pop' 2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.53/lib/fluent/output.rb:311:in
try_flush'
2014-09-14 16:17:28 -0400 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.10.53/lib/fluent/output.rb:132:in `run'
My system configuration file for fluentd
I am trying to use the fluentd docker log driver. I write stdout as json and have fluentd forward that to influxdb using this plugin.
fluentd.conf
<source>
@type forward
format json
port 24224
</source>
<match **>
@type influxdb
host influxdb
port 8086
dbname test
user admin
password admin
use_ssl false
measurement pingdom
flush_interval 1s
</match>
It seems as though it just forwards the json that it gets from docker. Most like because of the source setup, but since current influxdb only supports line protocol, how can we we forward from a source of json and have fluentd format that to line protocol?
Hi,
I am facing an issue when fluentd is starting and tries to connect to an unavailable InfluxDB.
Last trace from log is "Connecting to database".
If the DB is unavailable I would like fluentd to open its http source and store items in disk buffer.
Is that possible ?
2018-12-22 01:45:55 +0800 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2018-12-22 01:45:55 +0800 [info]: using configuration file: <ROOT>
<system>
log_level debug
</system>
<source>
@type http
@id input_http
port 8888
</source>
<filter mkd.**>
@type record_transformer
remove_keys _time
<record>
time ${record["_time"]}
</record>
</filter>
<match mkd.dev.depth>
@type influxdb
host "analytics.local"
port 8087
dbname "dev_mkd"
measurement "depth"
tag_keys ["e","s","d","t"]
time_precision "ns"
<buffer>
@type "file"
path "/var/log/td-agent/buffer/mkd.dev.depth"
flush_interval 10s
queued_chunks_limit_size 100
</buffer>
</match>
<match mkd.**>
@type stdout
@id output_stdout
</match>
</ROOT>
2018-12-22 01:45:55 +0800 [info]: starting fluentd-1.2.6 pid=22069 ruby="2.4.4"
2018-12-22 01:45:55 +0800 [info]: spawn command to main: cmdline=["/opt/td-agent/embedded/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/embedded/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-elasticsearch' version '2.11.11'
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-influxdb' version '1.0.0'
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-kafka' version '0.7.9'
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-record-modifier' version '1.1.0'
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.1.0'
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-s3' version '1.1.6'
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-td' version '1.0.0'
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.4'
2018-12-22 01:45:55 +0800 [info]: gem 'fluent-plugin-webhdfs' version '1.2.3'
2018-12-22 01:45:55 +0800 [info]: gem 'fluentd' version '1.2.6'
2018-12-22 01:45:55 +0800 [info]: adding filter pattern="mkd.**" type="record_transformer"
2018-12-22 01:45:55 +0800 [info]: adding match pattern="mkd.dev.depth" type="influxdb"
2018-12-22 01:45:55 +0800 [info]: adding match pattern="mkd.**" type="stdout"
2018-12-22 01:45:55 +0800 [info]: adding source type="http"
2018-12-22 01:45:55 +0800 [info]: #0 starting fluentd worker pid=22086 ppid=22081 worker=0
2018-12-22 01:45:55 +0800 [debug]: #0 buffer started instance=70077661079240 stage_size=1275 queue_size=0
2018-12-22 01:45:55 +0800 [info]: #0 Connecting to database: dev_mkd, host: analytics.local, port: 8087, username: root, use_ssl = false, verify_ssl = true
Is there a way to choose or override the default measurement/series name? Currently it defaults to the fluentd tag which is not ideal in some configruations.
After td-agent restart these are not flushed to InfluxDB and InfluxDB returns response with code 400 (Bad Request). Removing these buffer's files and restarting td-agent, everything is fine, InfluxDB returns 204.
[~]# ls -la /var/log/td-agent/buffer/
total 163852
drwxr-xr-x 2 td-agent td-agent 4096 Nov 17 14:29 .
drwxr-xr-x 3 td-agent td-agent 4096 Nov 17 03:20 ..
-rw-r--r-- 1 td-agent td-agent 6204727 Nov 17 14:36 openresty.buffer..b541800455821e280.log
-rw-r--r-- 1 td-agent td-agent 68346 Nov 17 11:47 openresty.buffer..q5417dc17eb4c3ac2.log
-rw-r--r-- 1 td-agent td-agent 8388194 Nov 17 11:55 openresty.buffer..q5417dc1cb04ba5c3.log
-rw-r--r-- 1 td-agent td-agent 8388383 Nov 17 12:02 openresty.buffer..q5417ddee752d0acd.log
-rw-r--r-- 1 td-agent td-agent 8388508 Nov 17 12:11 openresty.buffer..q5417df82b373f84c.log
-rw-r--r-- 1 td-agent td-agent 8388184 Nov 17 12:21 openresty.buffer..q5417e17983be7504.log
-rw-r--r-- 1 td-agent td-agent 8387911 Nov 17 12:30 openresty.buffer..q5417e3b708d850da.log
-rw-r--r-- 1 td-agent td-agent 8388467 Nov 17 12:40 openresty.buffer..q5417e5c5915f2753.log
-rw-r--r-- 1 td-agent td-agent 8387715 Nov 17 12:49 openresty.buffer..q5417e7ea0f3f354b.log
-rw-r--r-- 1 td-agent td-agent 8387881 Nov 17 12:57 openresty.buffer..q5417e9e221e86eb0.log
-rw-r--r-- 1 td-agent td-agent 8388236 Nov 17 13:06 openresty.buffer..q5417ebc5ac077d42.log
-rw-r--r-- 1 td-agent td-agent 8388493 Nov 17 13:15 openresty.buffer..q5417edd2c3c807b6.log
-rw-r--r-- 1 td-agent td-agent 8388199 Nov 17 13:24 openresty.buffer..q5417efca0eb5e84f.log
-rw-r--r-- 1 td-agent td-agent 8388511 Nov 17 13:34 openresty.buffer..q5417f1b799fed6ab.log
-rw-r--r-- 1 td-agent td-agent 8388219 Nov 17 13:43 openresty.buffer..q5417f3e4ecafe8e4.log
-rw-r--r-- 1 td-agent td-agent 8388417 Nov 17 13:49 openresty.buffer..q5417f5f0e5c500cf.log
-rw-r--r-- 1 td-agent td-agent 8388300 Nov 17 13:57 openresty.buffer..q5417f759ad405209.log
-rw-r--r-- 1 td-agent td-agent 8388106 Nov 17 14:05 openresty.buffer..q5417f93bb3a68ac4.log
-rw-r--r-- 1 td-agent td-agent 8388072 Nov 17 14:12 openresty.buffer..q5417faf8a4073910.log
-rw-r--r-- 1 td-agent td-agent 8388408 Nov 17 14:19 openresty.buffer..q5417fc9077170e98.log
-rw-r--r-- 1 td-agent td-agent 8388289 Nov 17 14:29 openresty.buffer..q5417fe1919044b32.log
is this known bug or something is bad with configuration?
<match metrics.openresty>
type influxdb
host X.X.X.X
port 8086
user metrics
password xxx
dbname metrics
time_key msec
time_precision ms
buffer_type file
buffer_path /var/log/td-agent/buffer/openresty.buffer
buffer_queue_limit 512
flush_interval 5s
</match>
I got an error message like following after I upgrade this plugin to v2.5:
2016-04-22 15:25:57 +0900 [error]: unexpected error error_class=InfluxDB::AuthenticationError error=#<InfluxDB::AuthenticationError: {"error":"error authorizing query: (username) not authorized to execute statement 'SHOW DATABASES', requires admin privilege"}>
I confirmed if I add admin privilege to that user, this error is not shown and the data is imported to InfluxDB correctly. But in my opinion, it's too much for data collection process.
Im trying to figure out how a line in a file must be formated to be forwarded to influx.
After 3 hours with guessing, looking at tests, previous issues and looking a little bit into the code, i think this ticket is my last hope.
Could you add a line to the config section with a example line? This would be a big help!
I think a good example would be the InfluxDB example:
cpu_load_short,host=server02 value=0.67
cpu_load_short,host=server02,region=us-west value=0.55 1422568543702900257
cpu_load_short,direction=in,host=server01,region=us-west value=2.0 1422568543702900257
What should my config and input line look like, to get the same data like the influxdb example, into my influx?:
<source>
type tail
path /tmp/dummy.log
pos_file /tmp/dummy.log.pos
tag dummy.test
format none
</source>
<match dummy.*>
type influxdb
host 10.10.10.1
port 8086
dbname db
user admin
password xxxxxx
use_ssl false
tag_keys ["host"]
time_precision s
</match>
Please check fluent/fluentd#1736
( I quite believe the issue with wrong escapes happen in the influx plugin)
In the work that I am doing, I do not know in advance the tags to be used in tag_keys. For every packet that is received I can look at the data and decide what the tags could be. Is there any way that I can set the 'tag_keys' per record that I am will be sending to influxDB?
For example, I could get a packet which could transform to {'interface-name': 'abc', 'interface-bandwidth': 1234, }. For this the tag should be 'interface-name'. Next I could get a packet which could transform to {'arp-ip': '1.2.3.4' and 'arp-timeout': 34}. Here the index should be 'arp-ip'. All of the packets are coming at the same time, so the timestamp would be the same and these should of to the same measurement. Above records are just example to simplify the use case.
c281115 ensures that no fields are empty strings. Although an empty string is an invalid value for a tag, InfluxDB does allow them for string fields.
For example, this is perfectly valid: test,foo=bar fizz="" 1465839830100400200
This chunk of code looks buggy.
fluent-plugin-influxdb/lib/fluent/plugin/out_influxdb.rb
Lines 113 to 120 in ab91b7e
'tag_keys' never works for strings or/and 'auto_tags' is false, it's right logic?
Dear fangli,
Today i view td-agent log and see error log: unable to parse... . I sure my format of message's correct, But plugin say unable to parse ( miss "," between "ran=408880 ssec=1448935558" ). And buffer stuck this error message cannot retry (because of error) or by pass error message when i restart td-agent. I must delete buffer. Please check & fix it
P/s: I ran td-agent with influxdb-plugin in 3 days but today it error. I delete buffer & restart , td-agent work back
2015-12-01 10:13:03 +0700 [warn]: temporarily failed to flush the buffer. next_retry=2015-12-01 10:34:00 +0700 error_class="InfluxDB::Error" error="unable to parse 'cdn.stream1.mecloud.vn,node=mc11,schema=http,domain=cdn.stream1.mecloud.vn,uri=/clip/2015/12/1/8/19/77050ecf7bccc5338afb810797050cd6_360.mp4,status=206,cache=HIT,ip=113.162.174.190,nation=VN,region=Ho\ Chi\ Minh,agent=GET\ /clip/2015/12/1/8/19/77050ecf7bccc5338afb810797050cd6_360.mp4?e=1448949944&s=8y0vl3kSYUgw7jkY-sXqAw\ HTTP/1.1,ran=408880 ssec=1448935558,byte=1188298,ust=0,usl=0,hit=1,xx2=1,xx3=0,xx4=0,xx5=0,byteos=0,bytevn=1188298,req=1,reqvn=1,reqos=0 1448935558': missing tag value\n" plugin_id="object:3feac88730c8"
2015-12-01 10:13:03 +0700 [warn]: suppressed same stacktrace
Hi!
I'm trying to insert in some sequentials writes in InfluxDB comming from a "not stable connection environment". Td-agent, basically reads from rabbitmq, and insert into a measurement schema.
The configuration in the server side is this:
<source>
@type amqp
host localhost
port 5672
vhost /
user admin
pass mypass
queue health_info
format json
durable true
tag_key true
</source>
<match **>
@type influxdb
dbname health
user admin
password mypass
use_ssl false
retry 0
num_threads 1
sequence_tag _seq
</match>
The problem comes when i cut off the connection between my client and server: when RabbitMQ is reconnected, all the stored messages pass between host and server in order, and they are diferents (they have an additional "timestamp" field) but they are written totally disordered at InfluxDB
Some example messages:
{"disk_usage": 2, "cpu_usage": 1.0, "percent_mem": 20.9, "disk_percent": 57.7, "used_mem": 193, "timestamp": "2017-01-07T17:41:32Z"}
{"disk_usage": 2, "cpu_usage": 0.5, "percent_mem": 20.9, "disk_percent": 57.7, "used_mem": 193, "timestamp": "2017-01-07T17:41:42Z"}
Some considerations:
a) This is not happening when connection is not cutted off between AMQP server altough you're writing these at InfluxDB. All messages are written in order.
b) RabbitMQ is passing once message (line) at time.
c) If i replace InfluxDB for file output plugin, this works correctly and all lines are written sequentially onto the file, as we were passed across.
Can this be a bug?
Thanks!
Practically, json format data will be flattened before sending to influxdb ,
example:
// product log
{
path: 'xxxxxxxx',
product:
{
id: 'xxx',
'name': 'xxx'
}
message: 'xxxx'
}
// task log
{
path: 'xxxxxxxx',
task:
{
id: 'xxx',
user:
{
id: 'xxx',
nickname: 'xxx'
},
product:
{
id: 'xxx',
'name': 'xxx'
}
},
message: 'xxxx'
}
logger.emit('info', flatten(obj))
if I want to mark all id as tags
I have to set tag_keys = [ 'product.id', 'task.user.id', 'task.product.id' ]
With more nested data structure, I have to mark more stupid chained key name manually
so could we add feature support wildcards
**.id as fluent <match>
pattern
or regex format /$id/
???
Hi,
I found that the backslash escaping in the tag/field values was not working correctly as expected.
With config:
<match mytag>
@type copy
<store>
@type stdout
</store>
<store>
@type influxdb
host influxdb
port 8086
dbname mydb
tag_keys ["hostname", "v.device_name", "v.mount_point"]
</store>
</match>
And input record (from metricbeat):
{"hostname":"my-machine","v.device_name":"D:\\","v.mount_point":"D:\\","v.total":26840395776,"v.free":5271150592,"v.available":5271150592,"v.files":0,"v.free_files":0,"v.used.pct":0.8036,"v.used.bytes":21569245184}
Will cause the following exception:
#0 taking back chunk for errors. chunk="55a01ef2e4d034e413d394ffd26be22c"
#0 failed to flush the buffer. retry_time=0 next_retry_seconds=2017-09-25 11:34:03 +0000 chunk="55a01ef2e4d034e413d394ffd26be22c" error_class=InfluxDB::Error error="{\"error\":\"unable to parse 'system.filesystem,hostname=my-machine,v.device_name=D:\\\\,v.mount_point=D:\\\\ v.total=26840395776i,v.free=5271150592i,v.available=5271150592i,v.files=0i,v.free_files=0i,v.used.pct=0.8036,v.used.bytes=21569245184i 1501729298688': invalid tag format\"}\n"
#0 /var/lib/gems/2.3.0/gems/influxdb-0.3.16/lib/influxdb/client/http.rb:85:in `resolve_error'
#0 /var/lib/gems/2.3.0/gems/influxdb-0.3.16/lib/influxdb/client/http.rb:35:in `block in post'
#0 /var/lib/gems/2.3.0/gems/influxdb-0.3.16/lib/influxdb/client/http.rb:53:in `connect_with_retry'
#0 /var/lib/gems/2.3.0/gems/influxdb-0.3.16/lib/influxdb/client/http.rb:26:in `post'
#0 /var/lib/gems/2.3.0/gems/influxdb-0.3.16/lib/influxdb/query/core.rb:78:in `write'
#0 /var/lib/gems/2.3.0/gems/influxdb-0.3.16/lib/influxdb/query/core.rb:57:in `write_points'
#0 /fluentd/plugins/out_influxdb2.rb:189:in `write'
#0 /var/lib/gems/2.3.0/gems/fluentd-0.14.21/lib/fluent/plugin/output.rb:1061:in `try_flush'
#0 /var/lib/gems/2.3.0/gems/fluentd-0.14.21/lib/fluent/plugin/output.rb:1286:in `flush_thread_run'
#0 /var/lib/gems/2.3.0/gems/fluentd-0.14.21/lib/fluent/plugin/output.rb:438:in `block (2 levels) in start'
#0 /var/lib/gems/2.3.0/gems/fluentd-0.14.21/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
The exception message shows that the record has been wrongly escaped:
(double escaped backslashes and a comma ,
before v.total
disappeared)
v.device_name=D:\\\\,v.mount_point=D:\\\\ v.total=26840395776i,v.free=5271150592i,v.available=5271150592i
I also checked the plugin code and printed the hash point
before calling @influxdb.write_points()
:
{:timestamp=>1501729298688, :series=>"system.filesystem", :values=>{"v.total"=>26840395776, "v.free"=>5271150592, "v.available"=>5271150592, "v.files"=>0, "v.free_files"=>0, "v.used.pct"=>0.8036, "v.used.bytes"=>21569245184}, :tags=>{"hostname"=>"my-machine", "v.device_name"=>"D:\\", "v.mount_point"=>"D:\\"}}
I guess that it is caused by influxdb-ruby
, but I think that asking here is a good start.
Thanks.
Environment:
fluentd 0.14.21
fluent-plugin-influxdb 1.0.0.rc3
influxdb-ruby 0.3.16
Original log line like this;
Parameters: {"commit_message"=>"Revert \"Merge Branch 'hot_fix' into 'master'\"\r\n\r\nSee MR !2"}
should be encoded on the line protocol like this;
production.log,message="Parameters ...snip... into 'master'\\\"\\r\\n\\r\\nSee MR !2\"}"
but out_influxdb encodes like this;
production.log,message="Parameters ...snip... into 'master'\\"\r\n\r\nSee MR !2\"}"
which InfluxDB can not parse and returns "unable to parse ... bad timestamp" error.
For your reference, out_file encodes it just right;
production.log {"message":"Parameters ...snip... into 'master'\\\"\\r\\n\\r\\nSee MR !2\"}"}
Versions:
fluent-plugin-influxdb:1.0.0.rc2
fluentd 0.14
influxdb 1.2
Any reason the below fluentd
configuration would have issues with flush_interval 1s
. It seems to be using the default 60s flush_interval
.
Example fluentd.conf
...
<match **>
@type influxdb
host influxdb
port 8086
dbname test
user admin
password admin
use_ssl false
measurement pingdom
flush_interval 1s
</match>
The influxdb-ruby library is moving to 0.9 support. Influxdb-ruby v0.1.9 will be the last version that supports 0.8. Without a version constraint on fluent-plugin-influxdb for influxdb-ruby, many new production deployments that have not upgraded to 0.9 will fail un-expectantly.
I recommend that we add a constraint to use only influxdb-ruby v0.1.9 so that we will not fail to work with 0.8 when the next version is released. It might be useful to start versioning fluent-plugin-influxdb such that version 0.1.* works with influxdb 0.8 while a fluent-plugin-influxdb 0.2.* version is compatible with influxdb 0.9. If people use Semver correctly, this should help so that we don't get any surprises in production.
It would be great to have sanity check for parameters passed to plugin.
At the moment when host is empty, fluentd loads the config and silently hangs.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.