Error in decoding JSON sent by druid about druid-exporter

[iamabhishek-dubey]

Sure I will be fixing this soon

[iamabhishek-dubey]

Please use this version of release
https://github.com/opstree/druid-exporter/releases/tag/v0.3

[freemanlutsk]

and another one (

An error has occurred while serving metrics:

10 error(s) occurred:
* collected metric "druid_tasks" { label:<name:"created_time" value:"2020-05-18T15:22:10.764Z" > label:<name:"datasource" value:"xxx-impressions" > label:<name:"index_group_id" value:"index_kafka_xxx-impressions" > label:<name:"task_status" value:"RUNNING" > gauge:<value:1 > } was collected before with the same name and label values
* collected metric "druid_tasks" { label:<name:"created_time" value:"2020-05-18T15:22:10.841Z" > label:<name:"datasource" value:"xxx-bids" > label:<name:"index_group_id" value:"index_kafka_xxx-bids" > label:<name:"task_status" value:"RUNNING" > gauge:<value:1 > } was collected before with the same name and label values
* collected metric "druid_tasks" { label:<name:"created_time" value:"2020-05-18T15:22:10.911Z" > label:<name:"datasource" value:"xxx-bids" > label:<name:"index_group_id" value:"index_kafka_xxx-bids" > label:<name:"task_status" value:"RUNNING" > gauge:<value:1 > } was collected before with the same name and label values
* collected metric "druid_tasks" { label:<name:"created_time" value:"2020-05-18T15:22:10.967Z" > label:<name:"datasource" value:"xxx-ad-data" > label:<name:"index_group_id" value:"index_kafka_xxx-ad-data" > label:<name:"task_status" value:"RUNNING" > gauge:<value:1 > } was collected before with the same name and label values
* collected metric "druid_tasks" { label:<name:"created_time" value:"2020-05-18T15:22:10.967Z" > label:<name:"datasource" value:"xxx-invalid-events" > label:<name:"index_group_id" value:"index_kafka_xxx-invalid-events" > label:<name:"task_status" value:"RUNNING" > gauge:<value:1 > } was collected before with the same name and label values
* collected metric "druid_tasks" { label:<name:"created_time" value:"2020-05-18T15:22:10.967Z" > label:<name:"datasource" value:"auction-events" > label:<name:"index_group_id" value:"index_kafka_auction-events" > label:<name:"task_status" value:"RUNNING" > gauge:<value:1 > } was collected before with the same name and label values
* collected metric "druid_tasks" { label:<name:"created_time" value:"2020-05-18T15:22:10.967Z" > label:<name:"datasource" value:"xxx-openrtb-events" > label:<name:"index_group_id" value:"index_kafka_xxx-openrtb-events" > label:<name:"task_status" value:"RUNNING" > gauge:<value:1 > } was collected before with the same name and label values
* collected metric "druid_tasks" { label:<name:"created_time" value:"2020-05-18T15:22:11.001Z" > label:<name:"datasource" value:"xxx-ad-data" > label:<name:"index_group_id" value:"index_kafka_xxx-ad-data" > label:<name:"task_status" value:"RUNNING" > gauge:<value:1 > } was collected before with the same name and label values
* collected metric "druid_tasks" { label:<name:"created_time" value:"2020-05-18T15:22:11.001Z" > label:<name:"datasource" value:"ad-requests" > label:<name:"index_group_id" value:"index_kafka_ad-requests" > label:<name:"task_status" value:"RUNNING" > gauge:<value:1 > } was collected before with the same name and label values
* collected metric "druid_tasks" { label:<name:"created_time" value:"2020-05-18T15:22:11.022Z" > label:<name:"datasource" value:"auction-events" > label:<name:"index_group_id" value:"index_kafka_auction-events" > label:<name:"task_status" value:"RUNNING" > gauge:<value:1 > } was collected before with the same name and label values

[freemanlutsk]

Thank You!
It seems duplicate errors still exist.

collected metric "druid_tasks_status_duration" { label:<name:"created_time" value:"2020-05-19T17:26:54.179Z" > label:<name:"datasource" value:"xxx-clicks" > label:<name:"index_group_id" value:"index_kafka_xxx-clicks" > label:<name:"task_status" value:"RUNNING" > gauge:<value:0 > } was collected before with the same name and label values

[iamabhishek-dubey]

I'll check this shouldn't be the case. I guess the API is returning the duplicate tasks.

[freemanlutsk]

Is there a way to skip this error? Or another way how to avoid this.
Thank You!

[iamabhishek-dubey]

It needs development effort, but don't worry I will fix this by tomorrow

[iamabhishek-dubey]

Hopefully, this will fix your problem for task name duplication
https://github.com/opstree/druid-exporter/releases/tag/v0.3

I have updated the same release for you. Let me know if the issue still persist

[freemanlutsk]

Thanks for your patience and quick response.
After latest fixes something wrong with "count(druid_tasks_duration) by (task_status)"

• Metrics do not refresh during a long time.
• Difference between Druid console and exporter.
  

[iamabhishek-dubey]

So task details are the information that druid is fetching not emitted. So you should configure your scrape interval in the Prometheus job block. I guess by default it is seconds.

[freemanlutsk]

- job_name: druid_monitoring
  honor_timestamps: true
  scrape_interval: 1m
  scrape_timeout: 10s
  metrics_path: /metrics
  ...

This is my config for druid's job.

[iamabhishek-dubey]

- job_name: druid_monitoring
  honor_timestamps: true
  scrape_interval: 10s
  scrape_timeout: 10s
  metrics_path: /metrics

Change scrape_interval to 10s

[freemanlutsk]

Thank You for advice.
It seems after restart exporter data was changed. Strange behaviour...

[freemanlutsk]

Yes, I am correct. After restart an exporter the date became the same as in the druid console.
And seems like it isn't just for druid_tasks_duration.

[freemanlutsk]

I checked exporter's logs, data from druid/indexer/v1/tasks is grabbing only once after start/restart, I guess it is a cause of my issues.

{"level":"info","msg":"GET request is successful for druid api","ts":"2020-05-21T06:42:43.732Z","url":"http://data:8888/druid/indexer/v1/tasks"}
{"level":"info","msg":"Successfully retrieved the data for druid's supervisors tasks","ts":"2020-05-21T06:42:43.742Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.769Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.769Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.769Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.769Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.769Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.769Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.769Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.769Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.769Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.769Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.769Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.769Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.769Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.770Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.771Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"err":"strconv.ParseFloat: parsing \"\u003cnil\u003e\": invalid syntax","level":"error","msg":"Unable to parse the duration value","ts":"2020-05-21T06:42:43.772Z"}
{"level":"info","msg":"Druid exporter started listening on :7072","ts":"2020-05-21T06:42:43.772Z"}