Histograms .avg uneven about veneur

[cory-stripe]

I'm not sure I fully understand the question yet, but one thing jumps out at me: The difference in the local veneur and aggregating veneur's flush periods. By using a 20 second interval on the aggregator, you're explicitly averaging a longer window of time. Might this be the source of your problem?

[ruilapa]

We had already tested with the same flush periods and the results were the same.

After comparing code from dd-agent and veneur, there is a slight difference in the way .avg are calculated, but with our sampleRate (1.000) it does not make a difference in the final result.

Is there something on the aggregation or forward that I'm not correctly configuring or seeing?
Was there a previous bug in histogram .avg or incompatibility between 1.3.1 and 1.7.0 ?

[cory-stripe]

There've been no changes in histograms, their underlying implementation or averages in any versions for a long time. I don't think that is the case. Do you have any illustrations of the differences?

[ruilapa]

We configured the aggregator to also inject metrics into an influxdb.
We noticed that the .avg metrics are not forwarded to it.

Is this correct? And how may I solve it?

[cory-stripe]

I am not sure why this would happen, as these metrics aren't special with regards to InfluxDB. It gets mostly the same metrics as Datadog or any other backend… (as far as I know)

[ruilapa]

A little text architecture:

servers (using RR DNS) --UDP--> veneur -f veneurs.cfg.txt --/import--> veneur -f aggregator.cfg.txt --> datadog and influxdb

Example metric generated by servers:
server.game_duration:605|h|@1.000|#version:n,environment:live,platform:mobile,host:blabla19,instance_tag:i-xxxxxx

Metrics injected by the aggregator in influxdb:

SHOW MEASUREMENTS WITH MEASUREMENT =~ /pool.game_duration/
name: measurements
name

server.game_duration.95percentile
server.game_duration.99percentile
server.game_duration.median

[cory-stripe]

Gotya,

Can you show a chart or samples of some of the differences in the avg?

Sorry for all the questions!

[ruilapa]

If the questions mean we might get answers... I'll answer those I can. ;)
But unfornately I cannot show the graphs side by side, because one involves multiple accumulated version's and some math spliting them.

I temporarily changed the veneurs config to aggregate .avg and they shipped the metric.
It can be seen in the datadog metrics summary screenshot they're named as "prod-veneur-proxy".

Rolled back, waited and:

This meant that the .avg metric is not calculated by veneur.

• https://docs.datadoghq.com/guides/metrics/#histograms

Further investigation in Datadog Metrics Summary, showed that .avg is the only metric affected, since all the other (count, max, min, ...) have all the servers hosts tags.

Question: If I use latest (1.8.0) and disable datadog on veneurs will all metrics be shipped to the aggregator? Or do I need to use veneur-proxy (if it receives UDP metrics)?

[ruilapa]

Question: If I use latest (1.8.0) and disable datadog on veneurs will all metrics be shipped to the aggregator? Or do I need to use veneur-proxy (if it receives UDP metrics)?