Latency metrics for batch reads? about asprom

[Xaelias]

Like I said last time, these metrics are just not readily available AFAIK.
I can double check (probably not this week though) and see if something changed since then though. Just harass me if I (or alicebob) doesn't come back to you :-)

[alicebob]

Hi!
thanks for the explanation. I'll have to have a look at what's available exactly and will get back to you (but that won't be today).

[Xaelias]

That's great! Thanks :-)

[alicebob]

what does asinfo -l -v latency: give you?

I'm looking at the docs at https://www.aerospike.com/docs/operations/monitor/latency/index.html
Are you referring to the batch-index key or the batch-sub-* commands?

Back in the day the histogram values were only available in the aerospike log file, not via the API. That changed for the basic latency values a few releases ago, so asprom reads the values via the API, but I'm still not quite sure which of the Configuration-enabled benchmarks are available via the API.

I do agree that the basic latency stats which are always enabled should always show up, even with a 0 value, for every namespace.

[Xaelias]

So running the asinfo command, I get things like this:

{ns1}-read:13:42:53-GMT,ops/sec,>1ms,>8ms,>64ms
13:43:03,73.0,0.14,0.00,0.00
{ns1}-write:13:42:53-GMT,ops/sec,>1ms,>8ms,>64ms
13:43:03,102.2,0.00,0.00,0.00
error-no-data-yet-or-back-too-small
error-no-data-yet-or-back-too-small
error-no-data-yet-or-back-too-small
{ns2}-write:13:42:53-GMT,ops/sec,>1ms,>8ms,>64ms
13:43:03,556.9,0.04,0.00,0.00
error-no-data-yet-or-back-too-small
error-no-data-yet-or-back-too-small
{ns3}-read:13:42:53-GMT,ops/sec,>1ms,>8ms,>64ms
13:43:03,11.4,0.88,0.00,0.00
{ns3}-write:13:42:53-GMT,ops/sec,>1ms,>8ms,>64ms
13:43:03,775.5,0.00,0.00,0.00
error-no-data-yet-or-back-too-small
error-no-data-yet-or-back-too-small

Which is not really helpfull... So you'll notice that ns2-read is not there. But the other metrics are apparently not available either...

It looks like batch-sub would be what you want to use. Whatever item(s) matche(s) the one(s) you use for regular reads.

[alicebob]

So there are two things going on:

• if there is no ns2-read row present in the output asprom won't give the prometheus key. Would it be possible to solve that in the prometeus query? Maybe with sum(...) or 0?

• non-default measurements such as batch-sub are not available via asinfo, but it seems only in the aerospike log file. Seems a bit of a stretch to start tailing log files, so if aerospike doesn't export the data via asinfo there is not much to do. Maybe you can ask the aerospike people how to get to that data?

[Xaelias]

Sorry I was on vacation and completely forgot to check on this ><
I'll send a quick email to support and see what they have to say :-)

[Xaelias]

'k it sounds like batch metrics can be had. But they advise not to have them on all the time (probably because of performances). So...
I guess you can close that issue.

Thanks for looking into it. And sorry for the late reply :-)

[alicebob]

Thanks for getting back :)

If you ever find some (cheap) source of the metrics let me know!

[jiayiyang]

Hi all,
I have the same question on this, can we have batch-index latency histogram in asprom? We have a namespace which only have batch read, but we cannot get related latency metrics via asprom, will this metric be supported?