alicebob / asprom

Aerospike prometheus exporter

Project status

Since Aerospike 4.5 (beginning 2020) the Aerospike team have their own Prometheus exporter. It's likely a better choice if you run version 4.5 or higher of Aerospike, especially if you run the Enterprise Edition. The exporter here might be a better choice is you run an older version. PRs are still welcome, but don't expect active maintenance to stay up to date with the current Aerospike.

https://github.com/aerospike/aerospike-prometheus-exporter

Aerospike Prometheus exporter

This follows the logic from asgraphite. Run a asprom collector against every node in the aerospike cluster.

Statistics collected:

• aerospike_node_*: node wide statistics. e.g. memory usage, cluster state.
• aerospike_ns_*: per namespace. e.g. objects, migrations.
• aerospike_sets_*: statistics per set: objects, memory usage
• aerospike_latency_*: read/write/etc latency rates(!), per namespace
• aerospike_ops_*: read/write/etc ops per second, per namespace

Binaries

The releases page has binaries.

Building

• install the Go compiler
• run make
• copy the ./asprom binary to where you need it

It's also easy to crosscompile with Go. You can build asprom for Linux on a Mac with: GOOS=linux GOARCH=amd64 go build and then copy the asprom binary over to your Linux machines.

asprom's Issues

add control over export port via env parameters

Nice to have feature

Latency metrics for batch reads?

Hey,

I tried to read the latency.go code but am a bit lost.
In one of our namespaces, we mostly do batch reads. You don't include these in things like aerospike_ops_read (we add aerospike_ns_batch_sub_read_success to it currently), and I don't see latency metrics (like aerospike_latency_read) at all.

I was wondering if:

• metrics existed at all for the batch latencies etc.?

• can we either include them with the regular metrics, or create 2 new set of metrics:
  1 for batch read/writes
  1 for regular read/writes (already exist)
  1 that's an aggregated view of the 2

That last one is useful because for instance right now we have:

sum(aerospike_ops_read) by (namespace) + sum(irate(aerospike_ns_batch_sub_read_success[5m])) by (namespace)

One of the problems is that for instance, on one of our namespace, we have literally 0 regular reads. We only do batch reads. Which means that the first gauge doesn't exist (until someone does a manual read on the cluster). And the sum of "undefined" + 3 is... "undefined" in prometheus.

Let me know if I can help with anything :-)

Thanks!

./main.go:168:34: cannot use hp (type []byte) as type string in argument to conn.Authenticate

Trying to fork and add the new compression-ratio metric, but can't run make. What version of go should I be using here?

https://www.aerospike.com/docs/operations/configure/namespace/storage/compression.html

go version go1.11.3 darwin/amd64

$ make
go build
# _/Users/alb0t/code/asprom
./main.go:168:34: cannot use hp (type []byte) as type string in argument to conn.Authenticate
make: *** [build] Error 2

relationship to asgraphite

@alicebob hi Harmen, I opened an issue over at the asgraphite repository about missing metrics. I had assumed that the metrics were missing due to missing implementation in the asgraphite library ... but now I am wondering if I misinterpreted the phrase "follows the logic from asgraphite" in the (asprom README)[https://github.com/alicebob/asprom/blob/master/README.md].

Is this library a reimplementation of asgraphite functionality, or does it "wrap" the asgraphite library? If the former, it appears I would be better off creating the asgraphite issue here instead of over there.

dashboard

Hi!
Thnx for aerospike exporter, good job!
Could you please share your grafana dashboard in json format?

Provision to pass credential via environment variables

Currently asprom takes the aerospike user credential as command line arguments. This allows other users to figure out the credentials by issuing a simple 'ps' command.
Please provide a option to provide credentials via environment variables.

aerospike_node_up 0

Hi!

Thanks for this great tool!

I'm having some issues getting the exporter working. I'm using the 1.8.0 binary. What versions of Aerospike are compatible? I'm using community edition 3.6.3.

2020/04/28 23:25:41 starting asprom. listening on :9145
2020/04/28 23:25:45 latency: missing measurements line
2020/04/28 23:25:50 latency: missing measurements line
2020/04/28 23:25:55 latency: missing measurements line

$ asinfo -l -v latency:
reads:23:29:34-GMT,ops/sec,>1ms,>8ms,>64ms
23:29:44,0.0,0.00,0.00,0.00
writes_master:23:29:34-GMT,ops/sec,>1ms,>8ms,>64ms
23:29:44,0.0,0.00,0.00,0.00
proxy:23:29:34-GMT,ops/sec,>1ms,>8ms,>64ms
23:29:44,0.0,0.00,0.00,0.00
udf:23:29:34-GMT,ops/sec,>1ms,>8ms,>64ms
23:29:44,0.0,0.00,0.00,0.00
query:23:29:34-GMT,ops/sec,>1ms,>8ms,>64ms
23:29:44,0.0,0.00,0.00,0.00

It looks like the node is not up?

# HELP aerospike_node_scrapes_total Total number of times Aerospike was scraped for metrics.
# TYPE aerospike_node_scrapes_total counter
aerospike_node_scrapes_total 7
# HELP aerospike_node_up Is this node up
# TYPE aerospike_node_up gauge
aerospike_node_up 0

latency: missing measurements line

2017/10/13 16:16:27 starting asprom. listening on :9145
2017/10/13 16:16:32 latency: missing measurements line
2017/10/13 16:16:47 latency: missing measurements line

I suspect this is because we have authentication enabled. Is there any work around?

thanks

Added TLS support configuration

Hi alicebob,
I have added a TLS support to the tool.
I changed the flags a bit and basically added an option to choose either secure connection or not.
Would love to contribute back, wdyt?

prometheus.Counter's Set method is deprecated

Hi! It seems that prometheus maintainers have changed Counter interface.

$ go build ./main.go:116: asc.up.Set undefined (type prometheus.Counter has no field or method Set) ./main.go:120: asc.up.Set undefined (type prometheus.Counter has no field or method Set)

Refer to prometheus/client_golang@3fb8ace

Security?

Hi,
This doesn't have support for a user/password/credentials file?
People really run their cluster without security enabled?

Thanks :-)

No binaries in release?

The last two releases don't have binaries. Is this intentional?

segfault

We're seeing asprom crashing on a regular basis.
When you have time to look into it :-)

Mar 02 17:49:55 <hostname> aerospike_exporter[72033]: 2018/03/02 17:49:55 starting asprom. listening on :9145
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]: 2018/03/02 17:51:09 read tcp 127.0.0.1:49048->127.0.0.1:3000: i/o timeout
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]: panic: runtime error: invalid memory address or nil pointer dereference
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x58 pc=0x659c4d]
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]: goroutine 11 [running]:
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]: asprom/vendor/github.com/aerospike/aerospike-client-go.(*Connection).Write(0xc42023b800, 0xc42044a1e0, 0x13, 0x1b, 0x0, 0x5, 0xc420046400)
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]:         /Users/alesieur/code/Go/src/asprom/vendor/github.com/aerospike/aerospike-client-go/connection.go:148 +0x6d
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]: asprom/vendor/github.com/aerospike/aerospike-client-go.(*info).sendCommand(0xc420186000, 0xc42023b800, 0x6, 0xc42014a070)
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]:         /Users/alesieur/code/Go/src/asprom/vendor/github.com/aerospike/aerospike-client-go/info.go:107 +0x72
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]: asprom/vendor/github.com/aerospike/aerospike-client-go.newInfo(0xc42023b800, 0xc4203f3e98, 0x1, 0x1, 0x0, 0x80, 0x320f4d50004aa984)
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]:         /Users/alesieur/code/Go/src/asprom/vendor/github.com/aerospike/aerospike-client-go/info.go:88 +0x233
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]: asprom/vendor/github.com/aerospike/aerospike-client-go.RequestInfo(0xc42023b800, 0xc4203f3e98, 0x1, 0x1, 0x35, 0xc420080000, 0xc4200100d0)
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]:         /Users/alesieur/code/Go/src/asprom/vendor/github.com/aerospike/aerospike-client-go/info.go:96 +0x4d
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]: main.nsCollector.collect(0xc4201503f0, 0xc42023b800, 0xc42023b7a0)
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]:         /Users/alesieur/code/Go/src/asprom/namespaces.go:227 +0x7d
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]: main.(*asCollector).Collect(0xc420072780, 0xc42023b7a0)
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]:         /Users/alesieur/code/Go/src/asprom/main.go:157 +0x200
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]: asprom/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc4200d3960, 0xc42023b7a0, 0xa7ace0, 0xc420072780)
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]:         /Users/alesieur/code/Go/src/asprom/vendor/github.com/prometheus/client_golang/prometheus/registry.go:383 +0x61
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]: created by asprom/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather
Mar 02 17:51:09 <hostname> aerospike_exporter[72033]:         /Users/alesieur/code/Go/src/asprom/vendor/github.com/prometheus/client_golang/prometheus/registry.go:384 +0x2ec
Mar 02 17:51:09 <hostname> systemd[1]: aerospike-exporter.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 02 17:51:09 <hostname> systemd[1]: aerospike-exporter.service: Unit entered failed state.
Mar 02 17:51:09 <hostname> systemd[1]: aerospike-exporter.service: Failed with result 'exit-code'.```

"Missing" statistics

Hey,

Since you've been so nice :-D
Any reason why you limited yourself to these metrics instead of grabbing all of them?

I am thinking specifically to 1 that is missing that we could use (but there are probably more), and that's master_objects and non_replica_objects (alongside replication-factor and effective_replication_factor).

Alexis