kbudde / rabbitmq_exporter

Prometheus exporter for RabbitMQ

📣 EOL Announcement

IMPORTANT: This exporter only works with RabbitMQ 3. Please use the official exporter for RabbitMQ 4 or newer. See #383 for details.

RabbitMQ Exporter

Prometheus exporter for RabbitMQ metrics. Data is scraped by prometheus.

Please note this an unofficial plugin. There is also an official plugin from RabbitMQ.com. See comparison to official exporter

Installation

Binary release

You can download the latest release on the release page. Docker images are push to docker hub

docker: rabbitmq container with network sharing

The rabbitmq_exporter is sharing the network interface with the rabbitmq container -> it is possible to use localhost and default user/password (guest).

1. Start rabbitMQ

    docker run -d -e RABBITMQ_NODENAME=my-rabbit --name my-rabbit -p 9419:9419 rabbitmq:3-management
   

2. Start rabbitmq_exporter in container.

    docker run -d --net=container:my-rabbit kbudde/rabbitmq-exporter
   

Now your metrics are exposed through http://host:9419/metrics. The management plugin does not need to be exposed.

Configuration

Rabbitmq_exporter can be configured using json config file or environment variables for configuration.

Config file

Rabbitmq_exporter expects config file in "conf/rabbitmq.conf". If you are running the exporter in a container (docker/kubernetes) the config must be in "/conf/rabbitmq.conf" The name of the file can be overriden with flag:

./rabbitmq_exporter -config-file config.example.json

You can find an example here. Note: If you are using a config file, you must provide all values as there is no default value.

Settings

Environment variable	default	description
RABBIT_URL	http://127.0.0.1:15672	url to rabbitMQ management plugin (must start with http(s)://)
RABBIT_USER	guest	username for rabbitMQ management plugin. User needs monitoring tag!
RABBIT_PASSWORD	guest	password for rabbitMQ management plugin
RABBIT_CONNECTION	direct	direct or loadbalancer, strips the self label when loadbalancer
RABBIT_USER_FILE		location of file with username (useful for docker secrets)
RABBIT_PASSWORD_FILE		location of file with password (useful for docker secrets)
PUBLISH_PORT	9419	Listening port for the exporter
PUBLISH_ADDR	""	Listening host/IP for the exporter
OUTPUT_FORMAT	TTY	Log ouput format. TTY and JSON are suported
LOG_LEVEL	info	log level. possible values: "debug", "info", "warning", "error", "fatal", or "panic"
CAFILE	ca.pem	path to root certificate for access management plugin. Just needed if self signed certificate is used. Will be ignored if the file does not exist
CERTFILE	client-cert.pem	path to client certificate used to verify the exporter's authenticity. Will be ignored if the file does not exist
KEYFILE	client-key.pem	path to private key used with certificate to verify the exporter's authenticity. Will be ignored if the file does not exist
SKIPVERIFY	false	true/0 will ignore certificate errors of the management plugin
SKIP_VHOST	^$	regex, matching vhost names are not exported. First performs INCLUDE_VHOST, then SKIP_VHOST. Applies to queues and exchanges
INCLUDE_VHOST	.*	regex vhost filter. Only matching vhosts are exported. Applies to queues and exchanges
INCLUDE_QUEUES	.*	regex queue filter. Just matching names are exported
SKIP_QUEUES	^$	regex, matching queue names are not exported (useful for short-lived rpc queues). First performed INCLUDE, after SKIP
INCLUDE_EXCHANGES	.*	regex exchange filter. (Only exchanges in matching vhosts are exported)
SKIP_EXCHANGES	^$	regex, matching exchanges names are not exported. First performed INCLUDE, after SKIP
RABBIT_CAPABILITIES	bert,no_sort	comma-separated list of extended scraping capabilities supported by the target RabbitMQ server
RABBIT_EXPORTERS	exchange,node,queue	List of enabled modules. Possible modules: connections,shovel,federation,exchange,node,queue,memory
RABBIT_TIMEOUT	30	timeout in seconds for retrieving data from management plugin.
MAX_QUEUES	0	max number of queues before we drop metrics (disabled if set to 0)
EXCLUDE_METRICS		Metric names to exclude from export. comma-seperated. e.g. "recv_oct, recv_cnt". See exporter_*.go for names

Example and recommended settings:

SKIP_QUEUES="RPC_.*" MAX_QUEUES=5000 ./rabbitmq_exporter

Extended RabbitMQ capabilities

Newer version of RabbitMQ can provide some features that reduce overhead imposed by scraping the data needed by this exporter. The following capabilities are currently supported in RABBIT_CAPABILITIES env var:

• no_sort: By default RabbitMQ management plugin sorts results using the default sort order of vhost/name. This sorting overhead can be avoided by passing empty sort argument (?sort=) to RabbitMQ starting from version 3.6.8. This option can be safely enabled on earlier 3.6.X versions, but it'll not give any performance improvements. And it's incompatible with 3.4.X and 3.5.X.
• bert: Since 3.6.9 (see rabbitmq/rabbitmq-management#367) RabbitMQ supports BERT encoding as a JSON alternative. Given that BERT encoding is implemented in C inside the Erlang VM, it's way more effective than pure-Erlang JSON encoding. So this greatly reduces monitoring overhead when we have a lot of objects in RabbitMQ.

Comparison to official exporter

official exporter:

• has runtime/erlang metrics
• aggregated or per-object metrics
• missing filter

This exporter:

• works also with older versions of rabbitmq
• has more configuration options/ filtering of objects
• (bad) depends on data from management interface which can be slow/delayed

probalby best solution is to use both exporters: comment from shamil

common errors / FAQ

msg: Error while retrieving data from rabbitHost statusCode: 500

This exporter expects capabilities from rabbitmq 3.6.8 or newer by default. If you are running older than 3.6.8 you must disable bert and no_sort with the setting RABBIT_CAPABILITIES=compat. If you are running 3.13.0 or newer you must disable no_sort with the setting RABBIT_CAPABILITIES=no_sort.

missing data in graphs

If there is a load balancer between the exporter and the RabbitMQApi, the setting RABBIT_CONNECTION=loadbalancer must be activated. See #131 for details.

build and test

This project uses goreleaser to build and release the project. You can build the project with the following command:

goreleaser build --snapshot

go build will also work, but it will not include the version information, uses cgo, etc.

To run the tests, use the following command:

go test -v ./...

If you have docker installed, you can run the tests with the following command:

go test -v ./... --tags integration

This will start a rabbitmq container and run the tests against it.

Metrics

The metrics are documented in the metrics.md file.

rabbitmq_exporter's Issues

Can we convert this into a Cloud-Foundry app ?

Hello!
I need to tunnel to the rabbitmq host to get to the port: 15672. Is there any way where I can tunnel? Or converting this app to a CF app

Add metrics for total number of messages

It would be very useful to also expose total number of messages in the cluster from overview page:
:15672/api/overview

"queue_totals":{"messages_ready":2810,"messages_ready_details":{"rate":10.8},"messages_unacknowledged":25,"messages_unacknowledged_details":{"rate":0.4},"messages":2835,"messages_details":{"rate":11.2}}

Since our team only care if the total number of messages in the cluster is not bigger than a certain number, this is highly desirable for us.

fix unstable tests

These two tests are failing sometimes:
TestQueueCount/Add_message_with_timestamp (5.02s)
TestQueueCount/Queue_policy (5.04s)

[Improvement] - Additional RabbitMQ metrics

There are several queueinfoitems that are currently not available as metric. https://www.rabbitmq.com/man/rabbitmqctl.1.man.html

• queue_durable
• queue_auto_delete
• queue_arguments
• queue_policy
• queue_exclusive
• queue_exclusive_consumer_tag
• queue_head_message_timestamp
• queue_state

Personally I was specifically looking for head_message_timestamp. Monitoring this timestamp enables me to verify there is no excessive delay in the system.
This items returns a timestamp, but for comparison in prometheus, it would be way easier if the return value was the same as time() (returns the number of seconds since January 1, 1970 UTC). Allowing for example: (time() - rabbitmq_queue_head_message_timestamp{queue="a.b",vhost="/c"}) > 120

buildinfo: branch wrong in docker build

Branchname is "HEAD" for container builds. Real branch name should be used.

rabbitmq_exporter_build_info{branch="HEAD",builddate="20161006-19:00:13",instance="192.168.17.5:9090",job="rabbitmq",revision="4b71698d906395e7cc3a97e9be93a67b250b8c25",version="0.15.0"}

Change default port to some unique value

9090 is already used by Prometheus. According to the documentation exporters should use register unique port number on the wiki and use it.

endpoint metrics are exported with value 0 if scraping failed

discovered during #56

Node endpoint scraping failed. Gauge metrics were exported with value 0.

• These metrics should be skipped during export.
• new metric to indicate failing endpoints

Include exchange name in queue metrics

Is this possible? It'd make it much easier to filter queues per exchange.

Config does not allow for non-HTTPS RABBIT_URL setting

The regex match in the config code only looks for a HTTP URL.

Exchange Stats

Would it be possible to add in exchange stats as gauge counters to this as well?

eg: message_stats: {
publish: 0,
publish_details: {
rate: 0
},
publish_in: 5696,
publish_in_details: {
rate: 0
},
publish_out: 5696,
publish_out_details: {
rate: 0
},
ack: 0,
ack_details: {
rate: 0
},
deliver_get: 0,
deliver_get_details: {
rate: 0
},
confirm: 0,
confirm_details: {
rate: 0
},
return_unroutable: 0,
return_unroutable_details: {
rate: 0
},
redeliver: 0,
redeliver_details: {
rate: 0
}
},

I doubt the rate would be needed, since it can be calculated, but I would definitely find those stats useful as well.

Issues with installing rabbitmq exporter via mesos marathon

Hi,
I am facing issues with deploying rabbitmq exporter via marathon.
I have deployed rabbitmq via marathon as well.
The below command works fine when deployed directly from the server.
docker run -d --net=container:my-rabbit kbudde/rabbitmq-exporter

But in Marathon the network settings allows only bridged,host and user settings and there is no option for container network.
--net=container:my-rabbit
Also, since the rabbitmq is deployed via marathon the containerID and Container Name will change after every restart so this will also create problem with container network mapping.

Is there a way to run this using bridged mode.
https://docs.docker.com/engine/reference/run/#network-settings
https://docs.mesosphere.com/1.8/usage/managing-services/service-ports/

Regards
Dinesh KR

Question: How can I do the default packagaing of this exporter? I don't want to use the Docker.

Hello Team,

Can anyone guide me on how to build this exporter without docker?

Thanks!

API endpoints not working

I am in the process of setting up a grafana dashboard to monitor our rabbitmq queues and came across this exporter.

I have been able to setup the rabbitmq_exporter itself and can successfully see metrics on /metrics but I am having issues with connecting my grafana instance to it.

I am just trying to set it up like a normal prometheus data source in grafana but it looks like the /api endpoints are not working for me.

Any requests to an api end point such as
http://localhost:9090/api/v1/query_range?query=prometheus_evaluator_duration_seconds%7Bquantile!%3D%220.01%22%2C%20quantile!%3D%220.05%22%7D&start=1502797539&end=1502797839&step=2

redirects me to the landing page

Is this a known issue or have I misinterpreted this project?

Metrics for partitions

Is it possible to add metrics for partitions? It is very useful to detect if the cluster is healthy or a split brains happened.
it is available on http://localhost:15672/api/nodes

{partitions,[{rabbit@smacmullen,[hare@smacmullen]},
 =>               {hare@smacmullen,[rabbit@smacmullen]}]}]```
A healthy cluster has no partitions 

{"partitions":[] ...

So I guess it can be a global metric with the value of 0 or more.

[question] disk_free_alarm/mem_alarm possible values?

What's is the possible disk_free_alarm and mem_alarm values? 0 or 1?
And why is it a counter? IMO it should be a gauge.

cant make RABBIT_CAPABILITIES work

I'm running exporter on kubernetes. everything work fine exept RABBIT_CAPABILITIES.

        env:
        - name: RABBIT_USER
          value: user
        - name: RABBIT_PASSWORD
          value: password
        - name: INCLUDE_QUEUES
          value: ".*"
        - name: RABBIT_CAPABILITIES
          value: "vhost"

its my config file.
and its log in kubernetes:

time="2017-09-06T20:05:16Z" level=info msg="Starting RabbitMQ exporter" BRANCH=master BUILD_DATE="20170904-20:48:49" OUTPUT_FORMAT=TTY PUBLISH_PORT=9090 RABBIT_CAPABILITIES= RABBIT_URL="http://localhost:15672" RABBIT_USER=user REVISION=532c8d61263b1f2f66a74221be4815d5509ca827 VERSION=0.20.0

rabbit version: RabbitMQ 3.6.11 on Erlang 19.2.1

Not a Issue, But a request if there exists some generic Grafana Dashboard :)

Kudos all, just deployed an exporter hanging on a tmux session scrapping the cluster details.

How are you guys doing the graphs. Learning about the rabbit is still a advent, wanna start early.
🤔 if some us have(could share) some cool dashing boards on 🐰 rabbit metrics.

Progress so far:

Waiting for some github examples 😄

"go get" fails

# go get github.com/kbudde/rabbitmq_exporter
# cd $GOPATH/src/github.com/kbudde/rabbitmq_exporter; git submodule update --init --recursive
No submodule mapping found in .gitmodules for path 'vendor/github.com/Sirupsen/logrus'
package github.com/kbudde/rabbitmq_exporter: exit status 1

Metrics Partition - Returns 0 In A Partitioned Network

RabbitMq Version: RabbitMQ 3.6.10, Erlang 19.2.1
Version: v0.23.0

Issue

My RabbitMq cluster is currently partitioned, each of the 4 nodes is in a partition. RabbitMQ management UI, rabbitmqctl and the HTTP API all indicate a partitioned cluster, but the rabbitmq_exporter returns rabbitmq_partitions as 0.

RabbitMQ management UI

Rabbitmqctl:

root@ribbit-0:/# rabbitmqctl cluster_status
Cluster status of node 'rabbit@ribbit-0'
[{nodes,[{disc,['rabbit@ribbit-0','rabbit@ribbit-1','rabbit@ribbit-2',
                'rabbit@ribbit-3','rabbit@ribbit-4']}]},
 {running_nodes,['rabbit@ribbit-0']},
 {cluster_name,<<"[email protected]">>},
 {partitions,[{'rabbit@ribbit-0',['rabbit@ribbit-1','rabbit@ribbit-2',
                                  'rabbit@ribbit-3','rabbit@ribbit-4']}]},
 {alarms,[{'rabbit@ribbit-0',[]}]}]

HTTP API

$ root@tdr:/home/user# curl -i http://rmq.k8s.cluster-points.com/api/nodes/rabbit@ribbit-2
{"partitions":["rabbit@ribbit-0","rabbit@ribbit-1","rabbit@ribbit-3","rabbit@ribbit-4"],"os_pid":"411","fd_total":1048576,
...

Prometheus console

It appears that the exporter is looking up the /api/overview API to determine partition count, but this api does not expose partition info. Rather, partition info is available on the /api/nodes endpoint.

Node statistics

Per-node statistics (memory, disk space, etc) would be a useful addition to this. I believe this information comes from hitting /api/nodes

Provide queue metrics counter for missing api data

Background:
Alertrule:
rabbitmq_queue_messages_ready >10 AND delta(rabbitmq_queue_messages_delivered_total[1m])<1
It should alert if there are >10 messages waiting and no message was processed within the last minute.

But it will just alert if at least one message was delivered since the queue was created

The exporter will not export a "rabbitmq_queue_messages_delivered_total" for this queue as the rabbitmq api is not providing the field for the 0 value. This could be surprising.

Improvement
Set missing counter to 0 and export them

metric `rabbitmq_running` changed in 0.25.1

0.24.0 had a metric per node (missing node label) - use to have one for each node in the cluster:

{
  "xhrStatus": "complete",
  "request": {
    "method": "GET",
    "url": "api/datasources/proxy/4/api/v1/query_range?query=rabbitmq_running&start=1512208710&end=1512212310&step=30"
  },
  "response": {
    "status": "success",
    "data": {
      "resultType": "matrix",
      "result": [
        {
          "metric": {
            "__name__": "rabbitmq_running",
            "app": "rabbitmq-prometheus-exporter",
            "instance": "10.1.120.16:9090",
            "job": "kubernetes-pods",
            "kubernetes_namespace": "default",
            "kubernetes_node_name": "ip-10-0-9-180.eu-west-1.compute.internal",
            "kubernetes_pod_ip": "10.1.120.16",
            "kubernetes_pod_name": "rabbitmq-prometheus-exporter-29248363-94f0k",
            "monitoring": "prometheus",
            "pod_template_hash": "29248363"
          },
          "values": [
            [
              1512211920,
              "1"
            ],
          ]
        },
        {
          "metric": {
            "__name__": "rabbitmq_running",
            "app": "rabbitmq-prometheus-exporter",
            "instance": "10.1.152.15:9090",
            "job": "kubernetes-pods",
            "kubernetes_namespace": "default",
            "kubernetes_node_name": "ip-10-0-7-170.eu-west-1.compute.internal",
            "kubernetes_pod_ip": "10.1.152.15",
            "kubernetes_pod_name": "rabbitmq-prometheus-exporter-1210648567-grl66",
            "monitoring": "prometheus",
            "node": "[email protected]",
            "pod_template_hash": "1210648567"
          },
... ,
        {
          "metric": {
            "__name__": "rabbitmq_running",
            "app": "rabbitmq-prometheus-exporter",
            "instance": "10.1.152.15:9090",
            "job": "kubernetes-pods",
            "kubernetes_namespace": "default",
            "kubernetes_node_name": "ip-10-0-7-170.eu-west-1.compute.internal",
            "kubernetes_pod_ip": "10.1.152.15",
            "kubernetes_pod_name": "rabbitmq-prometheus-exporter-1210648567-grl66",
            "monitoring": "prometheus",
            "node": "[email protected]",
            "pod_template_hash": "1210648567"
          },
          "values": [
          ]
        },
        {
          "metric": {
...
}

build 0.25.0 docker image missing

Probably caused by the failed build:
https://travis-ci.org/kbudde/rabbitmq_exporter/builds/307274283

How to monitor the particular Queue through Rabbitmq Exporter

Hi Team,

I have installed Rabbitmq Exporter as given on the github website but all the metrics are showing as Value 0 even after the RabbitMQ is already installed on the server. Also I want to monitor the no. of messages in particular queue(postupload.queue). How can I achieve this through the given rabbitmq exporter and prometheus.

Awaiting for your immediate response.

Regards,
Mobin Ansari

Update release workflow

The workflow needs to be adjusted. Currently it is not possible to create prerelease versions without affecting docker.

It should be possible to create a (pre)release in github and in docker for testing before creating a final stable release.

#56 should not happen again

Possible Memory Leak

I've really only been using a static build in a Docker container (built with cgo and Dockerfile FROM scratch) and it seems to have a memory leak - I left it running for a week or so and it climbed to ~700MB of RAM usage (10x as much as RabbitMQ itself; Prometheus set to a 15s scrape interval). Sadly I'm not familiar enough with go to fix it myself and create a pull request.

I'd also like to suggest some better log output than just this (TTY):
time="2016-07-01T05:11:34Z" level=info msg="Metrics updated successfully."

Maybe time taken for the scrape or just periodically reporting how often it was scraped in the last hour along with how long scrapes took min/max/avg or only reporting on errors.

Beside that: thank you for making this :)

Missing metric: consumer utilization

Although in the documentation it states that the metric "queue_consumer_utilisation" is available as a gauge, the exporter is not providing it.

Should I enable something or configure in Rabbit?

Thanks

grafana error: a.data.data is undefined

After executing
$ docker run -d -e RABBITMQ_NODENAME=my-rabbit --name my-rabbit -p 15672:15672 -p 9090:9090 rabbitmq:3-management

and

$ docker run -d --net=container:my-rabbit kbudde/rabbitmq-exporter

metrics is availible at http://localhost:9090/metrics
but when importing the dashboard in grafana, It shows error that a.data.data is undefined .

Please help.

RABBIT_URL behavior

The logic around the 'RABBIT_URL' flag assumes that all URLs will start with a prefix matching https?://. When I did not include this prefix (i.e., was doing internal routing and did not bother to set the protocol), the exporter silently set the URL to be the default http://localhost:15672. There are plenty of use cases like mine, where the protocol prefix is unnecessary; whether or not you want to keep this behavior, it would be nice to have some kind of error message generated and also some documentation.

Thanks!

Msg : Metrics updated

Hi @kbudde ,

I have successfully deployed a container in my environmnet for rabbitmq-exporter . The pod is also running , but am getting the following message :

time="2018-03-06T06:56:28-05:00" level=info msg="Metrics updated" duration=4.424415629s

And also am not able to view the metrics , as am getting the error :

504 Gateway Time-out

Hope this is similar to the issue #6 .
Please note that , I am using the latest version and exposed port 9419

Could you please help me to solve this.

With Regards,
Aparna Reji

change labels "for consistency and correctness"

promtool output:

rabbitmq_queue_disk_reads counter metrics should have "_total" suffix
rabbitmq_queue_disk_writes counter metrics should have "_total" suffix
http_request_duration_microseconds use base unit "seconds" instead of "microseconds"
rabbitmq_fd_total non-counter metrics should not have "_total" suffix
rabbitmq_queue_messages_ready_total non-counter metrics should not have "_total" suffix
rabbitmq_queue_messages_total non-counter metrics should not have "_total" suffix
rabbitmq_queue_messages_unacknowledged_total non-counter metrics should not have "_total" suffix
rabbitmq_sockets_total non-counter metrics should not have "_total" suffix

hi, in what frequency it pulls data? thanks

Some of queue metrics always return 0

Tested with:

BRANCH=heads/v0.26.0
BUILD_DATE="20180226-21:46:39" 
REVISION=b3d87239f3ad9e6dbdb5fd0cbf689eef056996eb 
VERSION=0.26.0

rabbitmq_queue_messages_delivered_total{durable="true",policy="Lazy",queue="mb_xxx.message_bus",vhost="/"} 2.3743397e+07
rabbitmq_queue_messages_delivered_total{durable="true",policy="Lazy",queue="message_bus_xx-ca1",vhost="/"} 0
rabbitmq_queue_messages_delivered_total{durable="true",policy="Lazy",queue="message_bus_xx-ca2",vhost="/"} 0
rabbitmq_queue_messages_delivered_total{durable="true",policy="Lazy",queue="message_bus_xx-ca3",vhost="/"} 0

Show users making connections

/api/connections gives user details. Can we have user based labels to connections?

{u'auth_mechanism': u'PLAIN',
u'channel_max': 65535,
u'channels': 1,
u'client_properties': {u'copyright': u'Copyright (c) 2007-2014 VMWare Inc, Tony Garnock-Jones, and Alan Antonuk.',
u'information': u'See https://github.com/alanxz/rabbitmq-c',
u'platform': u'linux-gnu',
u'product': u'rabbitmq-c',
u'version': u'0.5.3-pre'},
u'connected_at': 1501868641834,
u'frame_max': 131072,
u'garbage_collection': {u'fullsweep_after': 65535,
u'min_bin_vheap_size': 46422,
u'min_heap_size': 233,
u'minor_gcs': 3},
u'host': u'172.31.15.10',
u'name': u'172.31.0.130:32769 -> 172.31.15.10:5672',
u'node': u'rabbit@rmq-cluster-node-04',
u'peer_cert_issuer': None,
u'peer_cert_subject': None,
u'peer_cert_validity': None,
u'peer_host': u'172.31.0.130',
u'peer_port': 32769,
u'port': 5672,
u'protocol': u'AMQP 0-9-1',
u'recv_cnt': 22708,
u'recv_oct': 8905713,
u'recv_oct_details': {u'rate': 169.6},
u'reductions': 6257210,
u'reductions_details': {u'rate': 148.8},
u'send_cnt': 6,
u'send_oct': 573,
u'send_oct_details': {u'rate': 0.0},
u'send_pend': 0,
u'ssl': False,
u'ssl_cipher': None,
u'ssl_hash': None,
u'ssl_key_exchange': None,
u'ssl_protocol': None,
u'state': u'running',
u'timeout': 0,
u'type': u'network',
u'user': u'rmq_oms',
u'vhost': u'/'}

Also will like to add host, name , node to it.

How to monitor queue depth?

Hi, I've setup the exporter using docker, and testing prometheus in my machine. Everything seems to be hooked up correctly. /metrics endpoint is fine, logs have no errors, prometheus is getting all sorts of metrics as expected.

For example, I have 3 celery queues. Looking at the rabbitmq_queue_messages_delivered_total{durable="true"}graph, the lines are increasing as expected.

The problem is rabbitmq_queue_messages{durable="true"} is a flat line at 0 for all queues.

I would expect to see it going up and down as the queue is getting new tasks and processing them out to celery. Am I assuming wrong?

exporter failed to detect partitions.

Hi, we've been experiencing a network partition issue. However, the exporter failed to detect the partitions. The Gauge 'partitions' remains 0 the whole time.

We are running a 3-nodes rabbitMQ cluster. And the partitions happened today.
RabbitMQ 3.6.2, Erlang 18.3
here's the cluster_status

rabbitmqctl cluster_status
Cluster status of node '[email protected]' ...
[{nodes,
     [{disc,
          ['[email protected]',
           '[email protected]',
           '[email protected]']}]},
 {running_nodes,
     ['[email protected]',
      '[email protected]',
      '[email protected]']},
 {cluster_name,<<"[email protected]">>},
 {partitions,
     [{'[email protected]',
          ['[email protected]']}]},
 {alarms,
     [{'[email protected]',[]},
      {'[email protected]',[]},
      {'[email protected]',[]}]}]

And here's the API result. on /api/nodes (irrelevant keys removed)

[
  {
    "partitions": [],
    "name": "[email protected]",
    "running": true
  },
  {
    "partitions": [],
    "name": "[email protected]",
    "running": true
  },
  {
    "partitions": [
      "[email protected]"
    ],
    "name": "[email protected]",
    "running": true
  }
]

Note that there is a partitions info on machine 10-0-1-205.

Need ability to configure TLS to skip certificate validation, or use custom CA

I have a private CA that I use to sign certificates for RabbitMQ server and client use. The rabbitmq_exporter needs an option either to ignore certificate validation or to use a custom CA.

The release link is broken

https://github.com/kbudde/rabbitmq_exporter/blob/master/release is giving 404.

Not an issue - Rabbit MQ Exporter

The exporter here , is it only for rabbit mq deployed through docker or would this pull metrics from rabbit mq software deployed on local.

How to monitor a particular Queue related metrics

I have rabbitmq exporter setup and viewing the metrics in prometheus.
Assume I have 2 queues namely Queue1 and Queue2.
Is there a way to set alerts very specific to a queue

1. if number of messages in Queue2 > 50 ?
2. if number of unacknowledged messages in Queue1 > 20 ?
3. ( number of messages in Queue2 > 100) AND ( unacknowledged messages in Queue1 > 200)
4. number of ready messages in Queue1 >100
   ....

version info is not up2date in released version

version info is still 0.20.0 in latest release.

"invalid character '<' looking for beginning of value"

I've cloned your repo, built the source and configured everything. When I run the exe it gives me the default Go metrics but none of the Rabbit specific stuff.
Lots of error messages saying

"Error while decoding json"
"invalid character '<' looking for beginning of value

The bottom of the response contains this:

We run RabbitMQ 3.5.5

Any idea as to what might be wrong?

CPU consumption on scrape

Hi guys,

Im running the latest release v0.23.0.
The cpu get highly consumed whenever a scrape is done.
We have 28 queues that are being scraped.

Is this something that is know to happen, is there a way to fix it?

Thanks,
Zarko

hash of unhashable type: bert.bintag

Running rabbitmq_exporter 0.26.0 on Centos 7 targetting RabbitMQ v3.7.3, when bert is enabled I get the following exception when accessing the /metrics endpoint:

Mar  6 19:54:01 mq-int-c systemd: Starting Prometheus rabbitmq_exporter...
Mar  6 19:54:01 mq-int-c rabbitmq_exporter: time="2018-03-06T19:54:01Z" level=info msg="Using default certificate pool"
Mar  6 19:54:01 mq-int-c rabbitmq_exporter: time="2018-03-06T19:54:01Z" level=info msg="Starting RabbitMQ exporter" BRANCH=heads/v0.26.0 BUILD_DATE="20180226-21:46:39" REVISION=b3d87239f3ad9e6dbdb5fd0cbf689eef056996eb VERSION=0.26.0
Mar  6 19:54:01 mq-int-c rabbitmq_exporter: time="2018-03-06T19:54:01Z" level=info msg="Active Configuration" OUTPUT_FORMAT=TTY PUBLISH_ADDR= PUBLISH_PORT=9308 RABBIT_CAPABILITIES="no_sort,bert" RABBIT_EXPORTERS="[exchange node overview queue]" RABBIT_URL="http://localhost:15672" RABBIT_USER=prometheus
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: panic: runtime error: hash of unhashable type bert.bintag
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: goroutine 33 [running]:
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert.readMap(0x9b8960, 0xc4201291f0, 0x74, 0x0, 0x0)
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: /go/src/github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert/decode.go:388 +0x17b
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert.readTag(0x9b8960, 0xc4201291f0, 0x6f8100, 0xc420256520, 0x0, 0x0)
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: /go/src/github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert/decode.go:803 +0x12b3
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert.readSmallTuple(0x9b8960, 0xc4201291f0, 0x68, 0x0, 0x0, 0x0)
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: /go/src/github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert/decode.go:224 +0xd3
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert.readTag(0x9b8960, 0xc4201291f0, 0x6f3660, 0xc420259100, 0x0, 0x0)
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: /go/src/github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert/decode.go:791 +0x8bf
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert.readList(0x9b8960, 0xc4201291f0, 0x6c, 0x0, 0x0, 0xc4201e3080, 0x475574)
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: /go/src/github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert/decode.go:307 +0xc9
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert.readTag(0x9b8960, 0xc4201291f0, 0x12, 0xc42025d680, 0x12, 0x12)
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: /go/src/github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert/decode.go:799 +0xa64
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert.readList(0x9b8960, 0xc4201291f0, 0x6c, 0x0, 0x0, 0x9ef001, 0xc4202602d0)
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: /go/src/github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert/decode.go:307 +0xc9
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert.readTag(0x9b8960, 0xc4201291f0, 0x83, 0x0, 0x0, 0xc4201291f0)
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: /go/src/github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert/decode.go:799 +0xa64
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert.DecodeFrom(0x9b8960, 0xc4201291f0, 0x4, 0x0, 0x0, 0x7770b1)
Mar  6 19:54:03 mq-int-c systemd: rabbitmq_exporter.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: /go/src/github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert/decode.go:835 +0xa5
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert.Decode(0xc420272000, 0x84af, 0xfe00, 0xc4201251a0, 0x9ed600, 0x4, 0x4)
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: /go/src/github.com/kbudde/rabbitmq_exporter/vendor/github.com/kbudde/gobert/decode.go:839 +0x6f
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: main.(*rabbitBERTReply).MakeStatsInfo(0xc420258d40, 0x9ed6c0, 0x4, 0x4, 0xc420018010, 0x14, 0xc42001602d)
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: /go/src/github.com/kbudde/rabbitmq_exporter/bertmap.go:23 +0x55
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: main.getStatsInfo(0x77cc40, 0x16, 0xc42001604c, 0xa, 0xc420018010, 0x14, 0xc42001602d, 0x4, 0x0, 0x0, ...)
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: /go/src/github.com/kbudde/rabbitmq_exporter/rabbitClient.go:86 +0x11a
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: main.exporterQueue.Collect(0xc420124de0, 0xc420125020, 0xc4200fa960, 0x1, 0xffffffffffffffff)
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: /go/src/github.com/kbudde/rabbitmq_exporter/exporter_queue.go:73 +0x12c
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: main.(*exporterQueue).Collect(0xc420069860, 0xc4200fa960, 0x0, 0x0)
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: <autogenerated>:1 +0x4f
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: main.(*exporter).Collect(0xc42009fa00, 0xc4200fa960)
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: /go/src/github.com/kbudde/rabbitmq_exporter/exporter.go:68 +0x109
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: github.com/kbudde/rabbitmq_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather.func2(0xc42007cfe0, 0xc4200fa960, 0x9bb9e0, 0xc42009fa00)
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: /go/src/github.com/kbudde/rabbitmq_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:433 +0x61
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: created by github.com/kbudde/rabbitmq_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Gather
Mar  6 19:54:03 mq-int-c rabbitmq_exporter: /go/src/github.com/kbudde/rabbitmq_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:431 +0x2e1
Mar  6 19:54:03 mq-int-c systemd: Unit rabbitmq_exporter.service entered failed state.

You can see the exporter configuration at the top of the log, if there's any specific information you'd like about my rabbitmq configuration, let me know. The same error was encountered with rabbitmq_exporter 0.25.1.

A previous RabbitMQ install running 3.6.12 did not encounter this issue, there may have been change in the data representation output by RabbitMQ (at a guess, not being well-versed in BERT)?

File handle leak

I left this running for a few days on an AWS box with the wrong Rabbit password and it eventually ran out of file handles, depleting 7-8 per minute until it consumed all 1024. (See ls.txt)

See log.txt for stderr output.

Clean up connections missing

The connections was not closed. This might cause problems when running the rabbitmq management with an reverse-proxy.
See https://golang.org/pkg/net/http/ - "The client must close the response body when finished with it."

not able to see the data in prometheus after adding exporter

I have done the setup of rabbitmq and exporter, however nota ble to see the rabbitmq data in prometheus, I can see the data in rabbitmq exporter.

deployment.yaml

`apiVersion: apps/v1beta1
kind: Deployment
metadata:
 name: rezoomex-rabbitmq-deployment
 labels:
  app: queue
spec:
 replicas: 1
 template:
  metadata:
   labels:
    app: rabbitmq
  spec:
   containers:
    - name: rabbitmq
      image: rabbitmq:3.6.11-management-alpine
      env:
        - name: RABBITMQ_DEFAULT_USER
          value: user
        - name: RABBITMQ_DEFAULT_PASS
          value: password
      ports:
        - name: rabbitmq
          containerPort: 5672
          protocol: TCP
        - name: rabbitmq1
          containerPort: 15672
          protocol: TCP
    - name: rabbitmq-exporter  
      image: kbudde/rabbitmq-exporter
      ports:
        - name: exporter 
          containerPort: 9090
      env:
        - name: RABBIT_USER
          value: user
        - name: RABBIT_PASSWORD
          value: password`

---

Service.yaml

`apiVersion: v1
kind: Service
metadata:
  name: rabbitmq
  namespace: default
  labels:
    app: rabbitmq-service
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9090"  
spec:
  type: LoadBalancer
  ports:
  - name: rabbitmq
    port: 5672
    #targetPort: 15672
    protocol: TCP
  - name: rabbitmq1
    port: 15672
    protocol: TCP
  - name: exporter
    port: 9090
    targetPort: exporter
    protocol: TCP
  selector:
    app: rabbitmq`

when I logged into promethuus and serached for 'rabbitmq_queuesTotal' it showed me no data

missing node metrics in 0.25.0

After updating 0.25.0 from 0.24.0 node metrics are returned at 0.

Is there something missing here?

Example:

# exporter 0.25.0
rabbitmq_running{app="rabbitmq-prometheus-exporter",instance="10.1.160.4:9090",job="kubernetes-pods",kubernetes_namespace="default",kubernetes_node_name="ip-10-0-10-191.eu-west-1.compute.internal",kubernetes_pod_ip="10.1.160.4",kubernetes_pod_name="rabbitmq-prometheus-exporter-3897475324-mtlx7",monitoring="prometheus",pod_template_hash="3897475324"}	0

# exporter 0.24.0
rabbitmq_running{app="rabbitmq-prometheus-exporter",instance="10.1.152.15:9090",job="kubernetes-pods",kubernetes_namespace="default",kubernetes_node_name="ip-10-0-7-170.eu-west-1.compute.internal",kubernetes_pod_ip="10.1.152.15",kubernetes_pod_name="rabbitmq-prometheus-exporter-1210648567-grl66",monitoring="prometheus",node="[email protected]",pod_template_hash="1210648567"}	1
rabbitmq_running{app="rabbitmq-prometheus-exporter",instance="10.1.152.15:9090",job="kubernetes-pods",kubernetes_namespace="default",kubernetes_node_name="ip-10-0-7-170.eu-west-1.compute.internal",kubernetes_pod_ip="10.1.152.15",kubernetes_pod_name="rabbitmq-prometheus-exporter-1210648567-grl66",monitoring="prometheus",node="[email protected]",pod_template_hash="1210648567"}	1
rabbitmq_running{app="rabbitmq-prometheus-exporter",instance="10.1.152.15:9090",job="kubernetes-pods",kubernetes_namespace="default",kubernetes_node_name="ip-10-0-7-170.eu-west-1.compute.internal",kubernetes_pod_ip="10.1.152.15",kubernetes_pod_name="rabbitmq-prometheus-exporter-1210648567-grl66",monitoring="prometheus",node="[email protected]",pod_template_hash="1210648567"}	1
rabbitmq_running{app="rabbitmq-prometheus-exporter",instance="10.1.152.15:9090",job="kubernetes-pods",kubernetes_namespace="default",kubernetes_node_name="ip-10-0-7-170.eu-west-1.compute.internal",kubernetes_pod_ip="10.1.152.15",kubernetes_pod_name="rabbitmq-prometheus-exporter-1210648567-grl66",monitoring="prometheus",node="[email protected]",pod_template_hash="1210648567"}	1
rabbitmq_running{app="rabbitmq-prometheus-exporter",instance="10.1.152.15:9090",job="kubernetes-pods",kubernetes_namespace="default",kubernetes_node_name="ip-10-0-7-170.eu-west-1.compute.internal",kubernetes_pod_ip="10.1.152.15",kubernetes_pod_name="rabbitmq-prometheus-exporter-1210648567-grl66",monitoring="prometheus",node="[email protected]",pod_template_hash="1210648567"}	1

undefined: bert.Map

With the latest master branch:

# _/opt/prometheus/rabbitmq_exporter
./bertmap.go:89: undefined: bert.Map
./bertmap.go:90: cannot range over obj (type interface {})

rabbitmq_queue_messages.* not shown in prometheus

i have set this up newly, I am unable to see the metrics for sum by (vhost)(rabbitmq_queue_messages_ready) and all metrics for (vhost)(rabbitmq_queue_messages.*)

Do I need to enable any thing else? or set up any env variables?
I am using latest rabbitmq_exporter-0.20.0.linux-amd64.tar.gz and my rabbitmq is not a docker container