rama-nallamilli / kafka-prometheus-monitoring

Dockerised example of monitoring Apache Kafka with Prometheus and Grafana

kafka-prometheus-monitoring

Dockerised example of monitoring Apache Kafka with Prometheus and Grafana. This project makes use of the prometheus-jmx-exporter which is configured to extract metrics from Kafka's JMX server. These metrics are then exposed via HTTP GET and polled by Prometheus.

• kafka.yml - Kafka JMX polling configuration
• prometheus.yml - Prometheus metric polling configuration

Pre-Requisites

• install Docker and Docker Compose - https://docs.docker.com/
• set KAFKA_ADVERTISED_HOST_NAME in docker-compose.yml to match your docker host IP. (Note: Do not use localhost or 127.0.0.1 as the host ip if you want to run multiple brokers). See wurstmeister's docker-kafka for more information on configuring Kafka - https://github.com/wurstmeister/kafka-docker

Usage

docker-compose up

• View Prometheus UI - http://$DOCKER_HOST_IP:9090
• Grafana UI - http://$DOCKER_HOST_IP:3000 (admin:admin)
• Kafka metrics - http://$DOCKER_HOST_IP:8080/metrics

Sending Kafka messages

In order for the Kafka broker to expose JMX topic metrics you must send some messages to the topics.

cat kafka-messages | docker run -i -a stdin wurstmeister/kafka /opt/kafka/bin/kafka-console-producer.sh --broker-list $DOCKER_HOST_IP:9092 --topic customer
cat kafka-messages | docker run -i -a stdin wurstmeister/kafka /opt/kafka/bin/kafka-console-producer.sh --broker-list $DOCKER_HOST_IP:9092 --topic audit

Viewing Prometheus Metrics

The kafka metrics are pulled into Prometheus via the JMX exporter. These can be viewed in Prometheus by navigating to http://$DOCKER_HOST_IP:9090/graph, enter a metric name to view the graphs.

Viewing Graphs in Grafana

Grafana can be used to build a more meaningful dashboard of the metrics in Prometheus, navigate to Grafana on http://$DOCKER_HOST_IP:3000 (admin:admin). An example dashboard is available to import in dashboards/Kafka.json.

kafka-prometheus-monitoring's Issues

Enhance scrape rules

The prometheus/jmx_exporter base repo seems to noq include example configs for kafka both pre.0.8.2 and post that version.

The rules use regular expressions to potentially include more rules.

Are there downsides to incorporating these here?

Poll from multiple Kafka broker instances

Hello, nice job with this.
How would you use this to poll mulitple brokers in the same compose, besides running multiple intances of jmx_exporter?

Thanks

No license

Could you add an explicit license please?

graf-db service not starting

Hi, I am using your project and when starting the services via docker compose the graf-db service is exiting and docker logs graf-db-container-name doesn't show any output. Also if you can elaborate on the use of graf-db.

My Platform details
OS-Centos 7
Docker version 17.09.0-ce

Thanks,

Busted tarball?

On docker-compose up

Step 2/5 : RUN mkdir -p /opt /var/sky/zookeeper/data /var/sky/zookeeper/conf     && wget -q -O - http://apache.mirrors.pair.com/zookeeper/zookeeper-3.4.8/zookeeper-3.4.8.tar.gz | tar -xzf - -C /opt     && mv /opt/zookeeper-3.4.8 /opt/zookeeper
 ---> Running in d71ae1a366bf

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Might need to modify the URL or the tar command...

Adding consumer tracking

Is there any chance for you adding consumer rules/patterns?

Docker image of jmx_exporter

Hi there,
first thank you for implementing this solution. I try to connect prometheus and the kafka via the jmx_exporter on an openshift cluster.
When using the image for he jmx exporter compiled by the the Dockerfiles you provided, I get the following error message in the prometheus web ui:

Any idea how I can resolve this error?
Thanks!

Compatibility with grafana 6.x ?

Is there any updates for working with grafana 6, I've tried it and everything seems to be incorrect, no data available

Connection refused

Bug

kafka connection refused

Error Log

Feb 27, 2019 10:41:24 AM io.prometheus.jmx.JmxCollector collect
SEVERE: JMX scrape failed: java.rmi.ConnectException: Connection refused to host: 0.0.0.0; nested exception is: 
	java.net.ConnectException: Connection refused (Connection refused)
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
	at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
	at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:130)
	at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:227)
	at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:179)
	at com.sun.proxy.$Proxy3.newClient(Unknown Source)
	at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2430)
	at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:308)
	at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
	at io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:94)
	at io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:320)
	at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:73)
	at io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.<init>(CollectorRegistry.java:65)
	at io.prometheus.client.CollectorRegistry.metricFamilySamples(CollectorRegistry.java:56)
	at io.prometheus.client.exporter.MetricsServlet.doGet(MetricsServlet.java:41)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:735)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:648)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
	at org.eclipse.jetty.server.Server.handle(Server.java:365)
	at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
	at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
	at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
	at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)
	at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:204)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at java.net.Socket.connect(Socket.java:538)
	at java.net.Socket.<init>(Socket.java:434)
	at java.net.Socket.<init>(Socket.java:211)
	at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
	at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:148)
	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
	... 36 more

Kafka Docker Configuration

    kafka:
        image: confluentinc/cp-kafka:5.1.0
        container_name: kafka
        hostname: kafka
        ports:
            - "9092:9092"
            - "49998:49998"
        environment:
            KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
            KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka:9092'
            JMX_PORT: 49998

kafka-jmx-exporter

kafka-jmx-exporter:
    build: ./prometheus-jmx-exporter
    container_name: kafka-jmx-exporter
    ports:
      - "8091:8080"
    link
    - kafka
    environment:
      - JMX_PORT=49998
      - JMX_HOST=kafka
      - HTTP_PORT=8080
      - JMX_EXPORTER_CONFIG_FILE=kafka.yml
    networks:
        ray_network:

docker-compose up results with an error

When I run the docker-compose up command it gives an error.

Step 11/14 : RUN apk add --no-cache bash curl jq docker  && mkdir /opt  && chmod a+x /tmp/*.sh  && mv /tmp/start-kafka.sh /tmp/broker-list.sh /tmp/create-topics.sh /tmp/versions.sh /usr/bin  && sync && /tmp/download-kafka.sh  && tar xfz /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz -C /opt  && rm /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz  && ln -s /opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION} /opt/kafka  && rm /tmp/*  && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk  && apk add --no-cache --allow-untrusted glibc-${GLIBC_VERSION}.apk  && rm glibc-${GLIBC_VERSION}.apk
 ---> Running in 5b96cf36c572
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.8/community/x86_64/APKINDEX.tar.gz
(1/19) Installing ncurses-terminfo-base (6.1_p20180818-r1)
(2/19) Installing ncurses-terminfo (6.1_p20180818-r1)
(3/19) Installing ncurses-libs (6.1_p20180818-r1)
(4/19) Installing readline (7.0.003-r0)
(5/19) Installing bash (4.4.19-r1)
Executing bash-4.4.19-r1.post-install
(6/19) Installing nghttp2-libs (1.32.0-r0)
(7/19) Installing libssh2 (1.8.0-r3)
(8/19) Installing libcurl (7.61.1-r1)
(9/19) Installing curl (7.61.1-r1)
(10/19) Installing libmnl (1.0.4-r0)
(11/19) Installing jansson (2.11-r0)
(12/19) Installing libnftnl-libs (1.1.1-r0)
(13/19) Installing iptables (1.6.2-r0)
(14/19) Installing device-mapper-libs (2.02.178-r0)
(15/19) Installing libltdl (2.4.6-r5)
(16/19) Installing libseccomp (2.3.3-r1)
(17/19) Installing docker (18.06.1-r0)
Executing docker-18.06.1-r0.pre-install
(18/19) Installing oniguruma (6.8.2-r0)
(19/19) Installing jq (1.6_rc1-r1)
Executing busybox-1.28.4-r1.trigger
OK: 269 MiB in 71 packages
/bin/sh: illegal option -
ERROR: Service 'kafka' failed to build: The command '/bin/sh -c apk add --no-cache bash curl jq docker  && mkdir /opt  && chmod a+x /tmp/*.sh  && mv /tmp/start-kafka.sh /tmp/broker-list.sh /tmp/create-topics.sh /tmp/versions.sh /usr/bin  && sync && /tmp/download-kafka.sh  && tar xfz /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz -C /opt  && rm /tmp/kafka_${SCALA_VERSION}-${KAFKA_VERSION}.tgz  && ln -s /opt/kafka_${SCALA_VERSION}-${KAFKA_VERSION} /opt/kafka  && rm /tmp/*  && wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk  && apk add --no-cache --allow-untrusted glibc-${GLIBC_VERSION}.apk  && rm glibc-${GLIBC_VERSION}.apk' returned a non-zero code: 2

Message count calculation issue

Current message count is calculated using LogEndOffSet

in Kafka.xml
- pattern : kafka.log<type=Log, name=LogEndOffset, topic=(.+), partition=(.+)><>Value

in dashboard.json

          "targets": [
            {
              "expr": "kafka_log_log_value{topic=\"customer\"}",
              "intervalFactor": 2,
              "legendFormat": "",
              "metric": "",
              "refId": "A",
              "step": 20
            }

However, to correctly calculate number of message, it should use LogEndOffset - LogStartOffset .