paddycarey / kafka-docker

Dockerised Kafka and ZooKeeper for development and testing use

kafka-docker

This repository provides a Dockerised Kafka instance (based on Confluent Platform 3.0.1/Scala 2.11) for development and testing use.

NOTE: This image is NOT intended for use in production. Horrible things will probably happen if you attempt to do so.

Why?

Running Zookeeper and Kafka and configuring them to work together is a pain. This image runs both Zookeeper and Kafka together in a single container (managed by supervisord), pre-configured to work together out of the box.

Running Kafka

By default Kafka will be accessible on localhost:9092, localhost:9093 and localhost:9193 so long as the port is exposed at runtime:

$ docker run -ti -p 9092:9092 -p 9093:9093 -p 9193:9193 paddycarey/kafka

Kafka comes with a command line client that will take input from a file or from standard input and send it out as messages to the Kafka cluster. By default, each line will be sent as a separate message. You can use this client to test that Kafka is working. Run the producer and then type a few messages into the console to send to the server:

$ kafka-console-producer.sh --broker-list localhost:9092 --topic "test"
This is a message
This is another message

Kafka also has a command line consumer that will dump out messages to standard output.

$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic "test" --from-beginning
This is a message
This is another message

If you have each of the above commands running in a different terminal then you should now be able to type messages into the producer terminal and see them appear in the consumer terminal.

Accessing kafka on something other than localhost

This image is configured so that it can be accessed on localhost:9092 for a PLAIN connection or localhost:9093 for a SSL connection or localhost:9193 for SSL + SASL. If you need to access Kafka using a different hostname you can set the KAFKA_ADVERTISED_HOST environment variable.

$ docker run -ti -e "KAFKA_ADVERTISED_HOST=somehostname" -p 9092:9092 -p 9093:9093 -p 9193:9193 paddycarey/kafka

Or if using Docker compose, your docker-compose.yml might look something like:

kafka:
    image: paddycarey/kafka # or whatever your built
    ports:
        - "9092:9092"
        - "9093:9093"
        - "9193:9193"
    environment:
      KAFKA_ADVERTISED_HOST: kafka

consumer:
    build: .
    links:
      - kafka
    command: "python -u consumer.py"

Persistence

This image stores log data in /var/lib/kafka. You can use a Docker volume to persist this directory beyond the lifetime of a single container.

$ docker run -ti -v `pwd`/.data/:/var/lib/kafka -p 9092:9092 -p 9093:9093 -p 9193:9193 paddycarey/kafka

Building the image

To build the Docker image:

$ docker build -t paddycarey/kafka .

Note: Advanced users can modify the configuration files in config/kafka/ if necessary to change the behaviour of Kafka or Zookeeper.