openshift-kafka's Introduction

"Apache Kafka for OpenShift"

Run Apache Kafka and Apache ZooKeeper on OpenShift v3.

This was developed for demonstration purposes. Use in environments where you do not need persistence in the face of failures or configuration tuning for scale.

The architecture is as follows:

1 pod containing a container for Apache Kafka and Apache ZooKeeper
1 service to access the containers

Quick start

Load resources (Templates, DeploymentConfigs, Services)

oc create -f https://raw.githubusercontent.com/mattf/openshift-kafka/master/resources.yaml

Deploy the Apache Kafka + Apache Zookeeper pod

oc new-app apache-kafka

Follow the Apache Kafka Documentation Quick Start

Deploy a debugging container and connect to it

oc run -it --rm kafka-debug --image=mattf/openshift-kafka --command -- bash

Create a topic

bin/kafka-topics.sh --create --zookeeper apache-kafka --replication-factor 1 --partitions 1 --topic test

List topics

bin/kafka-topics.sh --list --zookeeper apache-kafka

Send some messages

bin/kafka-console-producer.sh --broker-list apache-kafka:9092 --topic test <<EOF
foo
bar
baz
EOF

Receive some messages

bin/kafka-console-consumer.sh --bootstrap-server apache-kafka:9092 --topic test --from-beginning

Credits

This is based on work original by Jim Minter

openshift-kafka's People

Contributors

Stargazers

Watchers

openshift-kafka's Issues

came across error from kafka console

i follow your use case to testing the kafka on openshift,
when i create topic on this testing kafka, it report error:

bash-4.2$ bin/kafka-console-producer.sh --broker-list apache-kafka:9092 --topic test
1
[2017-11-06 17:01:13,401] WARN Error while fetching metadata with correlation id 0 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:13,499] WARN Error while fetching metadata with correlation id 1 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:13,607] WARN Error while fetching metadata with correlation id 2 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:13,712] WARN Error while fetching metadata with correlation id 3 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:13,817] WARN Error while fetching metadata with correlation id 4 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:13,923] WARN Error while fetching metadata with correlation id 5 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:14,031] WARN Error while fetching metadata with correlation id 6 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:14,136] WARN Error while fetching metadata with correlation id 7 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:14,242] WARN Error while fetching metadata with correlation id 8 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:14,349] WARN Error while fetching metadata with correlation id 9 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:14,455] WARN Error while fetching metadata with correlation id 10 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:14,561] WARN Error while fetching metadata with correlation id 11 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:14,668] WARN Error while fetching metadata with correlation id 12 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:14,773] WARN Error while fetching metadata with correlation id 13 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:14,879] WARN Error while fetching metadata with correlation id 14 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:14,986] WARN Error while fetching metadata with correlation id 15 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:15,090] WARN Error while fetching metadata with correlation id 16 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:15,196] WARN Error while fetching metadata with correlation id 17 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:15,304] WARN Error while fetching metadata with correlation id 18 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:15,411] WARN Error while fetching metadata with correlation id 19 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:15,519] WARN Error while fetching metadata with correlation id 20 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
[2017-11-06 17:01:15,625] WARN Error while fetching metadata with correlation id 21 : {test=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
^Cbash-4.2$