POC the usage of Kafka with AVRO messages and the schema registry to manage the schemas.
docker-compose up -d
To start kafka and zookeeperdocker-compose exec kafka kafka-topics.sh --create --zookeeper zookeeper --replication-factor 1 --partitions 5 --topic page_visits
docker-compose run --rm java java -jar producer/target/java-producer-1.0-SNAPSHOT-jar-with-dependencies.jar <nb messages> http://schemaregistry:8081
To produce one oo more messages with javadocker-compose run --rm java java -jar consumer/target/java-consumer-1.0-SNAPSHOT-jar-with-dependencies.jar http://schemaregistry:8081
To launch a java consumerdocker-compose run --rm php php src/produce.php <nb messages>
To produce one or more messages with PHPdocker-compose run --rm php php src/consume.php
To launch a PHP consumerdocker-compose run --rm python python src/consume.py
To launch a Python consumerdocker-compose run --rm python python src/consume_raw.py
To launch a Python consumer for raw Kafka messages (without Avro deserialization)docker-compose down --remove-orphans -v
To stop everything
Note: All consumers are created in the same group and consume the same topic. It means that a message won't be consumed twice by these consumers. Note: As the topic is created with 5 partitions, you can't handle more than 5 messages at the same time.
The consumers group feature only works with librdkafka 0.9+ and currently php-rdkafka is incompatible with librdkafka 0.9.2 and librdkafka 0.9.3. You need to force the librdkafka version to 0.9.1 when installing it.
- librdkafka and php-rdkafka seems both to be under active development and pretty up-to-date with current kafka features
- The AVRO library does not seems to be well maintained but the project does not evolve that much so it does not seems to an issue.