ewolff / microservice-kafka

Demo to show how Apache Kafka can be used for communication between microservices

Microservice Kafka Sample

Deutsche Anleitung zum Starten des Beispiels

This is a sample to show how Kafka can be used for the communication between microservices.

The project creates Docker containers.

It uses three microservices:

• Order to create orders. This services sends messages to Kafka. It uses the KafkaTemplate.
• Shipment receives the orders and extract the information needed to ship the items.
• Invoicing receives the messages, too. It extracts all information to send out an invoice. It uses @KafkaListener just like Shipment.

This is done using a topic order. It has five partitions. Shipment and invoicing each have a separate consumer group. So multiple instances of shipment and invoicing can be run. Each instance would get specific events.

Technologies

• Spring Boot
• Spring Kafka
• Apache httpd
• Kafka
• Zookeeper
• Postgres
• Docker Compose to link the containers.

How To Run

See How to run for details.

Once you create an order in the order application, after a while the invoice and the shipment should be shown in the other applications.

Remarks on the Code

The microservices are:

• microservice-kafka-order to create the orders
• microserivce-kafka-shipping for the shipping
• microservice-kafka-invoicing for the invoices

The data of an order is copied - including the data of the customer and the items. So if a customer or item changes in the order system this does not influence existing shipments and invoices. It would be odd if a change to a price would also change existing invoices. Also only the information needed for the shipment and the invoice are copied over to the other systems.

The Order microservice uses Spring's KafkaTemplate to send message while the other two microservices use the annotation @KafkaListener on the methods that should be called if a new record comes in. All records are put in the order topic. It has five partitions to allow for scalability.

For tests an embedded Kafka server is used. A @ClassRule starts it. And a method annotated with @BeforeClass configures Spring Kafka to use the embedded Kafka server.

The orders are serialized as JSON. So the Order object of the order microservice is serialized as a JSON data structure. The other two microservices just read the data they need for shipping and invoicing. So the invoicing microservices reads the Invoiceobject and the delivery microservice the Deliveryobject. This avoids code dependencies between the microservices. Order contains all the data for Invoice as well as Delivery. JSON serialization is flexible. So when an Order is deserialized into Invoice and Delivery just the needed data is read. The additional data is just ignored.

There are three Docker container for the microservices. The other Docker containers are for Apache httpd, Kafka, Zookeeper and Postgres.

Incoming http request are handled by the Apache httpd server. It is available at port 8080 of the Docker host e.g. http://localhost:8080. HTTP requests are forwarded to the microservices. Kafka is used for the communication between the microservices. Kafka needs Zookeeper to coordinate instances. Postgres is used by all microservices to store data. Each microservices uses its own database in the Postgres instance so they are decoupled in that regard.

You can scale the listener with e.g. docker-compose scale shipping=2. The logs (docker logs mskafka_shipping_1) will show which partitions the instances listen to and which records they handle.

You can also start a shell on the Kafka server docker exec -it mskafka_kafka_1 /bin/sh and then take a look at the records in the topic using kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic order --from-beginning.

microservice-kafka's Issues

Oders are not processed

When I create an order it takes quite some time before I get the success message, but it seems the messaging is broken since I don't see any invoice or shipment being created.

If there are any logs I could provide I'd need some help to locate them since I am a newby when it comes to web/backend development.

docker compose fails

when trying out the docker compose section, I get this failure:

Building order
Step 1/4 : FROM openjdk:11.0.2-jre-slim
11.0.2-jre-slim: Pulling from library/openjdk
27833a3ba0a5: Pull complete
16d944e3d00d: Pull complete
6aaf465b8930: Pull complete
0684138f4cb6: Pull complete
67c4e741e688: Pull complete
783178df3a0c: Pull complete
Digest: sha256:da5605d071e9e41803f8b45f41f8c24ad6c26056190a871c8d6471982407d05b
Status: Downloaded newer image for openjdk:11.0.2-jre-slim
---> b7a931ed7d37
Step 2/4 : COPY target/microservice-kafka-order-0.0.1-SNAPSHOT.jar .
ERROR: Service 'order' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder600758216/target/microservice-kafka-order-0.0.1-SNAPSHOT.jar: no such file or directory

Proxy error

After building with maven and the docker build -> docker compose up -d

invoicing and shipping exit with code 1 and the call to localhost:8080 - clicking on order results in:

Proxy Error
The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request

Reason: DNS lookup failure for: order

Apache/2.4.18 (Ubuntu) Server at localhost Port 80

project builds, but tests fail

Hi, I wanted to try out your project, so cloned and tried out the how-to-run section. When the tests run after the build, there is a long period of getting these messages:

2019-09-30 19:51:25.888 WARN 88574 --- [ main] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-1, groupId=test] Connection to node -1 could not be established. Broker may not be available.

When the tests do end, this is the final output:

[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.234 s - in com.ewolff.microservice.order.kafka.OrderKafkaTest
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] OrderServiceTest.lastCreatedIsUpdated » IllegalState Failed to load Applicatio...
[INFO]
[ERROR] Tests run: 5, Failures: 0, Errors: 1, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] microservice-kafka ................................. SUCCESS [ 0.558 s]
[INFO] microservice-kafka-order ........................... FAILURE [02:19 min]
[INFO] microservice-kafka-shipping ........................ SKIPPED
[INFO] microservice-kafka-invoicing ....................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:20 min
[INFO] Finished at: 2019-09-30T19:43:11-07:00
[INFO] Final Memory: 42M/157M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test (default-test) on project microservice-kafka-order: There are test failures.
[ERROR]
[ERROR] Please refer to /Users/dstadler/Documents/code/k8s-mcsvs-java/microservice-kafka/microservice-kafka/microservice-kafka-order/target/surefire-reports for the individual test results.
[ERROR] Please refer to dump files (if any exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :microservice-kafka-order
C02Z65TVLVDQ:microservice-kafka dstadler$

Postgres init-user-db.sh not executed when using docker on Windows

The postgres container stopped prematurely with this error message in the log:
/usr/local/bin/docker-entrypoint.sh: line 170: /docker-entrypoint-initdb.d/init-user-db.sh: cannot execute: required file not found

The message is misleading, because the file really existed. Root cause; Bash cannot process files with CR/LF line endings.

Please add docker/.gitattibutes containing * text eol=lf or advice to set git config --global core.autocrlf false before cloning the repo.

BTW: I enjoyed reading "Microservices Praxisbuch".

returned a non-zero code: 137

I keep getting this error when trying to docker-compose build

Is there any issue with the file /apache/Dockerfile ?

ERROR: Service 'apache' failed to build: The command '/bin/sh -c apt-get install -y -qq apache2 && a2enmod proxy proxy_http proxy_ajp rewrite deflate headers proxy_connect proxy_html lbmethod_byrequests && mkdir /var/lock/apache2 && mkdir /var/run/apache2' returned a non-zero code: 137

Docker version 19.03.12, build 48a66213fe

Race condition of topic creation from KAFKA_CREATE_TOPICS & the Spring apps (?)

I was playing around with the demo, adding a third service and trying to scale the Spring app consumers and encountered the issue of for some reason getting stuck at having one partition, in spite of not modifying the docker-compose.yaml file KAFKA_CREATE_TOPICS from the original

Topic: order    PartitionCount: 1       ReplicationFactor: 1    Configs: 
        Topic: order    Partition: 0    Leader: 1001    Replicas: 1001  Isr: 1001

Though when I ran the app for the first time, I had no problems having the partitions be created correctly (and then scaling the consumers). I did some searching and found this issue, which seems to have the same issue: wurstmeister/kafka-docker#490

However the proposed solution there did not fix the issue for me in and of itself. I then found this topic wurstmeister/kafka-docker#569 which gave me a solution: disabling KAFKA_AUTO_CREATE_TOPICS_ENABLE. This solution seems to have permanently fixed the partition count issue for me.

(disclaimer: first time working with Kafka)

EDIT: I had some bugs in my extra Spring microservice code concerning json serialisation, without any particular startup error logs regarding this. Poking around in the app after the fix #24 I noticed this. Maybe probably that was the cause for allowing the race condition to occur.

Starting mskafka_zookeeper_1 ... error ERROR: for mskafka_zookeeper_1 Cannot start service zookeeper: driver failed programming external connectivity on endpoint mskafka_zookeeper_1 (de37724a560282572b4e7698cc26e584e0709fb4bb1401beef02cda986e894da): Error starting userland proxy: mkdir /port/tcp:0.0.0.0:2181:tcp:172.18.0.3:2181: input/output error ERROR: for zookeeper Cannot start service zookeeper: driver failed programming external connectivity on endpoint mskafka_zookeeper_1 (de37724a560282572b4e7698cc26e584e0709fb4bb1401beef02cda986e894da): Error starting userland proxy: mkdir /port/tcp:0.0.0.0:2181:tcp:172.18.0.3:2181: input/output error ERROR: Encountered errors while bringing up the project.

D:\Raj_Codings\microservice-kafka-master\docker>docker-compose up -d
mskafka_postgres_1 is up-to-date
Starting mskafka_zookeeper_1 ... error

ERROR: for mskafka_zookeeper_1 Cannot start service zookeeper: driver failed programming external connectivity on endpoint mskafka_zookeeper_1 (de37724a560282572b4e7698cc26e584e0709fb4bb1401beef02cda986e894da): Error starting userland proxy: mkdir /port/tcp:0.0.0.0:2181:tcp:172.18.0.3:2181: input/output error

ERROR: for zookeeper Cannot start service zookeeper: driver failed programming external connectivity on endpoint mskafka_zookeeper_1 (de37724a560282572b4e7698cc26e584e0709fb4bb1401beef02cda986e894da): Error starting userland proxy: mkdir /port/tcp:0.0.0.0:2181:tcp:172.18.0.3:2181: input/output error
ERROR: Encountered errors while bringing up the project.

D:\Raj_Codings\microservice-kafka-master\docker>docker-machine create --virtualbox-memory "4096" --driver virtualbox dev
Running pre-create checks...
Error with pre-create check: "VBoxManage not found. Make sure VirtualBox is installed and VBoxManage is in the path"

Invoice and Shipment services don't start due to missing topic ?

tried to run the application and i am able to create / submit an order - however the invoice and shipment services are not running. when checking the docker log files for those services i get the following hint :

2020-07-04 15:22:34.933 INFO 6 --- [ main] o.a.kafka.common.utils.AppInfoParser : Kafka startTimeMs: 1593876154931
2020-07-04 15:22:35.134 WARN 6 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'; nested exception is java.lang.IllegalStateException: Topic(s) [order] is/are not present and missingTopicsFatal is true
2020-07-04 15:22:35.135 WARN 6 --- [ main] o.s.b.f.support.DisposableBeanAdapter : Invocation of destroy method failed on bean with name 'tomcatMetricsBinder': java.lang.NullPointerException
2020-07-04 15:22:35.136 INFO 6 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Shutting down ExecutorService 'applicationTaskExecutor'
2020-07-04 15:22:35.136 INFO 6 --- [ main] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2020-07-04 15:22:35.137 INFO 6 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown initiated...
2020-07-04 15:22:35.144 INFO 6 --- [ main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Shutdown completed.
2020-07-04 15:22:35.146 INFO 6 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2020-07-04 15:22:35.157 INFO 6 --- [ main] ConditionEvaluationReportLoggingListener :

I want to change db from postgresql to mysql.

Is there any way to change postgres db to mysql. I was trying to do so but it's not working, can you help me out for this.

Question: use of `-XX:TieredStopAtLevel=1 -noverify`

Hello. I see that the Docker starts things with jvm flags -XX:TieredStopAtLevel=1 -noverify I can see how that might help with startup time and possibly memory. Seems that folks might copy those settings and I am wondering if they are recommended for production or simply there for fast demos?

Passwords in `.properties` file

Hi @ewolff ,

Our tool detected some instances of passwords in .properties file. Wanted to bring it to your attention:

microservice-kafka/microservice-kafka/microservice-kafka-invoicing/src/test/resources/application-test.properties

Line 4 in 918d538

spring.datasource.password=dbpass

and

microservice-kafka/microservice-kafka/microservice-kafka-invoicing/src/main/resources/application.properties

Line 13 in 918d538

spring.datasource.password=dbpass

and

microservice-kafka/microservice-kafka/microservice-kafka-order/src/test/resources/application-test.properties

Line 6 in 918d538

spring.datasource.password=dbpass

and

microservice-kafka/microservice-kafka/microservice-kafka-order/src/main/resources/application.properties

Line 11 in 918d538

spring.datasource.password=dbpass

and

microservice-kafka/microservice-kafka/microservice-kafka-shipping/src/test/resources/application-test.properties

Line 4 in 918d538

spring.datasource.password=dbpass

and

microservice-kafka/microservice-kafka/microservice-kafka-shipping/src/main/resources/application.properties

Line 13 in 918d538

spring.datasource.password=dbpass

Failed to execute ms-kafka-order

If the error shows,
Fatal error compiling: invalid target release: 10 -> [Help 1] mac
Either:

1. change the pom java properties version from 10 to the version 8 (as my system has java 8)
2. Update Java in local from your system java version to version 10.

How does Kafka Converts Order info to Invoice

Hello,
I was going through the code and trying to understand it. In OrderService class, method fireOrderCreatedEvent has kafkaTemplate.send("order", order.getId() + "created", order); statement, which sends Order object to Kafka messaging system.

private void fireOrderCreatedEvent(Order order) {
		kafkaTemplate.send("order", order.getId() + "created", order);
	}

And in OrderKafkaListener class you have the following method

@KafkaListener(topics = "order")
	public void order(Invoice invoice, Acknowledgment acknowledgment) {
		log.info("Revceived invoice " + invoice.getId());
		invoiceService.generateInvoice(invoice);
		acknowledgment.acknowledge();
	}

How did Order object serialize or converted to Invoice object? I am kind of confused here. Does Kafka implement this? or we need to manually do it here.

mvn package installation should be done via docker multi-stage build

https://docs.docker.com/develop/develop-images/multistage-build/

mvn package build issue

I am getting this issue with the build:

2017-08-23 09:00:25.822 ERROR 11897 --- [           main] o.s.k.support.LoggingProducerListener    : Exception thrown when sending a message with key='null' and payload='{  "id" : 1,  "customer" : {    "customerId" : 1,    "name" : "Wolff",    "firstname" : "Eberhard", ...' to topic order:

org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

I use Linux Mint (ubuntu based distro). Everything is pretty up to date. Maybe a port is in use, or I am missing something? Or is it the 'null' that is a problem?

I tried a fresh build with mvn -X package and redirected the output. Could not see anything obvious. Maybe you can try and rebuild it and see if anything has changed?

Request without post or get

Hi @ewolff ,

We followed your recommendations from this issue and found a few instances of request method without get or post

Source:

microservice-kafka/microservice-kafka/microservice-kafka-shipping/src/main/java/com/ewolff/microservice/shipping/web/ShippingController.java

Line 28 in 918d538

@RequestMapping("/")

and

microservice-kafka/microservice-kafka/microservice-kafka-order/src/main/java/com/ewolff/microservice/order/logic/OrderController.java

Line 45 in 918d538

@RequestMapping("/")

and

microservice-kafka/microservice-kafka/microservice-kafka-invoicing/src/main/java/com/ewolff/microservice/invoicing/web/InvoiceController.java

Line 28 in 918d538

@RequestMapping("/")