simplesteph / medium-blog-kafka-udemy

Supporting repository for the blog post at https://medium.com/@stephane.maarek/how-to-use-apache-kafka-to-transform-a-batch-pipeline-into-a-real-time-one-831b48a6ad85

Medium Blog Kafka Udemy

This is the source code supporting the blog post:

https://medium.com/@stephane.maarek/how-to-use-apache-kafka-to-transform-a-batch-pipeline-into-a-real-time-one-831b48a6ad85

Building

All the instructions are in run.sh

# Download Confluent Platform 3.3.0 https://www.confluent.io/download/
# Unzip and add confluent-3.3.0/bin to your PATH

# Download and install Docker for Mac / Windows / Linux and do
docker-compose up -d
# Alternatively start postgres manually on your laptop at port 5432 and username/password = postgres/postgres

# Start the Confluent platform using the Confluent CLI
confluent start

# Create all the topics we're going to use for this demo
kafka-topics --create --topic udemy-reviews --partitions 3 --replication-factor 1 --zookeeper localhost:2181
kafka-topics --create --topic udemy-reviews-valid --partitions 3 --replication-factor 1 --zookeeper localhost:2181
kafka-topics --create --topic udemy-reviews-fraud --partitions 3 --replication-factor 1 --zookeeper localhost:2181
kafka-topics --create --topic long-term-stats --partitions 3 --replication-factor 1 --zookeeper localhost:2181
kafka-topics --create --topic recent-stats --partitions 3 --replication-factor 1 --zookeeper localhost:2181

# Build and package the different project components (make sure you have maven installed)
mvn clean package

Running

All the instructions are in run.sh Sample instructions:

export COURSE_ID=1075642  # Kafka for Beginners Course
java -jar udemy-reviews-producer/target/uber-udemy-reviews-producer-1.0-SNAPSHOT.jar

Video Tutorial:

Learning Kafka

If you want to explore all that Kafka has to offer, you can learn Kafka with my Udemy courses:

• Kafka for Beginners
• Kafka Connect
• Kafka Streams
• Kafka Setup and Administration
• Confluent Schema Registry & REST Proxy
• Kafka Security

medium-blog-kafka-udemy's Issues

Maven build and pacakge doesn't work.

Command: mvn clean package
Complete output:

[INFO] Scanning for projects...
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.github.simplesteph.kafka.streams:udemy-reviews-fraud:jar:1.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-shade-plugin is missing. @ line 142, column 21
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.github.simplesteph.kafka.producer:udemy-reviews-producer:jar:1.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-shade-plugin is missing. @ line 105, column 21
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.github.simplesteph.kafka.streams:udemy-reviews-aggregator:jar:1.0-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-shade-plugin is missing. @ line 118, column 21
[WARNING] 
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING] 
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING] 
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] udemy-reviews-producer                                             [jar]
[INFO] udemy-reviews-fraud                                                [jar]
[INFO] udemy-reviews-aggregator                                           [jar]
[INFO] udemy-reviews-parent-project                                       [pom]
[INFO] 
[INFO] ----< com.github.simplesteph.kafka.producer:udemy-reviews-producer >----
[INFO] Building udemy-reviews-producer 1.0-SNAPSHOT                       [1/4]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- avro-maven-plugin:1.8.2:schema (default) @ udemy-reviews-producer ---
[INFO] 
[INFO] --- avro-maven-plugin:1.8.2:protocol (default) @ udemy-reviews-producer ---
[INFO] 
[INFO] --- avro-maven-plugin:1.8.2:idl-protocol (default) @ udemy-reviews-producer ---
[INFO] 
[INFO] --- build-helper-maven-plugin:3.0.0:add-source (add-source) @ udemy-reviews-producer ---
[INFO] Source directory: /home/sumedh/Documents/Development/Kafka/end-to-end/medium-blog-kafka-udemy-master/udemy-reviews-producer/target/generated-sources/avro added.
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ udemy-reviews-producer ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.6.1:compile (default-compile) @ udemy-reviews-producer ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 9 source files to /home/sumedh/Documents/Development/Kafka/end-to-end/medium-blog-kafka-udemy-master/udemy-reviews-producer/target/classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] No compiler is provided in this environment. Perhaps you are running on a JRE rather than a JDK?
[INFO] 1 error
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for udemy-reviews-parent-project 1.0-SNAPSHOT:
[INFO] 
[INFO] udemy-reviews-producer ............................. FAILURE [  0.949 s]
[INFO] udemy-reviews-fraud ................................ SKIPPED
[INFO] udemy-reviews-aggregator ........................... SKIPPED
[INFO] udemy-reviews-parent-project ....................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  1.041 s
[INFO] Finished at: 2020-09-16T15:50:32+05:30
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile) on project udemy-reviews-producer: Compilation failure
[ERROR] No compiler is provided in this environment. Perhaps you are running on a JRE rather than a JDK?
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

Question: what would happen if you didn't write timestamp extractor

https://github.com/simplesteph/medium-blog-kafka-udemy/blob/master/udemy-reviews-aggregator/src/main/java/com/github/simplesteph/kafka/streams/udemy/aggregator/ReviewTimestampExtractor.java

Now what would happen if we relied on default timestamp extractor? If app would behave wrongly what would exactly be wrongly done?

Reviews producer doesn't fetch the reviews

The application starts, as can be seen the output,

[2020-09-16 16:43:00,181] INFO [Producer clientId=producer-1] Instantiated an idempotent producer. (org.apache.kafka.clients.producer.KafkaProducer)
[2020-09-16 16:43:00,237] INFO Kafka version : 2.1.0 (org.apache.kafka.common.utils.AppInfoParser)
[2020-09-16 16:43:00,237] INFO Kafka commitId : eec43959745f444f (org.apache.kafka.common.utils.AppInfoParser)
[2020-09-16 16:43:00,238] INFO Application started! (ReviewsProducerMain)
[2020-09-16 16:43:00,239] INFO Stuff submit (ReviewsProducerMain)
[2020-09-16 16:43:00,239] INFO Latch await (ReviewsProducerMain)
[2020-09-16 16:43:00,311] INFO [Producer clientId=producer-1] ProducerId set to 2 with epoch 0 (org.apache.kafka.clients.producer.internals.TransactionManager)
[2020-09-16 16:43:00,986] WARN Invalid cookie header: "Set-Cookie: ud_cache_version=1; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,986] WARN Invalid cookie header: "Set-Cookie: ud_cache_release=0028d88b2f47b2a4928d; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,987] WARN Invalid cookie header: "Set-Cookie: ud_user_jwt=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,987] WARN Invalid cookie header: "Set-Cookie: ud_cache_device=None; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,987] WARN Invalid cookie header: "Set-Cookie: ud_cache_modern_browser=0; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,987] WARN Invalid cookie header: "Set-Cookie: exaff=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,987] WARN Invalid cookie header: "Set-Cookie: evi="Sg4="; expires=Fri, 16 Oct 2020 11:13:00 GMT; Max-Age=2592000; Path=/". Invalid 'expires' attribute: Fri, 16 Oct 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,987] WARN Invalid cookie header: "Set-Cookie: ud_firstvisit=2020-09-16T11:13:00.726900+00:00:1kIVMy:zGeZbMtC6immFfVnjmEYugKGQxM; expires=Thu, 16 Sep 2021 11:13:00 GMT; Max-Age=31536000; Path=/". Invalid 'expires' attribute: Thu, 16 Sep 2021 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,987] WARN Invalid cookie header: "Set-Cookie: __udmy_2_v57r=b44a0e4903304111b679e14745e84692; Domain=.udemy.com; expires=Thu, 16 Sep 2021 11:13:00 GMT; Max-Age=31536000; Path=/; Secure". Invalid 'expires' attribute: Thu, 16 Sep 2021 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,988] WARN Invalid cookie header: "Set-Cookie: eva=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,988] WARN Invalid cookie header: "Set-Cookie: ud_cache_brand=INen_US; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,988] WARN Invalid cookie header: "Set-Cookie: inaff=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,988] WARN Invalid cookie header: "Set-Cookie: ud_cache_language=en; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,988] WARN Invalid cookie header: "Set-Cookie: ud_cache_marketplace_country=IN; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,988] WARN Invalid cookie header: "Set-Cookie: __udmy_evid=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,989] WARN Invalid cookie header: "Set-Cookie: ud_rule_vars=eJyFjcsOwiAURH-lYas191L6gG8hIRRvK7GRCLSbpv8uiZq4czmTOXN2lm2cKdPVbD75HKIahbBAQkLTgEDEsesloehFS4PoJFcuhLsnpiq2azb5mPKbNVebSZdeMw4capA1dhWiwkYBXHqUvGlPACVodi6rxRY0h9XdTI52mrwzKazRkdls9HZcPm8hzvbh3Q8U6blS-mccoEXBv8aDHS_n2EbJ:1kIVMy:wjpX1EOv-_GEyfgJq2yDuM_CJxs; expires=Fri, 16 Sep 2022 11:13:00 GMT; Max-Age=63072000; Path=/". Invalid 'expires' attribute: Fri, 16 Sep 2022 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,989] WARN Invalid cookie header: "Set-Cookie: ud_cache_user=""; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,989] WARN Invalid cookie header: "Set-Cookie: udmy_branded_partner=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,989] WARN Invalid cookie header: "Set-Cookie: ud_cache_logged_in=0; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:00,989] WARN Invalid cookie header: "Set-Cookie: ud_cache_price_country=IN; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,478] WARN Invalid cookie header: "Set-Cookie: ud_cache_brand=INen_US; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,478] WARN Invalid cookie header: "Set-Cookie: ud_cache_price_country=IN; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,478] WARN Invalid cookie header: "Set-Cookie: evi="Sg4="; expires=Fri, 16 Oct 2020 11:13:01 GMT; Max-Age=2592000; Path=/". Invalid 'expires' attribute: Fri, 16 Oct 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,478] WARN Invalid cookie header: "Set-Cookie: ud_cache_user=""; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,478] WARN Invalid cookie header: "Set-Cookie: ud_cache_language=en; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,478] WARN Invalid cookie header: "Set-Cookie: ud_firstvisit=2020-09-16T11:13:01.294599+00:00:1kIVMz:EtF4-PS6q0wHvXiqOkKVtpHj6Xs; expires=Thu, 16 Sep 2021 11:13:01 GMT; Max-Age=31536000; Path=/". Invalid 'expires' attribute: Thu, 16 Sep 2021 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,478] WARN Invalid cookie header: "Set-Cookie: ud_cache_release=0028d88b2f47b2a4928d; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: exaff=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: udmy_branded_partner=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: ud_cache_logged_in=0; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: ud_cache_version=1; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: eva=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: ud_user_jwt=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: ud_cache_marketplace_country=IN; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: inaff=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: __udmy_evid=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: __udmy_2_v57r=9f5a4908f5fc47d89dd17de039798b23; Domain=.udemy.com; expires=Thu, 16 Sep 2021 11:13:01 GMT; Max-Age=31536000; Path=/; Secure". Invalid 'expires' attribute: Thu, 16 Sep 2021 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,480] WARN Invalid cookie header: "Set-Cookie: ud_rule_vars="eJyFzUsOwiAUheGtNEy15t5SLLAWEkJ5VKKRCLSTpnu3qTp2eJL_y1lJNXny1Tu9xBJrylIEZnoBPLBg-8Fx4RwOzgMVg-BjR6VN6R49kQ1ZFXmYUnVNs73pmk0I0eqS5my9XkyOZnx4tZeKpDyZZ7SKnPcRYt7Vcaidqd-kgw5aEC1eG0SJVAJeOs4ZFScACfCxx2H2r9mXf5hSZMh-eCPbGyOgR-U=:1kIVMz:GhW8HauXT_CnL8xoXhyTV6A8g4w"; expires=Fri, 16 Sep 2022 11:13:01 GMT; Max-Age=63072000; Path=/". Invalid 'expires' attribute: Fri, 16 Sep 2022 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,480] WARN Invalid cookie header: "Set-Cookie: ud_cache_device=None; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,480] WARN Invalid cookie header: "Set-Cookie: ud_cache_modern_browser=0; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies)
[2020-09-16 16:43:01,480] INFO Fetched 0 reviews (ReviewsFetcherThread)
[2020-09-16 16:43:01,530] INFO Closing (ReviewsFetcherThread)
[2020-09-16 16:43:01,531] INFO Closed (ReviewsFetcherThread)
[2020-09-16 16:43:01,641] INFO Closing Producer (ReviewsAvroProducerThread)
[2020-09-16 16:43:01,642] INFO [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)

, but due to response cookie header mismatch, the reviews are not fetched.

ReviewsFetcherThread is not pulling review for the first while loop iteration

Hi @simplesteph , I am implementing the assignment as you suggested on your Kafka Series courses. I realized that the code at https://github.com/simplesteph/medium-blog-kafka-udemy/blob/master/udemy-reviews-producer/src/main/java/com/github/simplesteph/kafka/producer/udemy/runnable/ReviewsFetcherThread.java#L40 with a while(keepOnRunning) loop needs to run at least once with 0 reviews in order to get the reviews on the next iteration. If you would like I can push my code to your review.

Tests in error when run mvn clean package.

Results :

Tests in error:
UdemyRESTClientTest.twoApiCallsReturnTwoDifferentSetsOfResults:114 » JSON JSON...
UdemyRESTClientTest.apiCallIsSuccessful:35 » JSON JSONObject["id"] not found.

Tests run: 5, Failures: 0, Errors: 2, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] udemy-reviews-producer ............................. FAILURE [ 14.525 s]
[INFO] udemy-reviews-fraud ................................ SKIPPED
[INFO] udemy-reviews-aggregator ........................... SKIPPED
[INFO] udemy-reviews-parent-project ....................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14.999 s
[INFO] Finished at: 2018-07-12T07:51:25+07:00
[INFO] Final Memory: 24M/166M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project udemy-reviews-producer: There are test failures.
[ERROR]
[ERROR] Please refer to /home/knel/Projects/abc/udemy-reviews-producer/target/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
knel@kso:~/Projects/abc$