flur / data-streaming_kafka_throughput

Data Streaming Kafka Throughput

Docker Setup

• run docker compose up -d

Producers and Consumers Setup

• install jdk
• install sbt (mac, windows)
• run sbt in terminal
• run project consumer or project producer
• then run to run app

to scale producers / consumer open new terminal window and do all instructions above

Plot Results

• install python 3
• init venv python3 venv venv
• activate venv that you just created source venv/bin/activate (for mac)
• install requirements pip3 install -r requirements.txt
• run python3 plot.py to see graphs (you should run producers and consumers before you can draw graphs)

Graphs

• One Producer One Partition One Consumer
• One Producer Two Partition One Consumer
  
• One Producer Two Partition Two Consumer
  
• One Producer Five Partition One Consumer
• One Producer Five Partition Two Consumer
• One Producer Ten Partition One Consumer
• One Producer Ten Partition Two Consumer
• One Producer Ten Partition Ten Consumer
• Two Producer Ten Partition Ten Consumer