sthapa123 / ignite-kafka-market-ticks-demo

Using Apache Ignite and Confluent Kafka For Real-Time Analytics of Market Data

Market Ticks Streaming and Analytics With GridGain and Confluent Kafka

Confluent Kafka and GridGain/Ignite Market Ticks Streaming and Processing Demo.

The demo shows how to process market ticks using Confluent Kafka as a streaming engine and GridGain/Ignite as an in-memory database with persistence layer. Confluent and GridGain are hooked up together via GridGain Kafka Connect integration. SQL is a primary language used by Confluent and GridGain for data processing.

GridGain Cluster Setup and Launch

Launching the Demo Cluster

• Download and decompose GridGain Enterprise Edition 8.7.5 or later
• Move {gridgain}/libs/optional/ignite-rest-http folder to {gridgain}/bin/libs
• Navigate to {gridgain}/bin folder and start a server node(s): ./ignite.sh -J-DIGNITE_JETTY_PORT=9080 {demo_project_root}/market_orders/cfg/gridgain-cfg.xml
• Start Bootstrapper class from your IDE to create demo tables and preload initial data. Plus, the cluster will be activated if it was started for the first time. The activation is required for clusters with Ignite Persistence.

Monitoring the Demo Cluster

• Sign up on console.gridgain.com and go to Monitoring screen. Download web-agent and decompose it.
• Navigate to web-agent, open default.properties and update node-uri parameter to node-uri=http://localhost:9080
• Execute ./ignite-web-agent.sh
• Go back to console.gridgain.com and refresh Monitoring screen.
• Now you have the cluster started and can monitor it!

Confluent Platform and GridGain Setup

If Java 9+ is used by default then switch to Java 8 in a command line window to be used for Confluent scripts. Use this command on Mac OS: export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_191.jdk/Contents/Home/"

Install Confluent and GridGain Connect

• Go to {gridgain}/integration/gridgain-kafka-connect/ and execute ./copy-dependencies.sh script from there
• Download and install Confluent Platform 5.2.1 or later following this guide
• Go to {confluent}/share/java and create folder gridgain-sink there
• Copy GridGain Connect into gridgain-sink with the following command: cp -R {gridgain}/integration/gridgain-kafka-connect/lib/ .

Prepare MarketOrder Source Connector

• Build this project's JAR with IntellijIdea or from your favourite IDE.
• Go back to Go to {confluent}/share/java folder and create market-orders folder there
• Copy MarketOrders Source connector into market-orders with the following command: cp -R {demo_project_root}/market_orders/out/artifacts/market_jar/market.jar .

Starting Confluent, Loading Source and Sink

• Go to {confluent}/bin folder and start Confluent CLI with default clusters: ./confluent start
• Update igniteCfg parameter with an actual path to the grdgain-cfg.xml in the file below {demo_project_root}/market_orders/cfg/gridgain-sink.properties
• Load GridGain Sink Connector: ./confluent load gridgain-sink -d {demo_project_root}/market_orders/cfg/gridgain-sink.properties
• Check that the Connector is up and running: ./confluent status gridgain-sink
• Load MarketOrders Source Connector: ./confluent load market-orders -d {demo_project_root}/market_orders/cfg/market-orders-source.properties
• Check that the Source Connector is running: ./confluent status market-orders

Go to Confluent Web Console (http://localhost:9021) and demonstrate the following queries:

select symbol, sum(order_quantity) as total, bid_price from MARKETORDER WINDOW TUMBLING (SIZE 10 seconds) where bid_price>500.0 group by symbol, bid_price having sum(order_quantity) > 10;

SELECT symbol, SUM(bid_price) as mp FROM MarketOrder GROUP BY(symbol);

Real-Time Analytics with GridGain

Events will be generated and arriving in the cluster in real-time. Run sample queries like those below for the demo purpose:

Total spendings per buyer:

SELECT first_name, last_name, SUM(bid_price) as mp FROM MarketOrder as m JOIN Buyer as b ON m.buyer_id = b.id GROUP BY (first_name, last_name) ORDER BY mp DESC;

Cash Distribution (fraction of money spent per symbol):

SELECT symbol, SUM(bid_price) as mp FROM MarketOrder GROUP BY(symbol) ORDER BY mp DESC;

Restart GridGain cluster (activate with WebConsole or the CMD tool if needed) and show that the day is still in the cluster.

Logging and Debugging

Use the command below to see the latest logs: ./confluent log connect -f