ibm-cloud-architecture / eda-rt-inventory-gitops

Real time inventory demo GitOps

See Mkdocs book view to get demonstration script and installation.

This project can let you deploy the real-time inventory demonstration on OpenShift or run it locally on your laptop using docker compose. For OpenShift deployment you can

• Use an existing Event Streams deployed (multi-tenants) in cp4i-eventstreams namesspace, the topic is using a naming convention to avoid conflicting with other topics alreadt defined in this cluster.
• Use this repository to deploy a 3 broker cluster in the rt-inventory-dev namespace

To deploy you can use a pure GitOps approach using OpenShift GitOps or use make to deploy with oc CLI (See the Makefile under the root folder).

Be sure to clone this repository

git clone https://github.com/ibm-cloud-architecture/eda-rt-inventory-gitops.git

Run locally

• Start IBM Event Streams, IBM MQ, Kafka Connector, the Store Simulator App, the Item aggregator App, the Store aggregator App, and KafDrop to get a user interface to Kafka

cd local-demo/kstreams
docker-compose up -d

• Should have following docker containers running

docker ps
# output
IMAGE                                             PORTS                                          NAMES
   cp.icr.io/cp/ibm-eventstreams-kafka:11.0.2                                                    kstreams-addTopics-1
   quay.io/ibmcase/item-aggregator              0.0.0.0:8081->8080/tcp                           item-aggregator
   quay.io/ibmcase/eda-kconnect-cluster-image   8080/tcp, 0.0.0.0:8083->8083/tcp                 kconnect
   obsidiandynamics/kafdrop                     0.0.0.0:9000->9000/tcp                           kafdrop
   quay.io/ibmcase/store-aggregator             0.0.0.0:8082->8080/tcp                           store-aggregator
   cp.icr.io/cp/ibm-eventstreams-kafka:11.0.2   0.0.0.0:9092->9092/tcp, 0.0.0.0:29092->9092/tcp  kafka
   cp.icr.io/cp/ibm-eventstreams-kafka:11.0.2   0.0.0.0:2181->2181/tcp                           zookeeper
   ibmcom/mq                                    0.0.0.0:1414->1414/tcp, 0.0.0.0:9157->9157/tcp, 0.0.0.0:9443->9443/tcp   ibmmq

See the demonstration script explanation specific to running locally.

Deploy yo OpenShift

Using existing Event Streams in multi-tenant

make multi_tenants

The trace for the execution will looks like:

amespace/rt-inventory-dev created
serviceaccount/rt-inv-job-sa created
role.rbac.authorization.k8s.io/secret-mgr created
rolebinding.rbac.authorization.k8s.io/rt-inventory-dev-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/secrets-to-sa created
job.batch/cpsecret created
Now using project "rt-inventory-dev" on server "https://api.rey.coc-ibm.com:6443".
job.batch/cp-ca-secret created
job.batch/cp-tls-usr-secret created
job.batch/cp-scram-usr-secret created
kafkatopic.eventstreams.ibm.com/eda-rt-item.inventory created
kafkatopic.eventstreams.ibm.com/eda-rt-items created
kafkatopic.eventstreams.ibm.com/eda-rt-store.inventory created
kafkauser.eventstreams.ibm.com/eda-scram-user created
kafkauser.eventstreams.ibm.com/eda-tls-user created
configmap/mq-config created
configmap/mq-mqsc-config created
queuemanager.mq.ibm.com/store-mq created
kafkaconnect.eventstreams.ibm.com/eda-connect-cluster created
kafkaconnector.eventstreams.ibm.com/mq-source created
serviceaccount/store-simulator created
rolebinding.rbac.authorization.k8s.io/store-simulator-view created
configmap/store-simulator-cm created
service/store-simulator created
deployment.apps/store-simulator created
route.route.openshift.io/store-simulator created
serviceaccount/store-inventory created
rolebinding.rbac.authorization.k8s.io/store-inventory-view created
configmap/store-inventory-cm created
service/store-aggregator created
deployment.apps/store-aggregator created
route.route.openshift.io/store-aggregator created
serviceaccount/item-inventory created
rolebinding.rbac.authorization.k8s.io/item-inventory-view created
configmap/item-inventory-cm created
service/item-inventory created
deployment.apps/item-inventory created
route.route.openshift.io/item-inventory created

To remove the deployments do:

make clean_multi_tenants

Deploying Event Streams in the demo project: rt-inventory-dev

List of related projects:

• A simulator app to send store event: refarch-eda-store-simulator
• Compute store inventory: refarch-eda-store-inventory
• Compute item inventory cross store: refarch-eda-item-inventory

eda-rt-inventory-gitops's Issues

Fix issue in the interactive query for store and item aggregators

Deploy rt-inventory demo in one command to pre-existing CP4I installation

Event-Streams is in own namespace: multitenant.
MQ broker will be in the demo namespace.
Kafka connector will be in the demo namespace
Each student can have his/her own demo namespace.
The topic name will be pre-fixed by one of std_1_... to std_50_.

Hands-on lab content

Goals of the hands-on lab: Help tech sellers develop quick proof of concepts around event streams. Need to feel confident.

So this mostly presents what we have and how they can leverage such assets to do a simple proof of concept.

Pre-requisites:

• git client, docker on developer local laptop
• OCP access with CP4I installed. -> need to define a unique structure of CP4I deployment so we can define GitOps accordingly.

What we want to present:
Lecture 1: 30mn

• Some business use case/value proposition for EDA
• Technical fit for streaming
• Explain what to look for in terms of opportunities and what to propose as PoC.
  Lab 1: 30 mn
• run event streams demonstration script.
  Lab 2: 30 min
• real-time inventory problem statement, data visibility, streaming
• ask participants to do a system design on how to address such a problem -
  -> Propose the solution for the system design

Lecture 2:

• Present EDA reference architecture and streaming architecture
• Present fit for purpose for the different technologies involved.
• Present the real-time demo

Lab 3:

• Deploy manually the solution to go over each steps or use make to deploy in one click.
• Delete the deployment

Explain the ES cluster configuration dimensions...

• Review event streams listeners and how to connect to the brokers - TLS for encryption between client and broker - mTLS authentication
• Review how to use kafka connect cluster with 1 MQ source and 2 sinks (Elasticsearch and cloud object storage)
• Review Kafka Streams processing and topology
• Review eda-quickstart code template
• Review basic Kafka Stream implementation for different problems

Support schema registry for all the components of the solution

the store simulator has the definition of the avro schema in backend/src/mai/avro folder

docker compose: issue with kafka connect to connect to MQ

Clean dependencies, move to test container in each project, quarkus 2.11.1

• item inventory alignment for test container, quarkus 2.11.1 and dependencies
• store inventory same
• store simulator same

Be able to demonstrate Mirror Maker 2 between to ES clusters

Taking staging and dev clusters, set a MM2 to the staging and replicates items topic.

I could be between cp4i-eventstreams when deployed to CoC OCP clusters.

Demo script will be: active - passive

• we have 2 clusters: dev running in rt-inventory-dev namespace as source
• cp4i-eventstreams or rt-inventory-staging cluster as target
• items, store.inventory, items.inventory topics to replicate
• make mm2 to deploy mm2 cluster and instance configuration

Add use case of dual cluster, MM2 replication and change to the topic partitions

Add 2 partition in source topic via gitops, see the topic in source and target cluster updated and MM2 adding new records to topic.