anjireddy1804 / confluentpoc

This is a poc to set up confluent kafka platform

This enviroment set up instructions are taken from Confluent .

Deploy Secure Confluent Platform

In this workflow scenario, you'll set up secure Confluent Platform clusters with mTLS authentication.

To complete this scenario, you'll follow these steps:

1. Set the current working directory.
2. Deploy Confluent For Kubernetes.
3. Generate certificates
4. Deploy configuration secrets.
5. Deploy Confluent Platform.
6. Tear down Confluent Platform.

Set the current workiing directory

Clone the git repo and set the current working directory

git clone <git url>

export WORKING_DIR=<working directory>/confluentpoc

Deploy Confluent for Kubernetes

1. Create confluent namespace

   kubectl create namespace confluent
   

2. Set up the Helm Chart:

   helm repo add confluentinc https://packages.confluent.io/helm
   

3. Install Confluent For Kubernetes using Helm:

   helm upgrade --install operator confluentinc/confluent-for-kubernetes
   

4. Check that the Confluent For Kubernetes pod comes up and is running:

   kubectl get pods
   

Install cfssl if required

1. Install cfssl program

VERSION=$(curl --silent "https://api.github.com/repos/cloudflare/cfssl/releases/latest" | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/')
VNUMBER=${VERSION#"v"}
wget https://github.com/cloudflare/cfssl/releases/download/${VERSION}/cfssl_${VNUMBER}_linux_amd64 -O cfssl
chmod +x cfssl
sudo mv cfssl /usr/local/bin

1. Install cfssljson program

VERSION=$(curl --silent "https://api.github.com/repos/cloudflare/cfssl/releases/latest" | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/')
VNUMBER=${VERSION#"v"}
wget https://github.com/cloudflare/cfssl/releases/download/${VERSION}/cfssljson_${VNUMBER}_linux_amd64 -O cfssljson
chmod +x cfssljson
sudo mv cfssljson /usr/local/bin
cfssljson -version

Generate certificates

Creating root certificates

1. create CA key pair to use in this scenario:

   mkdir $WORKING_DIR/certs/generated
   
   cfssl gencert -initca $WORKING_DIR/certs/ca-csr.json | cfssljson -bare $WORKING_DIR/certs/generated/ca -
   

2. Validate Certificate Authority

   openssl x509 -in $WORKING_DIR/certs/generated/ca.pem -text -noout
   

3. Make up server-domain.json files, below is for zookeeper, likewise, create for other confluent components.

   cat $WORKING_DIR/certs/zookeeper-domain.json
   
         {
            "CN": "zookeeper",
            "hosts": [
                "*.confluent.svc.cluster.local",
                "*.zookeeper.confluent.svc.cluster.local",
                "*.kafka.confluent.svc.cluster.local"
            ],
            "key": {
               "algo": "rsa",
                "size": 2048
            },
           "names": [
             {
               "C": "Universe",
               "ST": "Pangea",
               "L": "Earth"
             }
            ]
         }
   

4. Create server certificates for each component as below

   cfssl gencert -ca=$WORKING_DIR/certs/generated/ca.pem \
   -ca-key=$WORKING_DIR/certs/generated/ca-key.pem \
   -config=$WORKING_DIR/certs/ca-config.json \
   -profile=server $WORKING_DIR/certs/zookeeper-domain.json | cfssljson -bare $WORKING_DIR/certs/generated/zookeeper
   

5. Validate server certificate

   openssl x509 -in $WORKING_DIR/certs/generated/zookeeper.pem -text -noout
   

Deploy configuration secrets

1. Create a Kubernetes secrets for zookeeper, likewise, create for other confluent components:

   kubectl create secret generic tls-zookeeper \
   --from-file=fullchain.pem=$WORKING_DIR/certs/generated/zookeeper.pem \
   --from-file=cacerts.pem=$WORKING_DIR/certs/generated/ca.pem \
   --from-file=privkey.pem=$WORKING_DIR/certs/generated/zookeeper-key.pem
   

Provide authentication credentials

kubectl create secret generic credential \
--from-file=plain-users.json=$WORKING_DIR/creds-kafka-sasl-users.json \
--from-file=digest-users.json=$WORKING_DIR/creds-zookeeper-sasl-digest-users.json \
--from-file=digest.txt=$WORKING_DIR/creds-kafka-zookeeper-credentials.txt \
--from-file=plain.txt=$WORKING_DIR/creds-client-kafka-sasl-user.txt \
--from-file=basic.txt=$WORKING_DIR/creds-control-center-users.txt

Deploy Confluent Platform

1. Deploy Confluent Platform with the above configuration:

   kubectl apply -f $WORKING_DIR/confluent-platform-production-mtls.yaml
   

2. Check that all Confluent Platform resources are deployed:

   kubectl get confluent
   

3. Get the status of any component. For example, to check Control Center:

   kubectl describe controlcenter
   

Access control center

1. Set up port forwarding to Control Center web UI from local machine:

   kubectl port-forward controlcenter-0 8021:8021
   

2. Browse to Control Center and log in as the admin user with the Developer1 password:

   https://localhost:8021
   

Tear down

kubectl delete -f $WORKING_DIR/confluent-platform-production-mtls.yaml

kubectl delete secret credential

helm delete operator