inditextech / provider-kafka

Crossplane provider for Kafka

provider-kafka

provider-kafka is a Crossplane Provider that is used to manage Kafka resources.

Usage

1. Create a provider secret containing a json like the following, see expected schema here:

   {
     "brokers":[
       "kafka-dev-0.kafka-dev-headless:9092"
      ],
      "sasl":{
        "mechanism":"PLAIN",
        "username":"user",
        "password":"<your-password>"
      }
   }
   

2. Create a k8s secret containing above config:

   kubectl -n crossplane-system create secret generic kafka-creds --from-file=credentials=kc.json
   

3. Create a ProviderConfig, see this as an example.

4. Create a managed resource see, see this for an example creating a Kafka topic.

Development

Setting up a Development Kafka Cluster

The following instructions will setup a development environment where you will have a locally running Kafka installation (SASL-Plain enabled). To change the configuration of your instance further, please see available helm parameters here.

1. (Optional) Create a local kind cluster unless you want to develop against an existing k8s cluster.

2. Install the Kafka helm chart:

   helm repo add bitnami https://charts.bitnami.com/bitnami
   kubectl create ns kafka-cluster
   helm upgrade --install kafka-dev -n kafka-cluster bitnami/kafka \
     --version 20.0.5 \
     --set auth.clientProtocol=sasl \
     --set deleteTopicEnable=true \
     --set authorizerClassName="kafka.security.authorizer.AclAuthorizer" \
     --wait
   

   Username is "user", obtain password using the following

   kubectl -n kafka-cluster exec kafka-dev-0 -- cat /opt/bitnami/kafka/config/kafka_jaas.conf
   

   Create the Kubernetes secret by adding a JSON filed called kc.json with the following contents

   {
      "brokers": [
         "kafka-dev-0.kafka-dev-headless:9092"
      ],
      "sasl": {
         "mechanism": "PLAIN",
         "username": "user",
         "password": "<password-you-obtained-in-step-2>"
      }
   }

   Once this file is created, apply it by running the following command

   kubectl -n kafka-cluster create secret generic kafka-creds --from-file=credentials=kc.json

3. Install kubefwd.

4. Run kubefwd for kafka-cluster namespace which will make internal k8s services locally accessible:

   sudo kubefwd svc -n kafka-cluster
   

5. To run tests, export the KAFKA_PASSWORD environment variable using the password from step 2

   export KAFKA_PASSWORD="<password-you-obtained-in-step-2>"
   

6. (optional) Install the kafka cli.

7. (optional) Configure the kafka cli to talk against local Kafka installation:

   1. Create a config file for the client with the following content at ~/.kcl/config.toml:

      seed_brokers = ["kafka-dev-0.kafka-dev-headless:9092"]
      timeout_ms = 10000
      
      [sasl]
      method = "plain"
      user = "user"
      pass = "<password-you-obtained-in-step-2>"
      

      1. Verify that cli could talk to the Kafka cluster:

      export  KCL_CONFIG_DIR=~/.kcl
      
      kcl metadata --all
      

Building and Running the provider locally

Run against a Kubernetes cluster:

make run

Build, push, and install:

make all

Build image:

make image

Push image:

make push

Build binary:

make build