[jaeger] deploying Ingester about helm-charts

[arpitjindal97]

We can add config for provisioning Kafka for the user who don't have it and also users can provide details of exisiting kafka.
Ingester has support for kafka only. So, I think there is no point in allowing user to choose between kafka and other streaming services. (maybe in future when support for other is available)

Kafka enablement will be decided based on ingester

ingester:
  enabled: false
  annotations: {}
  image: jaegertracing/jaeger-ingester
  pullPolicy: IfNotPresent
  cmdlineParams: {}
  service:
    type: ClusterIP
  extraConfigmapMounts: []

Ingester will be enabled: false by default. If user has enabled it then only kafka configs will apply over collector.

For provisioning of kafka, It will be totally based provision value. Something like this

provisionDataStore:
  cassandra: true
  elasticsearch: false
  kafka: false

This is how I have tackled this situation :)

[arpitjindal97]

Yes, correct. I have everything configured and working. I will do PRs one by one

[arpitjindal97]

Actually, I added that part also few days back and it is not difficult. Just need to add few lines that's it. I used google provided kafka chart. Same as cassandra

[naseemkullah]

We should allow the end user to either provide existing Streaming Backend (Kafka) values, or for those who just want to get going and don't have kafka, we should add an option to provision it via https://github.com/confluentinc/cp-helm-charts/tree/master/charts/cp-kafka (That chart might need some TLC, but it's backed by Confluent so that's good).

Furthermore, I think we should specify that the backendtype is Kafka, and leave it open to it being something else in the future (NATS, Redis(Streams), etc.)

[naseemkullah]

Sounds, good.

the value of kafka brokers in collector should be templated in https://github.com/jaegertracing/helm-charts/blob/master/charts/jaeger/templates/_helpers.tpl like cassandra.host and elasticsearch.client.url .... we could call it kafka.brokers perhaps.

As for the topic, it should be left blank by default, which will default to jaeger-spans as mentioned in https://www.jaegertracing.io/docs/1.16/cli/#jaeger-collector-kafka

[naseemkullah]

Great, you have it working for a provisioned kafka from cp-charts as well?

[naseemkullah]

Nice, we should probably go with confluence backed cp-charts kafka since incubator repo will be obsolete within a year

[arpitjindal97]

I tried using cp-kafka, but they do not host kafka chart like we need it. They have packaged too many things together in a single chart for their Confluent platform.

$ helm show chart confluent/cp-helm-charts
apiVersion: v1
appVersion: "1.0"
dependencies:
- condition: cp-kafka.enabled
  name: cp-kafka
  repository: file://./charts/cp-kafka
  version: 0.1.0
- condition: cp-zookeeper.enabled
  name: cp-zookeeper
  repository: file://./charts/cp-zookeeper
  version: 0.1.0
- condition: cp-schema-registry.enabled
  name: cp-schema-registry
  repository: file://./charts/cp-schema-registry
  version: 0.1.0
- condition: cp-kafka-rest.enabled
  name: cp-kafka-rest
  repository: file://./charts/cp-kafka-rest
  version: 0.1.0
- condition: cp-kafka-connect.enabled
  name: cp-kafka-connect
  repository: file://./charts/cp-kafka-connect
  version: 0.1.0
- condition: cp-ksql-server.enabled
  name: cp-ksql-server
  repository: file://./charts/cp-ksql-server
  version: 0.1.0
- condition: cp-control-center.enabled
  name: cp-control-center
  repository: file://./charts/cp-control-center
  version: 0.1.0
description: A Helm chart for Confluent Platform Community Edition
name: cp-helm-charts
version: 0.4.0

I think it's not useful for us. We just to provide a working kafka for the user to try Ingester.

You can also give it a try :)

[naseemkullah]

I don't think you need to include all those charts, just the cp-kafka one... I'll give it a try when I find some time.

[arpitjindal97]

There is only one chart confluent/cp-helm-charts. All others come under this repository. They are not available to be used separately. Just give it a try then we will discuss how to proceed for kafka.

Till then, should I provide config for Ingester ?

[naseemkullah]

There is only one chart confluent/cp-helm-charts. All others come under this repository. They are not available to be used separately. Just give it a try then we will discuss how to proceed for kafka.

Ah my mistake, confluentinc/cp-helm-charts#374 opened! Let's see what they suggest.

Till then, should I provide config for Ingester ?

Sure feel free to get started with providing existing Kafka info 🚀 . You can put the incubator kafka chart as provisioned dependency as a placeholder but if cp-kafka ends up being the right one we will swap it out.

[naseemkullah]

solved via #35