Distributing Load behind ALB/NLBs about opentelemetry-collector-contrib

[github-actions]

Pinging code owners:

• exporter/loadbalancing: @jpkrohling
• exporter/prometheusremotewrite: @Aneurysm9 @rapphil

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[jpkrohling]

and due to the requirement of ordered-writes, we saw a significant percentage of metrics dropped by AMP

I'm afraid this is more about Prometheus than load balancing itself, but why are you getting those out-of-orders only when receiving data from multiple clusters? Are you having the same metric stream (metric name + attributes) coming from different clusters? Are you dropping the service.instance.id along the way?

[diranged]

I'm afraid this is more about Prometheus than load balancing itself, but why are you getting those out-of-orders only when receiving data from multiple clusters?

The data coming from each cluster has unique stream labels attached - so one cluster can not really impact another cluster in terms of out-of-order samples. I believe the out-of-order issue has to do with timing .. For a single given stream, imagine this scenario:

Given a single stream: my_metric{instance='1.1.1.1:9200, pod_uid='<unique uuid>'}

1. Initial connection through LB hits metrics-ingester-0:4317 and starts accepting datapoints.
2. Datapoints Sent: T1, T2, T3, T4
3. Connection is interrupted because metrics-intester-0 is going to be replaced in K8S
4. New connection through LB hits metrics-ingester-1:4317 and starts accepting datapoints...
5. Datapoints Sent: T5, T6, T7, T8
6. metrics-ingester-0 begins its shutdown... has a bunch of data it needs to flush..
7. metrics-intgester-1 performs a flush of the T5-T8 samples
8. metrics-ingester-1 tries to flush T1-T4 samples.. but hits an out-of-order error and the samples are dropped.

This is a contrived example ... but the point I am trying to demonstrate is that the timing of when the samples are flushed out to Prometheus matters. If two different OTEL collector pods end up with samples for the same stream, but they flush out of order, then you create the out-of-order error situation.

We tried using IP-based session stickiness ... but that didn't really work at all, and is problematic for a lot of reasons. Session stickiness based on some cookie would be useful, if the OTEL client supported it.

Are you having the same metric stream (metric name + attributes) coming from different clusters?

No - definitely unique streams from different clusters

Are you dropping the service.instance.id along the way?

No we are not