otel: conflicting Schema URL about descheduler

[damemi]

I think this is because of our import for the otel semconv that uses 1.12, which is pretty old https://github.com/kubernetes-sigs/descheduler/blob/748495a022b7445e3bcce0be20effc4890b8a9fc/pkg/tracing/tracing.go#L30C11-L30C51

I opened #1429 to bump this to 1.24 which should help. What's annoying is that we will have to keep these in sync, so I'd like to see if there's a way we can drop the direct semconv import. I think it's being used by some of the other otel imports too, and the conflict is probably coming from one of those being at a higher version.

Also not sure why this didn't come up in tests. @morremeyer is this a fatal error or is the descheduler still running with that error? Our otel implementation should be silent if it's not enabled so an error in this case is bad

[damemi]

@morremeyer thanks, to help debug why your pods aren't being evicted you can check the docs which list all the reasons a pod would be skipped by default. You can also set --v=4 to increase the logging verbosity to print a message why each pod is skipped.

I'll update my PR because I don't think we should be causing fatal errors if tracing fails to start up

[morremeyer]

@damemi it is a fatal error, it exits directly after the last log line. Once I disable it, it works fine.

I also have the problem that if set the helm values to

deschedulerPolicy:
  tracing:
    collectorEndpoint: monitoring-alloy.monitoring.svc.cluster.local:4317
    sampleRate: 1.0

It does not even recognize the config. I did not mention this in the initial comment since it also looks like the policy (see initial comment) does not do anything - it's running but no pods are evicted, even ones that look like they match the policy to me. I didn't comment about this because so far I think this is a user error and I'm still looking into why this is happening.

[morremeyer]

I agree, if tracing fails to start up, that should be a warning or error log message and then execution continues.

Thanks for looking into it this quick, I appreciate it a lot!