Scraping Metrics from a CronJob about descheduler

[damemi]

This is a problem that unfortunately affects metrics collection for any short-lived workload (I was working on the same issue for serverless recently). So, it's not just a descheduler issue and I don't think a deadline setting like you proposed is technically the right solution. I don't know if the Prometheus community has come to a broader solution for this type of problem.

Ultimately, short-lived workloads benefit from exporting their metrics to a listening server, rather than the Prometheus standard of waiting to be scraped by a server. This is how OpenTelemetry metrics work, and when a workload shuts down all metrics in memory are flushed to the collection endpoint.

So I think to really address this, we should consider updating our metrics implementation to use OpenTelemetry. We already use Otel for traces, so there is some benefit to using both. But the good news is we could do this without breaking existing Prometheus users either by:

• Use the Otel Prometheus bridge with our current prom instrumentation
• Switch our instrumentation entirely to Otel and use the Prometheus exporter to keep providing a prometheus endpoint

@yutkin Unfortunately this still doesn't fix your problem, because you're using a Prometheus server to scrape the endpoint. But if we implement Otel metrics, you could run an OpenTelemetry Collector with otlp receiver and Prometheus exporter, then point your Prometheus agent at that endpoint.

[seanmalloy]

Here is another option:

Prometheus has a push gateway for handling this, https://github.com/prometheus/pushgateway. I'm not super familiar with push gateway, but I believe the descheduler code would need to be updated to have an option to push metrics when running as a Job or CronJob.