Proposal: Support ML Monitoring about oteps

[jkwatson]

Hi @skonto. This is definitely something that I am interested in, being at verta. Have you thought about what kind of metric aggregations would be relevant for model monitoring, and how they might map to the OpenTelemetry metric instruments? Would we need some custom aggregations? Could they be done in-process, or would some sort of collector be more appropriate? Love to hear your thoughts on this topic.

[skonto]

Hi @jkwatson here are some initial thoughts. We already have enough to start with existing instruments and aggregations.
For example we can split model monitoring into two basic areas like model performance and model operational performance (terms can change). In the first area we need to do the work of identifying what we want to measure and then try map to the standard or extend it. For example we might find useful to measure data drift and use Jense-Shannon divergence implemented as a histogram. In the second domain we might consider disko i/o, uptime, cpu/mem utilization, scoring latency etc.
Of course we need input about what is useful to standardize, what people use in practice etc.
We could start researching this by doing the exercise of putting together a number of metrics that provide a good overview of a model's performance. cc @dineshg13

[jkwatson]

I think the interesting new work would be in model performance, and not necessarily in the operational performance, which I think would probably fall understand standard observability-style metrics.

With regards to drift... I can imagine a reference histogram being recorded somewhere, and then some kind of time-windowed cumulative histogram being used to compute the divergence. I'm not sure OTel itself really has facilities for "time windowed" cumulative histograms right now... there might be an opportunity to make a proposal to fulfill that need.

[skonto]

I'm not sure OTel itself really has facilities for "time windowed" cumulative histograms right now... there might be an opportunity to make a proposal to fulfill that need.

That would be interesting work, want to start a draft doc?

[jkwatson]

I'm not sure OTel itself really has facilities for "time windowed" cumulative histograms right now... there might be an opportunity to make a proposal to fulfill that need.

That would be interesting work, want to start a draft doc?

I don't think I have time to work on that at the moment, but I'd be very happy to take a look at a proposal from someone who did have time.