Prometheus fails to scrape Helidon metrics about helidon

[barchetta]

We have confirmation from MicroProfile that the spec is misleading:
https://groups.google.com/forum/?utm_source=digest&utm_medium=email#!topic/microprofile/t4YgzPpmOsA

"I'm part of the metrics community and I agree that what's specified in section 2.3, related to the Accept header isn't as specific as it ought to be. The way it's worded in the spec right now isn't right, as illustrated by Joe's example.

The way we'd intended for it to be interpreted was that if the incoming request accepts text/plain with a higher priority than application/json, then you should use Prometheus format. If the incoming request accepts application/json with a higher priority than text/plain, then use JSON format. If the incoming request doesn't accept either of those, then return a 406 status code."

There is now an issue to clarify this in the spec: eclipse/microprofile-metrics#291

[spericas]

Interesting, I agree. Take a look at HashRequestHeaders::bestAccepted.

[barchetta]

The issue appears to be that Prometheus sends an Accept header that looks like:

text/plain;version=0.0.4;q=1,*/*;q=0.1

So it is requesting text/plain with a quality of 1, but will also accept anything. One can argue this isn't correct (since Prometheus can't handle JSON for example). But that's not the point.

The point is that Helidon will generate JSON in this case, and not Prometheus text. This is because of this method:

    private void getMultiple(ServerRequest req, ServerResponse res, Registry... registries) {
        if (req.headers().isAccepted(MediaType.APPLICATION_JSON)) {
            res.send(toJsonData(registries));
        } else {
            res.send(toPrometheusData(registries));
        }
    }

Since the request headers will accept JSON, we generate JSON.

The logic needs to be more sophisticated. Since Prometheus prefers text/plain, that's what we should generate. To be explicit:

If the Accept header is text/plain;version=0.0.4;q=1,*/*;q=0.1 then we should generate a Prometheus text report.

If the Accept header is application/json;version=0.0.4;q=1,*/*;q=0.1 then we should generate a JSON report.

[tomas-langer]

I have requested clarification from Metrics spec on media types:
https://groups.google.com/forum/?utm_source=digest&utm_medium=email#!topic/microprofile/t4YgzPpmOsA

My current understanding of the spec would mean that we cannot change the behavior as requested by this issue.

[barchetta]

Thanks for asking for clarification. I hope this is a case of the MicroProfile spec being sloppy because the current behavior means that MicroProfile apps (and by extension Helidon apps) can't be scraped by Prometheus -- which is absurd.

[spericas]

@barchetta I agree. The spec may need to be clarified, but I doubt their intent was to ignore quality factors in an Accept header. We should really return text/plain for that value of Accept.