errors fetching Prometheus metric - text format parsing error about nri-prometheus

[paologallinaharbur]

Hello, likely it is related to #313

To doublecheck could you check the prometheus output and tell us if you notice differences querying with:

• No Accept header
• Header ‘Accept: application/openmetrics-text;version=0.0.1,text/plain;version=0.0.4;q=0.5,/;q=0.1’
• Header ’Accept: text/plain;version=0.0.4;q=1,/;q=0.1’

[paologallinaharbur]

Yes I meant with curl!
No, but we are going to address this issue soon.

By the way it is possible that the output changes according to the Accept Header included in the request, this is why this test is useful. We do not need the output, just to know if that escaping sequence is present. Thanks!

[joncolby]

Hi @paologallinaharbur - I am happy to help debug this.

How do you mean I should query the prometheus targets with these different Headers? Using curl? I probably won't notice a different in visual output, but I will do the experiment anyway. Then i will do a diff and checksum. I cannot share the complete output however. This problem happens even when there are no fields containing escape sequences \ in the prometheus metrics output, by the way.

Or is there a way to configure the nri-prometheus agent to use different headers when scraping targets?

[joncolby]

@paologallinaharbur -

I proxied to one of our pods, then called the metrics endpoint with curl. with no accept headers , it works as expected. but with both of the other headers, I get a 406 error.

A quick google search shows that some prometheus implementations return this code if they are not expecting other Accept headers. Especially java/spring boot.

❯ curl -v -o - -H "Accept: text/plain;version=0.0.4;q=1,/;q=0.1" http://localhost:9001/metrics
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9001 (#0)
> GET /prometheus HTTP/1.1
> Host: localhost:9001
> User-Agent: curl/7.64.1
> Accept: text/plain;version=0.0.4;q=1,/;q=0.1
>
< HTTP/1.1 406
< Content-Length: 0
< Date: Wed, 06 Jul 2022 12:41:16 GMT



❯ curl -v -o - -H "Accept: application/openmetrics-text;version=0.0.1,text/plain;version=0.0.4;q=0.5,/;q=0.1" http://localhost:9001/metrics
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9001 (#0)
> GET /prometheus HTTP/1.1
> Host: localhost:9001
> User-Agent: curl/7.64.1
> Accept: application/openmetrics-text;version=0.0.1,text/plain;version=0.0.4;q=0.5,/;q=0.1
>
< HTTP/1.1 406
< Content-Length: 0
< Date: Wed, 06 Jul 2022 12:41:41 GMT

[paologallinaharbur]

My bad, copying pasting the */* was transformed into */*due to markdown.

For example in my environment this returns as expected openmetrics-text

$ curl {endpoint} -H 'Accept: application/openmetrics-text;version=0.0.1,text/plain;version=0.0.4;q=0.5,*/*;q=0.1'

and this text/plain

$ curl localhost:8000 -H 'Accept: text/plain;version=0.0.4;q=0.5,*/*;q=0.1'

We have enough data to work on a fix, in the meanwhile I will close this ticket as a duplicate of #313, thanks again for reporting it!