[Suggestion] - BlockList additions to Confluence JMX Exporter config about data-center-helm-charts

[bianchi2]

@atl-mk I wonder if you can help here.

[bendeme]

I want to bump this and add that in the current state jmx-exporter config for confluence is unusable and dangerous to any prometheus setup having to ingest it. Maybe there should be a warning that this is not intended for production use until a reasonable exclusion list is added to control growth of metric scrape.

[bianchi2]

Asked a question here prometheus/jmx_exporter#904

As a workaround you can turn off Confluence specific JMX beans by passing -Dconfluence.jmx.disabled=true to JVM args. You will have JVM metrics only though (which in many cases is good enough for basic monitoring)

[bianchi2]

@mattj-relativity thanks for raising this issue. Indeed, metrics can be quite noisy and high cardinality can definitely be an issue. Did I get you right - your suggested excludeObjectNames did not work and you still see those metrics produced? So it's rather a matter fo finding the right syntax to exclude selected metrics?

[mattj-relativity]

Regarding syntax issues

- 'com.atlassian.confluence.metrics:category01=logging,*'
this line worked. but the attempts to handle anything in category00 have not worked. not sure why.

Regarding Scope of block list

I have not completed figuring out all the items i will need to block list. I would appreciate some feedback on how to approach figuring that out. ATM I am basically working against the existing attributes used in the grafana charts and what i can see as suspect in iterative exporter scrapes. Not even sure that's the right way to be approaching this.

Regarding severity of impact

I do want to be clear... this isn't just a performance bug. It effectively makes the jmx exporter unusable on confluence. in the default configuration under any load you will see the exporter trend toward infinity in terms of time it takes to scrape the exporter endpoint. It will eventually exceed any reasonable timeout in prometheus. You can increase the timeouts, until such time as it just endlessly times out. it will additionally generate such severe cardinality growth in prometheus that it will OOM prometheus.

[bianchi2]

Another possible workaround I can think of is metricRelabelings in a ServiceMonitor. https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/running-exporters.md

I haven't had a chance to verify it and see if it's possible to drop metrics by category.