kamon-io / kamon-prometheus

kamon prometheus integration

This reporsitory has been moved.

Since March 2020 all the Kamon instrumentation and reporting modules were moved to Kamon's main repository at https://github.com/kamon-io/kamon. Please check out the main repository for the latest sources, reporting issues or start contributing. You can also stop by our Gitter Channel.

kamon-prometheus's Issues

Legal characters are excluded in names and tags

Hello there,
I'm pretty new to Prometheus, but I have colleagues that built up a running Prometheus metrics infrastructure.
What I've observed is that they use colons in names a lot, and that Kubernetes happily publishes metrics with dashes in tags. The kamon-prometheus adapter however strips these characters from names and tags - is this intentional?

Prometheus exporter not working

Hi all,
so I'm new to a project using the prometheus exporter and have seen that it does not work properly. Behaviour is the metrics endpoint not returning any data anymore:

curl localhost:9095/metrics -v
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9095 (#0)
> GET /metrics HTTP/1.1
> Host: localhost:9095
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/plain; version=0.0.4; charset=utf-8
< Date: Mon, 20 Jan 2020 11:52:29 GMT
< Connection: keep-alive
< Content-Length: 0
<
* Connection #0 to host localhost left intact
* Closing connection 0

Digging a bit I think it must have stopped working 2019-12-20T00:00:00Z for all users of this lib, I'll try to elaborate and I hope I'm wrong :).

In class PrometheusReporter I found

private val _snapshotAccumulator = PeriodSnapshot.accumulator(Duration.ofDays(365 * 5), Duration.ZERO)

Which sets a time frame of five years for some accumulation. Based on that accumulation time frame, a date gets calculated (_nextTick, currently it's calculated as 2024-12-18T00:00:00Z) which is used in various places. As far as I understand it, it's the max time frame that should be used for accumulation of data?

The function PeriodSnapshot.Accumulator.peek, which gets called periodically to calculate the data to be written, uses that nextTick to say until when it wants to read data. Before composing the data, it tries to tidy up (cleanStaleEntries). That function calculates a cutoff date, until when it considers data to be valid:

val cutoff = currentInstant.minus(stalePeriod)

currentInstant in that case is set to _nextTick, 5 years in the future. I'm not 100% sure what the correct value would be there, but using _nextTick currently calculates something 4 years into the future, which discards all data.

I'll prepare a short PR to fix it at least "a bit", but please have a look at what time would be the "right" there. I'd use from instead, but if you know better please tell me.

reference.conf has unquoted value

The default setting for "hostname" in reference.conf is:
hostname = 0.0.0.0 but should be hostname = "0.0.0.0"
It works on newer versions TypeSafe's "Config" library, but fails on older ones, since it tries to parse it as a float

Publish Scala 2.13 build of 2.0.0-SNAPSHOT(2.0.0-RC1)

Running kamon-prometheus 2.0.0-RC1 scala-2.12 with kamon-core 2.0.0-RC1 scala-2.13 cause an exception.

2019-07-04 13:59:47.991 | WARN  | main | k.m.ModuleRegistry | Failed to create instance of module [Prometheus Reporter]
java.lang.NoClassDefFoundError: scala.Serializable
	at java.base/java.lang.ClassLoader.defineClassImpl(Native Method)
	at java.base/java.lang.ClassLoader.defineClassInternal(ClassLoader.java:479)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:448)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:1114)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:902)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:808)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:766)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:1052)
	at kamon.prometheus.PrometheusReporter.<init>(PrometheusReporter.scala:45)
	at kamon.prometheus.PrometheusReporter.<init>(PrometheusReporter.scala:42)
	at kamon.prometheus.PrometheusReporter$Factory.create(PrometheusReporter.scala:101)
	at kamon.module.ModuleRegistry.createModule(ModuleRegistry.scala:282)
	at kamon.module.ModuleRegistry.$anonfun$load$3(ModuleRegistry.scala:94)
	at kamon.module.ModuleRegistry$$Lambda$486.0000000069D71C20.apply$mcV$sp(Unknown Source)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at scala.Option.fold(Option.scala:264)
	at kamon.module.ModuleRegistry.$anonfun$load$2(ModuleRegistry.scala:97)
	at kamon.module.ModuleRegistry.$anonfun$load$2$adapted(ModuleRegistry.scala:90)
	at kamon.module.ModuleRegistry$$Lambda$485.0000000069D71420.apply(Unknown Source)
	at scala.collection.immutable.List.foreach(List.scala:312)
	at kamon.module.ModuleRegistry.load(ModuleRegistry.scala:90)
	at kamon.ModuleLoading.loadModules(ModuleLoading.scala:66)
	at kamon.ModuleLoading.loadModules$(ModuleLoading.scala:65)
	at kamon.Kamon$.loadModules(Kamon.scala:19)
	at kamon.Init.init(Init.scala:45)
	at kamon.Init.init$(Init.scala:42)
	at kamon.Kamon$.init(Kamon.scala:19)
	...
Caused by: java.lang.ClassNotFoundException: scala.Serializable
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:768)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:1052)
	... 88 common frames omitted

Is it possible to publish kamon-prometheus 2.0.0-RC1 build for Scala 2.13

Grafana dashbord

It would be great to have dashbord example like kamon-docker-grafana-graphite

akka-http integration

I am currently using
https://github.com/MonsantoCo/kamon-prometheus
which provides a way to integrate with akka-http. It will be nice to have the same feature in this module.

Regards

In custom buckets configuration, string having dot is not working

Not working -
custom {
"akka.actor.processing-time" = [0.1, 1.0, 10.0]
}

Though, string without dot works, below conf works fine.
custom {
"plain_string" = [0.1, 1.0, 10.0]
}

Probable fix would be, in PrometheusReporter.scala --> def readCustomBuckets,

"k" - variable needs to be in double quotes while passing to, getDoubleList(k),

I might be wrong.

Thanks

Allow overriding the bucketing configuration in a per-metric basis

It would be great (even necessary in my opinion) to be able to configure specific bucketing configuration for selected metrics. Imaging adding something like this to the configuration:

kamon.prometheus.buckets.custom {
  "akka.actor.processing-time" = [ 0.1, 2, 3, 4, 5, 10] // or whatever makes sense
}

And then all matching metrics would use this bucketing config, while the rest will use the default settings.

No config/usage example

Would be nice to have an example of the configuration so I know how to use it.

Question: how to scrape the prometheus reporter in v2.0.0?

To manually scrape the reporter in v1, you can do something like this:

val reporter = new PrometheusReporter()
Kamon.addReporter(new PrometheusReporter())

val data = reporter.scrapeData()

But in v2, Kamon.addReporter doesn't exist anymore. What's the correct way of doing this in v2?

content type should be "text/plain; version=0.0.4; charset=utf-8"

This is what the Prometheus Java Client servlet returns.
https://github.com/prometheus/client_java/blob/master/simpleclient_common/src/main/java/io/prometheus/client/exporter/common/TextFormat.java

Update readme to reflect that kamon 1.0.0 isn't really available

It is misleading as the dependencies are just not there.

PrometheusReporter thread is not daemon

The thread should be daemon by default and users should be allowed to configure it.