Support for Custom Collectors about client_ruby

[a0s]

+1 for Custom Collectors
Examples of usage:

1. Getting instantly values from ActiveRecord::Base.connection_pool.stat
2. Getting instantly values from Puma server (through control socket)
3. Getting instantly values from GC.stat
4. etc..

[dmagliola]

Hi @directionless,
I would like to understand this problem better...
Do you have any examples where you need this functionality?
Thank you

[directionless]

Since submitting this, I've stopped working on the project that needed this. However, the need was real, and I believe should I pick up ruby and prometheus I'll encounter it again.

It generally centers around importing metrics that are generated elsewhere. The python docs have a reasonably clear explanation:

Sometimes it is not possible to directly instrument code, as it is not in your control. This requires you to proxy metrics from other systems.

@a0s has some good examples.

Mine were business/app data, for example:

• total users User.count
• active users ActiveRecord::SessionStore::Session.where("updated_at > ?", 15.minutes.ago).count
• Similar for other important data models
• Etc.

I usually want these metrics for several reasons. First, it makes for a simple business analytics system. I've already got metrics and graphs, I may as well add in a little bit of BI. Second, when debugging application issues, I usually also want those numbers on a graph. Third, I may want to set alerts. More concretely, I probably know that my application performance is strongly correlated with the number of widgets in the system. So I want to be monitoring that widget count, and comparing it.

For that use case, I also ran into #9 and ended up writing a fork of the gitlab prometheus-client-mmap gem. Submitted upstream at https://gitlab.com/gitlab-org/prometheus-client-mmap/merge_requests/24

[a0s]

@directionless

BTW,

total users User.count
active users ActiveRecord::SessionStore::Session.where("updated_at > ?", 15.minutes.ago).count

I think there is a better place for this. There is postgres_exporter. You able to add custom metrics with custom SQL scripts. Moreover, making requests like a User.count on a Postgres-side is better for performance.

[lawrencejones]

Hey @directionless!

I totally get your use case, and I think they are valid things to want to export in Prometheus. We export many similar metrics from our services.

Can I ask what is missing to support this, though? I've always produced a Collector class that adheres to the Rack middleware interface, then place this in the scrape middleware chain and have it pull the metrics just before responding to a scrape.

As a result, I'm not sure what we need from the library that is missing?

[a0s]

@directionless I open-sourced my small library for client_ruby. And add prototype of CustomCollector.

[directionless]

To reiterate -- I am no longer working on the project that had this use case. So I cannot easily evaluate how well this would replace it. But, comments inline...

I think there is a better place for this. There is postgres_exporter.

While I think that's better for some things, as active record models get more complicated, I'm very leery of separating the reporting logic from it. And, as in your library, that's not going to work for puma stats.

Can I ask what is missing to support this, though? I've always produced a Collector class that adheres to the Rack middleware interface, then place this in the scrape middleware chain and have it pull the metrics just before responding to a scrape.

That sounds interesting, and may cover this, but I don't know that I understand what that would do. Initially it sounds like it would work, is there an example somewhere of how that would work? I can imagine it either exactly meeting the original request for custom collectors. Or feeling like re-implementing a lot of existing features.

There's probably also something caveat about timing. Some metrics are cheap and I want to pull them frequently. Other metrics are expensive, and I want to pull them less frequently. Though perhaps that's best done by sticking some cache+ttl in the metric pull.

[dmagliola]

@lawrencejones How do you feel about having a CustomCollector class that basically implements that Rack interface, and calls a couple of overridable methods (before, after, around, maybe?) so that all you need to do is subclass it, set up your metrics declaration, and override one of those methods to set / increment those metrics in them. (And add to the middleware chain)

It's essentially the same as you said, but it may be more clear / easier to use as an interface?

Either way, I think this is mostly solved by documentation. We possibly don't need that class, just a code snippet in the README on how you'd implement your own custom collector that is easy to understand.

Either way, I don't think this should block the 1.0 release

[lawrencejones]

I'm not sure, thinking about it, a metric that when read fetches its value could be really useful, especially in the fork world.

But it's entirely separate from the 1.0 release- literally nothing should hold that back, there is no reason to delay the 1.0 release as far as I can see. The number of people who adopt Prometheus in ruby every day and use the pre 1.0 grows every day, so the faster this is released the better for the community.

[dmagliola]

Agreed on that last point. Will get your thoughts on the metric that fetches when read next time I see you :-)

[hiroshi]

I made quick and dirty hack to this kind of purpose for my rails app.

# config/initializers/prometheus.rb
require 'prometheus/client'
Prometheus::Client.registry.gauge(:room_temperature_celsius, docstring: '...', labels: [:room])

# app/controller/prometheus_exporter.rb
require 'prometheus/client'
require 'prometheus/client/formats/text'

class PrometheusExporterController < ApplicationController
  def index
    gauge = Prometheus::Client.registry.get(:room_temperature_celsius)
    gauge.set(21.534, labels: { room: 'kitchen' })
    render plain: Prometheus::Client::Formats::Text.marshal(Prometheus::Client.registry)
  end
end

# config/routes.rb
get '/metrics' => 'prometheus_exporter#index', format: false

And It works for me.

$ curl http://localhost:5000/metrics
# TYPE room_temperature_celsius gauge
# HELP room_temperature_celsius ...
room_temperature_celsius{room="kitchen"} 21.534