add service metrics / instrumentation / telemetry about pomerium

[travisgroth]

Short writeup on technology/libraries/approach

Target technology

My take on the current ecosystem is that most new applications are trending towards prometheus as an exposition format. It is simple, human friendly, and the team behind Prometheus is working on making it a standard.

Additionally, most other monitoring stacks have adapters for prometheus. Influx, datadog, elastic, all appear to cover it.

Thus, I believe whatever approach is taken should take care to support prometheus as a format as a priority. If any other metrics formats/strategies can be supported simultaneously, that is a bonus.

Library/framework support

With the goal of prometheus format, the prometheus client is an obvious choice. It is widely used and heavily tested. It doesn't easily support other formats, however. Some other choices in the ecosystem:

• https://godoc.org/github.com/go-kit/kit/metrics. Their interface abstraction keeps your code from needing to understand the different concrete metric formats, but every integration and metrics set must be maintained separately. By their own admission, if you know your target stack, you should use it directly.

• https://github.com/armon/go-metrics. Low verbosity and the in memory dump could be interesting during debugging. It only supports summary type metrics, however, which is somewhat limiting; they also don't seem to support help messages. Does support statsd, which I think is a good "also" format.

• https://github.com/segmentio/stats. Docs are a bit rough, especially after the latest upgrade. Does support a lot of output formats without too much work. No less verbose to create metrics than prometheus itself.

• https://opencensus.io/quickstart/go/metrics/. This is a google backed/derived project. Definitely early days, but the go support appears solid. If the community coalesces around this project, it will become the library and all exposition formats become a new plugin. Prometheus is supported out of the box. We also gain tracing support at the same time. It looks like their velocity has slowed down recently due to this: https://opentelemetry.io/.

My recommendation is to leverage either segmentio/stats or opencensus, but I'm leaning more toward opencensus. It is going to attract more support than segmentio/stats, and we get a bit more with the tracing ecosystem. This is a bit of a leap of faith that they will continue adding formats and gain traction, but it looks promising.

Very open to feedback and any more considerations on those two libraries and the stock prometheus client.

Approach

Rough pass on code level architecture:

• Refactor the logging http handler middleware to be generic
• Create an internal/metrics package to host generic middleware and metrics
• Create and wire up http and grcp middleware on each service
• Set up a configurable second HTTP listener for prometheus format
• Try to keep the metrics implementations as shielded from the rest of the code as possible

Some sample metrics:

pomerium_http_server_requests{service="proxy", host="${proxy_from}", method="get", status="200"}
pomerium_http_client_requests{service="proxy", host="${proxy_to}", method="get", status="200"}
pomerium_grpc_client_requests{service="proxy", host="${proxy_to}", method="${rpcMethod}", status="xxx"}
pomerium_grpc_server_requests{service="authenticate", method="${rpcMethod}", status="xxx"}
pomerium_config_last_reload_successful{service="authorize"}
pomerium_config_last_reload_success_timestamp_seconds{service="authorize"}

Other things to look at: proxy/authn/authz connection stats, go internal stats, proxy upstream error types.

[desimone]

👍 big plus one from me for OpenCensus (and soon, OpenTelemtry).

See also:

• Merging OpenTracing and OpenCensus: Goals and Non-Goals
• Microservices demo featuring open census
• Unified Observability in Kubernetes and Beyond with OpenCensus - Morgan McLean & Jaana Burcu Dogan
• Correlating traces and logs with OpenCensus

[desimone]

@travisgroth Now that the bulk of the work is done, what do you think about closing this in favor of small individual issues (adding exporters etc).

Also 🎊

[travisgroth]

Yes I think we've covered this thoroughly. Let's close. 🎉

[desimone]

Great. And thank you for spearheading this.