metrics: set up alerting about hush-house

[scottietremendous]

@taylorsilva - I agree with this. We have other potential avenues to dive into Prometheus and hush-house doesn't need to be held back because of it.

[scottietremendous]

We'll probably want alerts for Datadog SLIs/SLOs

[scottietremendous]

Chore: Bump Grafana to 6.0

[YoussB]

note: the domain doesn't redirect http requests to https

[taylorsilva]

It seems possible to use Prometheus with stackdriver, though the integration has some limitations and is in beta (see here). Based on one medium blog post, seems possible though!

Our current plan is to setup alerts with Grafana and create another long-term issue to implement alerts with Prometheus. We don't want to tackle prometheus now because we have a lot of learning to do about it. We'd rather get hush-house production-ready sooner rather than later, which means using Grafana for alerts in the short-term.

Thoughts?
cc @YoussB @cirocosta @scottietremendous

[cirocosta]

sounds goood 😁 thankss!

[YoussB]

EOD

• Added two alerts using Grafana. One for high disk usage and a second for high number of containers
• We setup PagerDuty integration with hush-house's Grafana and for the previous two alerts

Next Steps:

• Think about what kind of alerts we want for hush-house and add them using Grafana.
• Figure out why Grafana's HTTP is not redirecting to HTTPS and make it redirect.

[scottietremendous]

Let's make a new issue around diving into what suite of metrics we can provide on hush house outside of what we currently offer on Wings.

[cirocosta]

Hey,

@pivotal-bin-ju and I were looking at Grafana performing the redirect, and it turns out that Grafana itself doesn't do that (differently from Concourse, which does redirect automatically), meaning that we'd need something else to perform the redirects (like nginx).

We'll leave this to another issue in order to not be blocked on this (something that the other metrics dashboard doesn't provide anyway).

Thanks!