Set up alerting for ffnet health about infra

[CodeSandwich]

The alerts are added for:

• any node being down
• any node having best block height grow by less than 1 in 10 minutes or by 40 in an hour.

[geigerzaehler]

One issue we have with the current setup is that we get false alerts for nodes being down when miners are pre-empted. To check miner availability we need to distinguish between nodes being down because its pod is rescheduled due to a VM being pre-empted and a node being down because it has crashed.

I found that kube-state-metrics exposes a metric for a the pod phase. We could use the “Failed” status of that metrics to determine whether a node crashed.

[geigerzaehler]

Here are my initial thoughts on this:

As a channel we should choose an existing medium. For us that means email or IRC. I would prefer IRC (a public channel on freenode) since it is more easily managed (people can choose to join the channel instead of an admin managing an email list) and discussion about the incident can happen in that channel.

For checking alerts we can choose between Grafana and Alertmanager. If we use Grafana we can define alerts for all our metric sources which might include Google Stackdriver for K8s cluster monitoring. Alertmanager would only integrate with Prometheus and we would need to run an alertmanager for every K8s cluster.

Neither Grafana nor Alertmanager have built-in support for IRC notifications. We’d need to investigate a bridge like irccat which we would need to run ourselves.

As alert conditions I suggest the following

• Alert when the average block import rate of a node deviates too much from one per minute. The margin of error should be smaller for averages over larger time windows.
• Alert when a node is not connected to any peer
• Alert when kubernetes pods are not running
• Alert when kubernetes containers restart

[CodeSandwich]

Grafana alerts look extremely useful. IRC seems completely alienated though, I couldn't find any reasonable way to easily integrate it, neither Grafana nor any of its integratables supports IRC. Gmail can't do it either, not even with addons. Riot.im can integrate with IRC, but not with Grafana or email. The irccat looks like the most reasonable solution, requires some extra steps, but should do the job.

Is it a good idea to make IRC our official public notification platform? Nowadays it really looks like a dead technology.

[geigerzaehler]

Is it a good idea to make IRC our official public notification platform? Nowadays it really looks like a dead technology.

I wouldn’t call it dead technology. But the need for DIY integration is a real downside. We should investigate how easy email would be. Maybe a Google Group makes sense. Then it is easier to manage it.

[geigerzaehler]

@CodeSandwich and I discussed this. We think it’s best to have the alerting on Grafana and notify us via an email group. We’ll start with alerts for low block import rate all nodes to check network health and alerts on whether all nodes that we run are up. We’ll document this in the repo for now and then integrate it into terraform later.

[CodeSandwich]

Possible future alerts:

• Any node in the cluster is down
• Any node in the cluster has less than 4 peers (the minimum guaranteed by the size of the cluster)
• Blocks are mined too quickly (this may be monitored indirectly with block import rate, but adding the last block mining time would be more reliable)
• Blocks are mined too slowly (this is going to be covered indirectly with the low import rate, but again last block mining time would be more reliable)
• Too many invalid transactions are being proposed (possible DDoS attack)

Many false alerts can be muted by disabling some or all the checks while the node is in sync mode: radicle-dev/radicle-registry#402

To consider: the email group should be completely closed for all senders except this one Grafana instance.

[CodeSandwich]

The group is live: https://groups.google.com/forum/#!forum/radicle-registry-ffnet-cluster-alerts

[CodeSandwich]

@geigerzaehler Some of the steps to complete this goal have been completed, the others have been split into smaller issues. Can we close it now?