status-im / clusterviz Goto Github PK

Problem

We would like to monitor if connectivity of our cluster is correct.

Solution

I played with some graph algorithms and figure out that we can describe our cluster in three states:

1. green -- means that cluster forms a biconnected graph which means that even if one of our nodes goes down, our cluster still stays connected,
2. yellow -- our cluster forms a connected graph, however, there is a threat that in case of a single node failure it stops being connected,
3. red -- our cluster is disconnected.

Due to the fact that we run multiple Whisper nodes on a single server, connectivity should be checked on server and node levels (does a single server failure break the connectivity? does a single node failure break the connectivity?).

Ideally, we should do that for the whole cluster (both DO and GC fleets) and for each fleet separately. The latter in case one fleet is unavailable in some country or the DC goes down.