bbengfort / kahu

Replica management and monitoring service.

Kahu

Replica management and monitoring service.

About

Kahu is a service that manages replica configurations for my research. Each replica has a small client that uses the Kahu API to report its status as well as receive configuration updates. The Kahu web interface is used to monitor the replicas and push configuration changes for various experiments or applications.

Resources

• Dashboard template based on Start Bootstrap Admin (GitHub).
• Icons from Font Awesome v4.7.0.

kahu's Issues

inactive hosts can't send heartbeats

Right now the machine auth checks to make sure the replica is active; but this prevents inactive hosts from sending heartbeat messages. Instead the following should happen:

1. Create ActiveMachine vs Machine permission classes
2. Allow all machines regardless of active or not post to heartbeat
3. Allow only active machines to post to latencies
4. In neighbors, if the machine is inactive, then send back no targets

Implement user/machine token auth + permissions

Right now we're using X-API-KEY headers in kind of a janky way. Instead, we should subclass TokenAuthentication and provide for both user and machine user authentication.

Instead of doing "validate machine user" --- we should create a permissions class to define machine vs. user permissions to the API.

Add cluster for replica management

Add cluster to group replicas meaningfully.

Reset latency

Add ability to reset the latency between a source and a target as follows:

1. Archive the current latency distribution (either in new table or with an archived field)
2. Reset the current latency distribution to all zeros

Other requirements:

• Create an API endpoint on the LatencyViewSet to reset on request
• Reset in an HTML view?

active endpoint

Create an endpoint to active/deactivate replicas.

Overview map displays locations with inactive replicas

Overview map should only show the locations of active replicas.

Deploy new version of Kahu

Don't forget the new environment!

Also need to wipe and restore the database from scratch.

Update health to 5 minutes

Update online health to 5 minutes instead of 2 minutes.

fetch tokens

Create fetch tokens via username and password method for geonet to update instances.

geonet tests

Write tests for the geonet app.

1. Test the boto api by mocking boto
2. Test deserialization of describe instances responses
3. model tests and cacheing (only one model per replica, etc)
4. test apply and apply to regions
5. views tests (refresh is called on post, modified timestamps, etc).

simulator

Create a multiprocessing script that can simulate api requests from replicas for testing purposes.

AWS instances listing

Create a status mechanism to see what instances are running or not running on AWS.

Add a "promote" button to store the instance as a replica.

This will be one point of integration with GeoNet

Add replica detail

List processes running on servers

On heartbeat, list the running processes of interest; namely:

• alia
• honu
• daytimer

ping latencies in replica detail has weird ordering

Check the ordering of ping latencies in the replica detail page - it appears that more recent pings are at the bottom, not at the top.

replicas tests

Implement tests:

1. Geoip tests (mock maxmind api)
2. location is only looked up on no ip or when ip changed
3. concurrent latency update tests
4. recompute latency distribution tests
5. health timing tests
6. signals tests
7. views tests (login required, machine required, query string ordering)
8. only admins can fetch tokens from /api/replicas/tokens/
9. models tests
10. authentication tests
11. permissions tests on api (user, machine, active machine)
12. utils tests
13. only admins can create or update replicas from the api
14. on api: replica list, only active replicas are displayed

Updates to heartbeat (hostname, geoip)

Add hostname to heartbeat and do geoip lookup if the ip address has changed.

No error message on change password view

No error flashes when you're trying to change your password after forgot password and the passwords don't match or are too short.

promote instance to replica

Create a tool to quickly promote an instance to a replica via the geonet aws status view.

allow replicas to be created by geonet

By modifying the replica viewset to allow admin users to create and update replicas.

Add description field to replicas.

Add a description field to replicas that give text details about it (e.g. what machine its running on).

Ping latency bar chart

Instead of a table listing ping latencies, instead add a bar chart that shows average latency to each location.

Modify latency endpoint with neighbors action

Rather than return the neighbors list at GET /api/latency return it in GET /api/latency/neighbors.