intuit / foremast-brain

Foremast-brain is a component of Foremast project.

Foremast Brain

Foremast-brain makes health judgments of Foremast, a service health detection and canary analysis system for Kubernetes. There are two main criteria that Foremast-brain evaluates:

1. Check if the baseline and current health metric have the same distribution pattern.
2. Calculate the historical model and detect current metric anomalies.

Foremast-brain will make a judgment, Healthy or Unhealthy, based on the evaluation result.

Please check out the architecture and design for more details.

How to use

Overwriting default algorithm and parameters

There are multiple sets of parameters that can be overwritten.

Machine learning algorithm related parameters -- used for post-deployment use cases

• ML_ALGORITHM -- Algorithm which you want to run. Please refer to AI_MODEL for all the supported algorithms
• MIN_HISTORICAL_DATA_POINT_TO_MEASURE -- Minimum historical data points size
• ML_BOUND -- Measurement is upper bound, lower bound or upper and lower bound
• ML_THRESHOLD -- Machine learning algorithm threshold

Performance, fault-tolerant related parameters

• MAX_STUCK_IN_SECONDS -- Max process time until another foremast-brain process will take over and reprocess
• MAX_CACHE_SIZE -- Max cached model size

Pairwise algorithm parameters -- used for pre-deployment use cases

• ML_PAIRWISE_ALGORITHM -- There are multiple options: ALL, ANY, MANN_WHITE, WILCOXON, KRUSKAL, etc.
• ML_PAIRWISE_THRESHOLD -- Pairwise algorithm threshold
• MIN_MANN_WHITE_DATA_POINTS -- Minimum data points required by Mann-Whitney U algorithm
• MIN_WILCOXON_DATA_POINTS -- Minimum data points required by Wilcoxon algorithm
• MIN_KRUSKAL_DATA_POINTS -- Minimum data points required by Kruskal algorithm

How to make changes:

You can add algorithm names and different parameters. Please refer foremast-brain for details.

The following is an example of ES_ENDPOINT:

env:
- name: ES_ENDPOINT
  value: "http://elasticsearch-discovery.foremast.svc.cluster.local:9200"

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

License

This project is licensed under the Apache License - see the LICENSE file for details

foremast-brain's Issues

Make master-dev branch has all the dev and master features

Tag existing master branch and merge dev to master

add notebook for one of foremast-brain ml algm

Maybe use Pipenv instead of a requirements.txt?

A requirements.txt cannot track versions of transitive dependencies without explicit freezing which makes for the possibility of dependency drift and unpredictible builds. It also leaves it up to each developer to use virtualenvs, which can make for a "it works on my machine" type scenario to occur with specific dependency versions.

The current officially recommended packaging tool from Python.org is Pipenv and it manages quite a few of these issues with lockfiles and automatic virtual environments. It can also generate a requirements.txt on the fly to fit in with existing build pipelines. I think it is something worth considering, although it isn't pressing until a transitive dependency breaks on an update and causes an issue.

This is a talk from the most recent pycon about the tool, which might answer more questions than I can off the top of my head.

Multi-Metric support

We'd like to support multiple health metrics for example error, latency, volume, cpu/memory/io/network related metric
The rough idea is

1. two metrics: bivariable Normal distribution
   2.two or metrics: LSTM deep learning, etc.

create makefile to add pylint check

Fix build status inconsist

https://www.travis-ci.org/intuit/foremast-brain status is success but https://github.com/intuit/foremast-brain readme status is failed

this appears on dev branch.