jjthomas / rule_engine

Anomaly classification with rules

Rule Engine

Rule Engine (RE) creates an interpretable anomaly classifier from many one-feature and two-feature decision rules. It works natively on labeled, categorical training data, and automatically discretizes continuous features. It searches over all rules of the forms if (F1 = X) => anomaly and if (F1 = X && F2 = Y) => anomaly, where F1 and F2 are features and X and Y are possible values for those features. RE creates a classifier out of all rules classifying at least c training examples with at least p precision, where c and p are provided by the user. The classifier classifies a test example as an anomaly if any of its rules fire. RE can prune overlapping rules to improve overall precision and display rules in an easily understandable format.

RE is fast, with a high-performance C++ backend and experimental GPU and FPGA acceleration available. It can scale to large datasets with ~1000 features.

See examples/rookies.py for the key APIs and example usage.

A presentation on this work is available here: https://cs.stanford.edu/~jjthomas/dataai-pres.pdf.

Installation

On macOS, install llvm with brew install llvm.

Install rule_engine with <FLAG> pip install . from the top of this repo, where <FLAG> can be empty (CPU), GPU=1 or FPGA=1 depending on the desired accelerator. If there are complaints about missing Python.h, you may need to install the package python3-dev or python3-devel. The GPU build requires the nvcc compiler. Details on the FPGA build are below.

Run our example from the examples directory with python3 rookies.py.

FPGA Setup

FPGA-accelerated installation must be performed on an Amazon F1 instance with a recent FPGA AMI (tested with Centos AMI 1.9.1). Before running pip install, clone the F1 SDK from https://github.com/aws/aws-fpga. Run source sdk_setup.sh from the top of aws-fpga. Then follow the XDMA driver installation instructions at aws-fpga/sdk/linux_kernel_drivers/xdma/xdma_install.md. Set the environment variable F1_SDK to the path to the aws-fpga/sdk directory in the .bashrc or equivalent.

If you see an error during pip install about the static keyword in the AWS SDK, simply delete the keyword from the offending location and everything should work.

To run a Python script using rule_engine with FPGA acceleration, instead of directly calling python, use fpga_python, e.g. fpga_python rookies.py.

To see the RTL for the FPGA image and to actually build it yourself, check out https://github.com/jjthomas/DataCubeFPGA.

Example Data

We include an example dataset examples/Rookies.csv, which includes rookie-year stats for all NBA players through 2017. The class variable "IFAS" indicates whether the player ever became an all-star in their career.

Other anomaly detection datasets are available here: https://github.com/GuansongPang/anomaly-detection-datasets.