e9link / ai-graph

Train an artificial intelligent model to recognize abnormal traffic graphs or KPI graphs. Great for performance or IT engineers to figure out abnormal graphs from thousands of traffic graphs like cacti or KPI graphs from telecom OSS.

AI-Graph

The project is to train artificial intelligent models to recognize abnormal traffic graphs or KPI graphs. It can be used by performance or IT engineers to figure out abnormal graphs from thousands of traffic graphs like cacti or KPI graphs from telecom OSS.

First release, ai_cacti_test.py with pre-trained model pm_graph_variables5.ckpt identifies three types of graph: 1) Sudden traffic drop graphs indicating outage; 2) Plateau graphs indicating capacity limitation; 3) Normal graphs which exclude outage and plateau graphs.
It uses five conventional neural network layers and one full connected layers as follow,

Installation Procedure

1. Install python3.

2. Install tensorflow 1.4. Tensorflow versions higher than 1.4 might have compatible issues.
   

    sudo apt-get install python3-pip python3-dev python-virtualenv
    virtualenv --system-site-packages ~/venvs/tensorflow
    source ~/venvs/tensorflow/bin/activate
    sudo pip3 install tensorflow==1.4

3. Install required python3 module.
   

    sudo pip3 install pysftp
    sudo pip3 install numpy
    sudo pip3 install python36-tkinter
    sudo pip3 install matplotlib

4. Download AI_Graph_master.zip from Github and unzip to local AI-Graph-master directory.
   

5. Download the following two files(too big, cannot upload to github) and put into AI-Graph-master directory.
   

    pm_graph_variables5.ckpt.data-00000-of-00001
https://drive.google.com/open?id=1nhWdIWnYf6ywCWPpN3OonRb_yNVgu0mT

    gi-imperil.npy

https://drive.google.com/open?id=1PKVtxquGk_FxKIKN1WdWboc8OKvIntGz

5. Install nmidDataExport v1.1.0.
   https://www.urban-software.com/products/nmid-plugins/nmiddataexport/

6. Replace setup.php files in the nmiddataexport directory. I change setup.php to export 24h graph data.
   

7. In cacti server, select Console/Devices/Graph Lists/Graph management/Automated Export - Add to Export. Then Cacti will output 24h graph data to /usr/share/cacti/plugins/nmidDataExport/export/

Running

1. Test any cacti csv files in .\pm_graph\data\test
       python3 ai_cacti_test.py

     CSV file format is shown as follow. Title and Step field are optional. It must have Date column to let ai_cacti_test.py to extract in/out data.

    *Title: 'Great Plains GPC/IMPRENEETH/1023 PRI' *
    Step: 60
    Total Rows: 1440
    Graph ID: 2945
    Host ID: 30

    Date                   In   Out
    5/2/2018 9:31    1.6   4.83

2. Test nmidDataExport files in cacti server at /usr/share/cacti/plugins/nmidDataExport/export/. ai_cacti_test.py sftp the csv files in cacti server at /usr/share/cacti/plugins/nmidDataExport/export/ and checks which one is abnormal.
   --s specify the cacti server name.
       python3 ai_cacti_test.py --s cacti

3. You can put the new 24h training data in .\pm_graph\data\normal, .\pm_graph\data\outage, .\pm_graph\data\plateau and run the following command to tune the model.
       python3 ai_cacti.py --d ./data/ --v pm_graph_variables5.ckpt