maechler / ml4iiot

A framework to implement Machine Learning methods for time series data.

Machine Learning 4 IIoT

This is the repository of ml4iiot a framework to implement Machine Learning methods for time series data.

Example: Autoencoder on FFT of IMS Bearing Data Set

This example shows an autoencoder being trained on the Fast Fourier transform of the acceleration of bearing 1 of the IMS Bearing Data Set. The configuration of this experiment can be found in config/ims_bearing_data_set_autoencoder.yaml. The autoencoder learns (adjusts its weights) and predicts until the 2004-02-17 and after that does only predictions anymore (see do_fit). The results show that the upcoming crash shows up in an increasing reconstruction error of the autoencoder.

• Run IMS Bearing Data Set Autoencoder example: python ml4iiot/cli_runner.py -c config/ims_bearing_data_set_autoencoder.yaml
• Plot acceleration of bearing 1: python ml4iiot/plot_csv.py -p out/YYYY_MM_DD/ims_bearing_data_set/HH_MM_SS_csv_output.csv -itime acceleration_bearing_1
• Plot reconstruction error: python ml4iiot/plot_csv.py -p out/YYYY_MM_DD/ims_bearing_data_set/HH_MM_SS_csv_output.csv -itime absolute_reconstruction_error_moving

Architecture

We abstract the entire process into a pipeline consisting of different processing steps. Each pipeline must have an input, an algorithm and an output step. Before and after the algorithm step there can be any number of pre- or post-processing steps.

A pipeline can configured using YAML or JSON files and can be started with the CLI runner.

The main abstraction between individual pipeline steps are pandas DataFrames. Each step can add additional columns to the dataframe but should not alter existing columns.

Development

Setup virtualenv

pip install virtualenv;
python -m virtualenv env;
source env/bin/activate;

Install dependencies

pip install -r requirements.txt;
pip install -e .;

Testing

Run unit tests with: python -m unittest discover tests/*

Leave virtualenv

• deactivate

CLI runner

python ml4iiot/cli_runner.py -c config/your_config.yaml

Performance

Resample the training data in advance and use integer timestamps instead of formatted date strings to speed up trainings. Use the following commands to profile your code:

python -m cProfile -o out/cli_runner.profile ml4iiot/cli_runner.py -c config/your_config.yaml

snakeviz out/cli_runner.profile 

Docker

version: '3'
services:
  ml4iiot:
    image: still/to/be/published
    volumes:
      - ./ml4iiot/config:/usr/src/ml4iiot/config

Pipeline configuration

A pipeline always consists of an input adapter, an output adapter as well as an algorithm in between.

pipeline:
  input:
    class: ml4iiot.input.csv.CsvInput
    config:
      windowing_strategy:
        class: ml4iiot.input.windowing.CountBasedWindowingStrategy
        config:
          window_size: 100
          stride_size: 100
          batch_size: 20000
      delimiter: ','
      csv_file: /path/to/your/data.csv
      index_column: time
      columns:
        time:
          type: datetime
          datetime_format: timestamp
        sensor_value: float

  algorithm:
    class: ml4iiot.algorithm.stochastic.average.ExponentialWeightedMovingAverage
    config:
      column_mapping:
        sensor_value: sensor_value_average

  output:
      class: ml4iiot.output.compound.CompoundOutput
      config:
        output_adapters:
          - class: ml4iiot.output.std.StdOutput
            config:
              show_columns_progress:
                - column: index
                - column: sensor_value
                - column: sensor_value_average
          - class: ml4iiot.output.plot.PlotOutput
            config:
              show_plots: True
              figures:
                  plots:
                    - column: sensor_value
                      color: blue
                    - column: sensor_value_average
                      color: red

Inputs

• ml4iiot.input.csv.CsvInput
• ml4iiot.input.kafka.KafkaInput

Windowing strategies

• ml4iiot.input.windowing.TimeBasedWindowingStrategy
• ml4iiot.input.windowing.CountBasedWindowingStrategy

Algorithms

• ml4iiot.algorithm.stochastic.average.ExponentialWeightedMovingAverage
• ml4iiot.algorithm.stochastic.average.ExponentialWeightedMovingMinMaxAverage
• ml4iiot.algorithm.stochastic.mad.Mad
• ml4iiot.algorithm.stochastic.zscore.ModifiedZScore
• ml4iiot.algorithm.clustering.dbscan.DBSCAN
• ml4iiot.algorithm.autoencoder.fullyconnected.FullyConnectedAutoencoder
• ml4iiot.algorithm.autoencoder.cnn.CNNAutoencoder
• ml4iiot.algorithm.autoencoder.cnn.BottleneckCNNAutoencoder
• ml4iiot.algorithm.autoencoder.lstm.ReconstructionLSTMAutoencoder
• ml4iiot.algorithm.autoencoder.lstm.PredictionLSTMAutoencoder
• ml4iiot.algorithm.autoencoder.variational.VariationalAutoencoder

Processing

• ml4iiot.processing.normalization.MinMaxScaler
• ml4iiot.processing.smoothing.MovingExponentialSmoothing
• ml4iiot.processing.transform.Average
• ml4iiot.processing.transform.StandardDeviation
• ml4iiot.processing.transform.Minimum
• ml4iiot.processing.transform.Maximum
• ml4iiot.processing.transform.FastFourierTransform
• ml4iiot.processing.control.SkipDataFrame

Outputs

• ml4iiot.output.compound.CompoundOutput
• ml4iiot.output.std.StdOutput
• ml4iiot.output.plot.PlotOutput
• ml4iiot.output.csv.CsvOutput
• ml4iiot.output.kafka.KafkaOutput
• ml4iiot.output.config.ConfigOutput