sensorsini / cartpolesimulation

This repository contains CartPole simulator with its GUI, implemented controller (LQR) and generator of random desired position trace. It also contains files to train and test RNN predicting future states of a CartPole.

README

Tutorial

Check our tutorial which will guide you through installation and explain basic functionalities!

A lengthy explanation of the code is provided in this video series.

Installation

Get the code from Github:

git clone --recurse-submodules https://github.com/SensorsINI/CartPoleSimulation.git

Create conda environment with

conda create -n CartPoleSimulation python=3.8
conda activate CartPoleSimulation

If SI_Toolkit (System Identification Toolbox) or Control Toolkit folders are empty, use these lines to pull all submodules:

git submodule update --init --recursive
git submodule update --recursive --remote

Preferable way to install python packages: pip install -r requirements.txt in a conda env or pip venv.

This file was created and tested on macOS, running on Linux and Windows may require some minor changes and/or installing some packages manually. However, the Simulator should work on all major OSes (macOS, Linux, Windows).

Note that you might want to uninstall the ptvsd package unless you use VS Code. This package enables debugging in VS Code, however causes proliferation of useless warnings if imported in another environment.

We created the requirements.txt on macOS with pip list --format=freeze > requirements.txt. It required some minor manual corrections (deleting the package versions) to make the re-installation smooth. It should be also possible to create this file with conda list -e > requirements.txt.

Basic Operation

1. Run GUI: Run python run_cartpole_gui.py from top-level path.
2. Run a single experiment: Open run_data_generator.py. In the marked section, you can define your experiment. For a single run, set number_of_experiments = 1. Then open ./config.yml to modify controller-related parameters. For example, you can choose there whether MPPI should run with the true model ('predictor_ODE' == Euler Integrator) or with a neural network ('predictor_autoregressive_neural'). Once set, run python -m run_data_generator. It will create a new folder ./Experiment_Recordings/ and store a csv log file in it.

Run a Machine Learning Pipeline

You can use this repository to generate training data, train a neural network model using SI_Toolkit, and run the resulting controller.

1. Define all the parameters in config_data_gen.yml to your liking.
2. Run python -m SI_Toolkit_ASF.run.run_data_generator_for_ML_Pipeline. This will create a new experiment folder ./SI_Toolkit_ASF/Experiments/Experiment-[X]/. You will work in this folder from now on. Within there, in Recordings/ there is now a set of CSVs saved and split up into Train/Validate/Test folders. Also, copies of the current configuration files are saved there.
3. In ./SI_Toolkit_ASF/config_training.yml you can now set paths/path_to_experiment: to the newly created one. All pipeline-related scripts access this parameter to know which data to work on and where to store the models.
4. Normalize the data using python -m SI_Toolkit_ASF.run.Create_normalization_file. This creates a normalization file within the experiment folder set in step 3.
5. Train a model using either TensorFlow (default) or PyTorch. Specify the library to use in ./SI_Toolkit_ASF/config_training.yml. Type python -m SI_Toolkit_ASF.run.Train_Network -h for a list of parameters you can define. Some default values are set in the same config as in step 3 and can also be modified there. Now run the Train module with all parameter flags that you wish. You will want to specify the network architecture. Training will store a new model within a subfolder Models/ in the chosen experiment folder.
6. Test the model. Adjust the parameters in ./SI_Toolkit_ASF/config_testing.yml to select the correct test file and model. Run python -m SI_Toolkit_ASF.run.Run_Brunton_Test which selects the test run set in config and compares the model's predictions versus true model behavior in Brunton-style plots. You can again see all params with the flag -h. If the script breaks, set a smaller --test_len.
7. Run MPPI with the trained model. Define an experiment in run_data_generator.py, select "predictor_autoregressive_neural" in the top-level config file, and run python -m run_data_generator. The results can be replayed in GUI.

Structure:

The CartPole class in CartPole folder corresponds to a physical cartpole. It contains methods for setting cartpole parameters, calculating dynamical equation, drawing cartpole and many more.

To perform an experiment CartPole needs an "environment". This environment is provided with CartPole GUI - suitable to visualize dynamical evolution and impact of parameter change - and Data Generator, with which user can easily generate hours of experimental data. They can be started by running run_cartpole_gui.py and run_data_generator.py respectively.

The CartPole loads controllers from Controllers folder. One can specify in GUI or in Data Generator which controller should be used. Adding controllers is easy: They just have to fit the template provided in Control_Toolkit/Controllers/template_controller.py. CartPole-specific controllers can be defined in Control_Toolkit_ASF/Controllers If a file in Controllers folder is named controller_....py and contains the function with the same name (controller_...) it will be automatically detected and added to the list of possible controllers. The MPC controller can be used in combination with any of the optimizers in Control_Toolkit/Optimizers. You can define new optimizers as well

Operation hints:

While the documentation of parameters is still missing in Readme, they are extensively commented in CartPole/cartpole_model.py, GUI/gui_default_params.py and run_data_generator.py. Look inside for more details.

In the “Manual Stabilization” mode you can provide the control input (motor power related to the force acting on the cart) by hovering/clicking with your mouse over the lower chart. Due current Cart-Pole system parameters everything happens to fast to make it doable now. Try to change length of the pole and make motor power weaker in cartpole_model.py to make this task feasible.

In the “LQR-Stabilization” mode an automatic (LQR) controller takes care that the Pole stays in the upright position. You can provide the target position of the cart by hovering or clicking with your mouse over the lower chart. The same is true for do-mpc controller which is mpc implementation based on true cartpole ODE done with do-mpc python library

Quit button is provided because when launched from some IDEs (e.g. Spyder on Windows 10) the standard cross in the window corner may not work correctly.

The "CSV file name" text box is used for naming a file to be saved or to be loaded. The path is assumed relative to ./SI_Toolkit_ASF/Experiments/. If left empty while saving, the default name is given. If left empty while loading data, the latest experiment will be loaded.

Folding convention

Files regions are folded with #region #endregion syntax For Pycharm default, for Atom install

Parameter exploration with NNI

For intelligent parameter space exploration with NNI, we have 2 special files :

1. modeling/rnn_tf/search_space.json : Search space for parameter search
2. config.yml : Configuring the NNI experiments.

Step 1: In Modeling/TF/Train.py comment line:

    train_network()

and uncomment lines:

    nni_parameters = nni.get_next_parameter()
    train_network(nni_parameters)

Step 2 : nnictl create --config Modeling/TF/NNI/config.yml Step 3 : Open the url as displayed on terminal after Step 1

cartpolesimulation's Issues

Pass all controller attributes in constructor

• Currently
  • Controllers fetch configuration values directly from CartPoleSimulation config
• Proposed change
  • On controller instantiation, pass all required attributes as kwargs
• Rationale
  • The controller should not place assumptions on where its configuration is located. Instead, it is the job of each implementation to pass these values correctly.
  • Advantages
    • Less error-prone (no silent failures by fetching from the wrong config)
    • More easily visible which parameters the controller needs to be fully spec'd
    • Config management decoupled from controller

Controllers don't follow target position

After pulling the newest changes from master the controllers do not move the cart to the target position. Can you recreate this problem? I'm guessing the target position is not scaled correctly somewhere, since the newest changes on master address this topic.

controlDisturbance

I used controlDisturbance while training neural imitator to force the simulated cartpole to explore suboptimal trajectories. However this variable seems to also affect MPPI predictions at least if based on ODE predictor. This shouldn't probably be like this. One should double check it, maybe within a broader check of how adding noise and latency (also to "measurement") affects simulation vs predictors vs physical cartpole functioning. This behaviour might not be consistent.

Seed

There is a seed for data generator both at the top of config_data_gen and and the bottom of config. Bug or otherwise confusing.

Missing install commands

To make the environment operate I also had to add numba sympy pyyaml and tensorflow to the conda install list. Could you add those to the instructions?

Check if config_data_gen.controller is irrelevant in CartPole gym mode

Taken from a TODO in code

Streamline Random Seeding

Currently: Random number generation happening in different places w or without a well-specified RNG. Work through repo to ensure all random numbers are generated with well-spec'd seeds.

Change data type of state

Currently: Cart state is a namespace with angle, angleD, angleDD, position, positionD, positionDD

Decision Marcin & Frederik: Convert state representation to numpy array throughout the repo. Advantage: Performance gains for mppi parallelization and RNN computations. Disadvantage: Order of variables must be clearly documented.

The following variables are saved in the state array in alphabetical order: angle, angle.cos, angle.sin, angleD, angleDD, position, positionD, positionDD. The order will be documented in code.

Replace Python `exec` statements where possible

I discovered that there is a number of exec(str) statements that import from other modules. I discovered that import_module from importlib is a safer alternative, e.g.

Class = getattr(import_module("package.module"), "ClassName")