sisl / automotivesaferl

Training and evaluation scripts for applying formal methods and reinforcement learning to autonomous driving problems.

Safe Planning Under Uncertainty for Autonomous Driving

Experiments on safe planning under uncertainty for autonomous driving. This code base combines reachability analysis, reinforcement learning, and decomposition methods to compute safe and efficient policies for autonomous vehicles.

Reference: M. Bouton, A. Nakhaei, K. Fujimura, and M. J. Kochenderfer, “Safe reinforcement learning with scene decomposition for navigating complex urban environments,” in IEEE Intelligent Vehicles Symposium (IV), 2019.

Installation

For julia version >1.1, it is recommended to add the SISL registry and JuliaPOMDP registry first:

pkg> registry add https://github.com/sisl/Registry
pkg> registry add https://github.com/JuliaPOMDP/Registry
pkg> add https://github.com/MaximeBouton/PedCar.jl
pkg> add https://github.com/MaximeBouton/AutomotiveSafeRL

To install all the dependencies manually, run the following in the Julia REPL:

using Pkg
Pkg.add(PackageSpec(url="https://github.com/sisl/Vec.jl"))
Pkg.add(PackageSpec(url="https://github.com/sisl/Records.jl"))
Pkg.add(PackageSpec(url="https://github.com/sisl/AutomotiveDrivingModels.jl"))
Pkg.add(PackageSpec(url="https://github.com/sisl/AutoViz.jl"))
Pkg.add(PackageSpec(url="https://github.com/sisl/AutoUrban.jl"))
Pkg.add(PackageSpec(url="https://github.com/sisl/AutomotiveSensors.jl"))
Pkg.add(PackageSpec(url="https://github.com/JuliaPOMDP/RLInterface.jl"))
Pkg.add(PackageSpec(url="https://github.com/JuliaPOMDP/DeepQLearning.jl"))
Pkg.add(PackageSpec(url="https://github.com/sisl/AutomotivePOMDPs.jl"))
Pkg.add(PackageSpec(url="https://github.com/MaximeBouton/POMDPModelChecking.jl"))

Folder structure

• src/ contains the implementation of the safe RL policy and the decomposition method.
• RNNFiltering/ contains data_generation and training_script for the ensemble RNN belief updater
• training_scripts/ contains training scripts for the safe RL and RL policies
• evaluation/ contains evaluation scripts to evaluate RL, safe RL, and baseline policies.
• notebooks/ contains jupyter notebook for visualization and debugging.

Code to run

To visualize any of the policy use notebooks/interactive_evaluation.ipynb

For a detailed description of the evaluation scenarios run notebooks/evaluation_scenarios.ipynb

Other notebooks are used for prototyping and debugging.

All scripts used to run the experiments in the paper are available, most of them have command line arguments. Check those arguments to see what can be changed.

Solving for the safety mask using Model Checking

Run training_scripts/pedcar_vi.jl to compute the safety mask using value iteration. This part is computationally expensive and requires parallelization over many cores.

Training an RL agent

Run training_scripts/pedcar_dqn.jl to train an RL agent with or without the safety mask.

Training the belief updater

First, run the RNNFiltering/generate_dataset.jl to creat synthetic data to train the RNN updater on. Then run RNNFiltering/bagging_training.jl to train one RNN. Look at the bash script RNNFiltering/train.sh to check how to properly set the seed.

Evaluating the algorithm Run evaluation/evaluation.jl or evaluation/parallel_evaluation.jl to evaluate the algorithm (this requires a solved VI mask, trained RL policy and trained RNN filer).

Main Dependencies

• AutomotivePOMDPs.jl contains all the driving scenario and MDP models
• POMDPModelChecking.jl
• DeepQLearning.jl (Flux.jl backend)

automotivesaferl's Issues

interactive_evaluation.ipynb LoadError: UndefVarError: SceneRecord not defined

Hi,
After installing all the required packages I am having this error. Below is the screenshot of the notebook.

Any help with this will be really appreciated.
Thanks,
Aizaz Sharif

LoadError: UndefVarError: Vehicle not defined

Hello, I have a question and ask for your help. I encountered this error while installing the packages(AutomotiveDrivingModels). Below is a screenshot of the notebook.




Looking forward to your reply !
Thanks!

ERROR : add https://github.com/MaximeBouton/AutomotiveSafeRL but exist file.Delete the file not work!

(@v1.6) pkg> add https://github.com/MaximeBouton/AutomotiveSafeRL
Updating git-repo https://github.com/MaximeBouton/AutomotiveSafeRL
ERROR: expected the file src/AutomotiveSafeRL.jl to exist for package AutomotiveSafeRL at F:\C_TEMP\user_temp\jl_QLzXut
Then i delete AutomotiveSafeRL at F:\C_TEMP\user_temp\jl_QLzXut` .
But it does not work!

(@v1.6) pkg> add https://github.com/MaximeBouton/AutomotiveSafeRL
Updating git-repo https://github.com/MaximeBouton/AutomotiveSafeRL
ERROR: expected the file src/AutomotiveSafeRL.jl to exist for package AutomotiveSafeRL at F:\C_TEMP\user_temp\jl_PVMB7g

interactive_evaluation.ipynb issue

While running the first cell:
"using Revise
includet("../evaluation/helpers.jl")
includet("../src/render_helpers.jl")
"
and after going through some package installation issues i got the following error:

┌ Info: Precompiling POMDPModelChecking [abefb91b-a28c-5ab9-9bd9-026e532d7b0e]
└ @ Base loading.jl:1192
┌ Error: evaluation error
│ mod = Main
│ ex = mutable struct InterpolationOverlay{M} <: (SceneOverlay where M <: Union{CarMDP, PedMDP, PedCarMDP})
#= /home/munir/Documents/workspace/AutomotiveSafeRL-master/src/render_helpers.jl:227 =#
verbosity::Int
#= /home/munir/Documents/workspace/AutomotiveSafeRL-master/src/render_helpers.jl:228 =#
color::Colorant
#= /home/munir/Documents/workspace/AutomotiveSafeRL-master/src/render_helpers.jl:229 =#
font_size::Int
#= /home/munir/Documents/workspace/AutomotiveSafeRL-master/src/render_helpers.jl:230 =#
id::Int
#= /home/munir/Documents/workspace/AutomotiveSafeRL-master/src/render_helpers.jl:231 =#
mdp::M
#= /home/munir/Documents/workspace/AutomotiveSafeRL-master/src/render_helpers.jl:232 =#
models::Dict{Int64, DriverModel}
#= /home/munir/Documents/workspace/AutomotiveSafeRL-master/src/render_helpers.jl:233 =#
ped_id::Int64
#= /home/munir/Documents/workspace/AutomotiveSafeRL-master/src/render_helpers.jl:234 =#
car_id::Int64
#= /home/munir/Documents/workspace/AutomotiveSafeRL-master/src/render_helpers.jl:235 =#
obs::Scene
end

File missing

I am trying to install the https://github.com/MaximeBouton/AutomotiveSafeRL from here as per the read me file but I am getting this error.

expected the file src/AutomotiveSafeRL.jl to exist for package AutomotiveSafeRL at /tmp/jl_PLWsWQ

No such file exists in the repository.

evaluation_scenarios.ipynb issue

While running the include section (at the very beggining), it looks like there are both TensorBoardLogger and RLInterface export the method "reset" so there's going to be a method conflict.
Here's the warning that pops up before the error
WARNING: both TensorBoardLogger and RLInterface export "reset!"; uses of it in module DeepQLearning must be qualified

The Error:
UndefVarError: reset! not defined

Stacktrace:
[1] getproperty(::Module, ::Symbol) at ./sysimg.jl:13
[2] top-level scope at none:0
[3] include at ./boot.jl:317 [inlined]
[4] include_relative(::Module, ::String) at ./loading.jl:1044
[5] include(::Module, ::String) at ./sysimg.jl:29
[6] include(::String) at ./client.jl:392
[7] top-level scope at In[10]:3

My Platform details:
Ubuntu 18.04
Julia Version 1.0.4 (2019-05-16)
conda 4.6.14
I do have Tensorflow installed for Python..but I never used it on Julia.