sisl / pomdpstresstesting.jl

Adaptive stress testing of black-box systems within POMDPs.jl

POMDPStressTesting.jl

Adaptive stress testing of black-box systems, implemented within the POMDPs.jl ecosystem.

See the documentation for more details.

Citation

If you use this package for research purposes, please cite the following:

@article{moss2021pomdpstresstesting,
  title = {{POMDPStressTesting.jl}: Adaptive Stress Testing for Black-Box Systems},
  author = {Robert J. Moss},
  journal = {Journal of Open Source Software},
  year = {2021},
  volume = {6},
  number = {60},
  pages = {2749},
  doi = {10.21105/joss.02749}
}

Interface

To stress test a new system, the user has to define the GrayBox and BlackBox interface outlined in src/GrayBox.jl and src/BlackBox.jl.

GrayBox Interface

The GrayBox simulator and environment interface includes:

• GrayBox.Simulation type to hold simulation variables
• GrayBox.environment(sim::Simulation) to return the collection of environment distributions
• GrayBox.transition!(sim::Simulation) to transition the simulator, returning the log-likelihood

BlackBox Interface

The BlackBox system interface includes:

• BlackBox.initialize!(sim::Simulation) to initialize/reset the system under test
• BlackBox.evaluate!(sim::Simulation) to evaluate/execute the system under test
• BlackBox.distance(sim::Simulation) to return how close we are to an event
• BlackBox.isevent(sim::Simulation) to indicate if a failure event occurred
• BlackBox.isterminal(sim::Simulation) to indicate the simulation is in a terminal state

Functions ending with ! may modify the Simulation object in place.

Solvers

Several solvers are implemented.

Reinforcement learning solvers

• MCTSPWSolver

Deep reinforcement learning solvers¹

• TRPOSolver
• PPOSolver

Stochastic optimization solvers

• CEMSolver

Baseline solvers

• RandomSearchSolver

Example

An example implementation of the AST interface is provided for the Walk1D problem:

• Julia source: test/Walk1D.jl
• Jupyter notebook: Walk1D.ipynb
• Descriptive tutorial-style write-up: walk1d.pdf (created using TeX.jl)

Installation

Install the POMDPStressTesting.jl package via:

] add POMDPStressTesting

Testing

To run the test suite, you can use the Julia package manager.

] test POMDPStressTesting

Contributing

We welcome contributions! Please fork the repository and submit a new Pull Request.

---

Package maintained by Robert Moss: [email protected]

¹ TRPO and PPO thanks to Shreyas Kowshik's initial implementation.

pomdpstresstesting.jl's Issues

Move CEM solver to CrossEntropyMethod.jl

Once CrossEntropyMethod.jl is in the Julia registry, then we can move the POMDPs.action implemented CEM solver from POMDPStressTesting.jl into the CrossEntropyMethod.jl package.

Oversized return value for get_action

The function get_action (for the DRL solvers) returns rand(distribution, policy.solver.action_size), but since distribution is already multivariate, this allocates and fills a square matrix with samples (of which only the first is relevant / used) -- I think the intended return value is rand(distribution). I can submit a pull request if that's preferred.

Register package with JuliaRegistries/General

Hold off until MCTS.jl and CrossEntropyMethod.jl are registered (dependencies).

Move to Github CI

Travis build badge is red now. Let's move to Github CI.

Difference and similarities between Adaptive Stress Testing packages

I try to learn about Adaptive Stress Testing and the "Stanford Intelligent Systems Laboratory" provides three different Toolboxes:

• https://github.com/sisl/AdaptiveStressTestingToolbox
• https://github.com/sisl/AdaptiveStressTesting.jl
• https://github.com/sisl/POMDPStressTesting.jl

Could you explain the differences and similarities between these packages or guide me where I should start to learn about the implementation of Adaptive Stress Testing?

Problems connecting to AutomotvieSimulator.jl

Hello, I am currently trying to implement an example of the POMDPStressTesting.jl with the environment of AutomotiveSimulator.jl. Unfortunately, I am very new to Julia, and I am struggling to find my mistake.

I tried to guide my attempts along the Walk1D.jl example.

What I wanted to achieve is a simple scenario with two vehicles where the adversarial tries to modify the initial speed of the vehicle and the distance metric is literally the distance between the two vehicles on the straight roadway.

This is the code I want to execute:

using POMDPStressTesting # this package
using Distributions      # for the Normal distribution
using Parameters         # for @with_kw default struct parameters
using AutomotiveSimulator
using AutomotiveVisualization

# initialize the simulation based on https://sisl.github.io/AutomotiveSimulator.jl/dev/tutorials/straight_roadway/

# * GrayBox Environment Setup

# https://nbviewer.jupyter.org/github/sisl/POMDPStressTesting.jl/blob/master/notebooks/Walk1D.ipynb#GrayBox.Simulation

@with_kw mutable struct AutoSimSetup
    roadway:Roadway = gen_straight_roadway(1, 2000.0)
    scene::Scene = Scene([
        Entity(VehicleState(VecSE2(10.0,0.0,0.0), roadway, 8.0), VehicleDef(), 1),
        Entity(VehicleState(VecSE2(50.0,0.0,0.0), roadway, 12.5), VehicleDef(), 2),
        ])
    models::Dict{Int, LaneFollowingDriver} = Dict{Int, LaneFollowingDriver}(
        1 => StaticLaneFollowingDriver(0.0), # always produce zero acceleration
        2 => IntelligentDriverModel(v_des=12.0), # default IDM with a desired speed of 12 m/s
    )
    timestep::Float64 = 0.1 # typical time step of the simulation
    nticks::Int64 = 100
    endtime::Float64 = timestep * nticks
    eventdistance::Float64 = 30.0
end

# stores simulation-related values
@with_kw mutable struct AutoSim <: GrayBox.Simulation
    params::AutoSimSetup = AutoSimSetup() # Setup Parameters
    distance_to::Float64 = 1000 # distance between the two vehicles
    init_velocity2::Float64 = 12.5
    scene::Scene =  Scene([
        Entity(VehicleState(VecSE2(10.0,0.0,0.0), params.roadway, 8.0), VehicleDef(), 1),
        Entity(VehicleState(VecSE2(50.0,0.0,0.0), params.roadway, init_velocity2), VehicleDef(), 2),
        ])
    time::Float64 = 0.0 # current time
    distribution::Distribution = Normal(12.5, 1)
end

GrayBox.environment(sim::AutoSim) = GrayBox.Environment(:init_velocity2 => sim.distribution)

function GrayBox.transition!(sim::AutoSim, sample::GrayBox.EnvironmentSample)
    sim.time += sim.params.timestep
    sim.init_velocity2 = sample[init_velocity2].value
    sim.scene = simulate(sim.scene, sim.params.roadway, sim.params.models, 1, sim.params.timestep)
    return logpdf(sample)::Real
end

# * BlackBox System Setup: https://nbviewer.jupyter.org/github/sisl/POMDPStressTesting.jl/blob/master/notebooks/Walk1D.ipynb#Black-box-System

function BlackBox.initialize!(sim::AutoSim)
    sim.params.roadway = gen_straight_roadway(1, 2000.0)
    sim.params.scene = Scene([
        Entity(VehicleState(VecSE2(10.0,0.0,0.0), sim.params.roadway, 8.0), VehicleDef(), 1),
        Entity(VehicleState(VecSE2(50.0,0.0,0.0), sim.params.roadway, 12.5), VehicleDef(), 2),
    ])
    sim.params.models = Dict{Int, LaneFollowingDriver}(
        1 => StaticLaneFollowingDriver(0.0), # always produce zero acceleration
        2 => IntelligentDriverModel(v_des=12.0), # default IDM with a desired speed of 12 m/s
    )
    sim.params.timestep = 0.1
    sim.params.endtime = 10.0
end

function BlackBox.distance(sim::AutoSim)
    return 40.0
end

BlackBox.isevent(sim::AutoSim) = false #extract_featuresdistance_to(2, sim.params.roadway, sim.params.scene, [1]) <= sim.params.eventdistance

BlackBox.isterminal(sim::AutoSim) = BlackBox.isevent(sim) || sim.time >= sim.params.endtime

function BlackBox.evaluate!(sim::AutoSim, sample::GrayBox.EnvironmentSample)
    logprob::Real = GrayBox.transition!(sim)
    d::real = BlackBox.distance(sim)
    event::Bool = BlackBox.isevent(sim)
    return (logprob::Real, d::Real, event::Bool)
end

# * AST Setup and Running: https://nbviewer.jupyter.org/github/sisl/POMDPStressTesting.jl/blob/master/notebooks/Walk1D.ipynb#AST-Setup-and-Running

function setup_ast(seed=0)
    # Create gray-box simulation object
    sim::GrayBox.Simulation = AutoSim()

    # AST MDP formulation object
    mdp::ASTMDP = ASTMDP{ASTSampleAction}(sim)
    mdp.params.debug = true # record metrics
    mdp.params.top_k = 10   # record top k best trajectories
    mdp.params.seed = seed  # set RNG seed for determinism

    # Hyperparameters for MCTS-PW as the solver
    solver = MCTSPWSolver(n_iterations=1000,        # number of algorithm iterations
                          exploration_constant=1.0, # UCT exploration
                          k_action=1.0,             # action widening
                          alpha_action=0.5,         # action widening
                          depth=sim.params.nticks) # tree depth

    # Get online planner (no work done, yet)
    planner = solve(solver, mdp)

    return planner
end

planner = setup_ast();

action_trace = search!(planner)

final_state = playback(planner, action_trace, sim->sim.x)

failure_rate = print_metrics(planner)

d3tree = visualize(planner) # re-runs the search to output the tree, then visualizes it

I am using the following packages with Julia 1.6.1 in Ubuntu 20.04

julia> import Pkg; Pkg.status()
     Project scpsLL v0.1.0
      Status `~/git/SCPS-LL/scpsLL/Project.toml`
  [0f7eb3e9] AdversarialDriving v0.1.0 `https://github.com/sisl/AdversarialDriving.jl#master`
  [6aa42d20] AutomotiveSimulator v0.1.1
  [0faf7fd0] AutomotiveVisualization v0.1.1
  [4c6ed407] BayesianOptimization v0.2.5
  [31c24e10] Distributions v0.23.12
  [bb4c363b] GridInterpolations v1.1.2
  [7073ff75] IJulia v1.23.2
  [a40420fb] LocalApproximationValueIteration v0.4.2
  [db97f5ab] LocalFunctionApproximation v1.1.0
  [8314cec4] PGFPlotsX v1.2.10
  [182e52fb] POMDPPolicies v0.4.1
  [e0d0a172] POMDPSimulators v0.3.12
  [6fc570d8] POMDPStressTesting v0.5.0 `https://github.com/sisl/POMDPStressTesting.jl#master`
  [a93abf59] POMDPs v0.9.3
  [d96e819e] Parameters v0.12.2
  [91a5bcdd] Plots v1.15.2
  [3065fe53] RLInterface v0.3.7 `https://github.com/JuliaPOMDP/RLInterface.jl#master`
  [71555da5] Reel v1.3.2

I have the following bug, where I don't quite understand how to solve the bug.

Exception has occurred: MethodError
MethodError: Cannot `convert` an object of type Nothing to an object of type Real
Closest candidates are:
  convert(:Type{T}, !Matched::Gray24) where T<:Real at /home/luttkule/.julia/packages/ColorTypes/6m8P7/src/conversions.jl:114
  convert(::Type{T}, !Matched::Gray) where T<:Real at /home/luttkule/.julia/packages/ColorTypes/6m8P7/src/conversions.jl:113
  convert(::Type{T}, !Matched::Base.TwicePrecision) where T<:Number at twiceprecision.jl:250
  ...

Stacktrace:
 [1] default_action(unused#::MCTS.ExceptionRethrow, mdp::ASTMDP{ASTSampleAction}, s::ASTState, ex::MethodError)
   @ MCTS ~/.julia/packages/MCTS/ww2qH/src/default_action.jl:3
 [2] action_info(p::MCTS.DPWPlanner{ASTMDP{ASTSampleAction}, ASTState, ASTSampleAction, typeof(POMDPStressTesting.AST.rollout), MCTS.RandomActionGenerator{Random._GLOBAL_RNG}, typeof(POMDPStressTesting.AST.go_to_state), Random._GLOBAL_RNG}, s::ASTState; tree_in_info::Bool)
   @ MCTS ~/.julia/packages/MCTS/ww2qH/src/dpw.jl:79
 [3] (::POMDPModelTools.var"#action_info#kw")(::NamedTuple{(:tree_in_info,), Tuple{Bool}}, ::typeof(POMDPModelTools.action_info), p::MCTS.DPWPlanner{ASTMDP{ASTSampleAction}, ASTState, ASTSampleAction, typeof(POMDPStressTesting.AST.rollout), MCTS.RandomActionGenerator{Random._GLOBAL_RNG}, typeof(POMDPStressTesting.AST.go_to_state), Random._GLOBAL_RNG}, s::ASTState)
   @ MCTS ~/.julia/packages/MCTS/ww2qH/src/dpw.jl:19
 [4] search!(planner::MCTS.DPWPlanner{ASTMDP{ASTSampleAction}, ASTState, ASTSampleAction, typeof(POMDPStressTesting.AST.rollout), MCTS.RandomActionGenerator{Random._GLOBAL_RNG}, typeof(POMDPStressTesting.AST.go_to_state), Random._GLOBAL_RNG}; return_tree::Bool, verbose::Bool)
   @ POMDPStressTesting ~/.julia/packages/POMDPStressTesting/zbluM/src/solvers/mcts.jl:80
 [5] search!(planner::MCTS.DPWPlanner{ASTMDP{ASTSampleAction}, ASTState, ASTSampleAction, typeof(POMDPStressTesting.AST.rollout), MCTS.RandomActionGenerator{Random._GLOBAL_RNG}, typeof(POMDPStressTesting.AST.go_to_state), Random._GLOBAL_RNG})
   @ POMDPStressTesting ~/.julia/packages/POMDPStressTesting/zbluM/src/solvers/mcts.jl:76
 [6] top-level scope
   @ ~/git/SCPS-LL/scpsLL/src/AutomotiveSimulator/pomdp-test.jl:109

In the default_action.jl of the MCTS package is the exception thrown on line 3:

default_action(:ExceptionRethrow, mdp, s, ex) = rethrow(ex)

I get the following expressions for s and mdp.

s
ASTState
  t_index: Int64 1
  parent: Nothing nothing
  action: Nothing nothing
  state: Nothing nothing
  hash: UInt64 0x037b19cd36727d90
  q_value: Float64 0.0
  terminal: Bool false

mdp
ASTMDP{ASTSampleAction}
  params: ASTParams
  sim: AutoSim
  sim_hash: UInt64 0x037b19cd36727d90
  t_index: Int64 2
  current_seed: Nothing nothing
  top_paths: DataStructures.PriorityQueue{Any, Float64, Base.Order.ForwardOrdering}
  metrics: POMDPStressTesting.AST.ASTMetrics
  dataset: Array{Any}0,
  rate: Float64 NaN
  predict: Nothing nothing

I assume that the error is that parent, action and state are all nothing in s

default_action is called in the dpw.jl file in the action_info function.

s:ASTState

goes into the function and is originally passed as initstate in the mcts.jl file of the POMDPStressTesting package.

initstate = AST.initialstate(mdp)

So something must be wrong in the initstate definition, but I don't know how to find out what.

Any help or idea is really appreciated.

Thanks and kind regards

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

Please fix the 2 doc missing elements

Missing docstring.
Missing docstring for episodic_figures. Check Documenter's build log for details.

Missing docstring.
Missing docstring for distribution_figures. Check Documenter's build log for details.

Connect Crux.jl deep RL solvers

Crux.jl already adheres to the POMDPs interface, so it should be a matter of wrapping a search! function around the solvers.

PR's welcome 🤗

Use Plots.jl instead of PyPlot and Seaborn

See https://github.com/sisl/POMDPStressTesting.jl/blob/master/src/visualization/figures.jl

Move TRPO/PPO to new package: PolicyOptimization.jl

The TRPO and PPO implementations are general enough to be in their own solver package in the POMDPs.jl ecosystem. I've already encapsulated these solvers into the DeepRL module.

Some TODOs:

• Extract AST specific code and generalize TRPO and PPO solvers
• Test on different POMDPs.jl example

Among other things to be appended to this list.

DRL solvers using hash as state value

Hi! I've been using your package, and I've run into an issue with the DRL solvers.

The function get_action passes the state into the neural network policy.μ, but this state is computed in convert_s to be the hash of the ASTState. My impression is that a hash value is not really a meaningful input to a NN, and it seems like it would invalidate much of the learning, effectively reducing the DRL algorithms to a random search.

If the GrayBox interface is extended to allow a Vector{Float64} state to be specified and stored in the ASTState at each update, this can be extracted in convert_s and passed into the NN. I've made those changes in my local copy to get things working, but perhaps there's a solution that's more in line with your vision for the package, in terms of genericity, etc. If you'd like, I can submit a pull request.