sisl / verticalcas

VerticalCAS

This repository has three main components:

• Generate MDP table policy
• Compress MDP table policy by training a neural network
• Visualize MDP and neural network policies

Generate MDP Policy

Required Julia Packages: Printf, POMDPs, POMDPModelTools, LocalFunctionApproximation, GridInterpolations, Distributed, SharedArrays, StaticArrays, HDF5

Tested with Julia v1.0

The policy is generated in parallel via Julia by running julia -p NUM_PROCS genTable.jl in the GenerateTable folder, where NUM_PROCS is the number of processors you want to use. The top of genTable.jl specifies where the table should be written to as an HDF5 file.

Train Neural Network

Required Python Packages: numpy, h5py, theano, keras

Tested with Python v2.7

After generating the table, the table needs to be formatted into training data for the neural network. To do this, run python genTrainingData.py in the GenerateNetworks folder. The top of the file specifies the MDP table policy folder and the format for the resulting training data files. Note that the table is split into separate files, one for each previous advisory, which allows separate networks to be trained for each previous advisory.

Next, run python trainVertCAS.py PREV_ADV, where PREV_ADV is the index of the previous advsiroy you want to train. Options at the top of the file allow you to specify where the training data is stored and where the .nnet files should be written. There are additional options for the user to specify additional setting for network training.

Visualize the Policies

Required Julia Packages: GridInterpolations, Interact, PGFPlots, Colors, ColorBrewer, HDF5

Tested with Julia v0.6

After generating MDP policies and training neural networks, the policies can be visualized. There is an example Jupyter notebook in the PolicyViz folder that shows how the policies can be interactively visualized. I have found that Julia v1.0 has problems showing the interactive features, but Julia v0.6 seems to work correctly.

verticalcas's Issues

Some questions about the Generate MDP Policy

Sorry，I have some questions.
When I use gentable. Jl, here is the problem：

ArgumentError: Package ParallelValueIteration not found in current path:

• Run import Pkg; Pkg.add("ParallelValueIteration") to install the ParallelValueIteration package.

Stacktrace:
[1] require(into::Module, mod::Symbol)
@ Base .\loading.jl:871
[2] eval
@ .\boot.jl:360 [inlined]
[3] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
@ Base .\loading.jl:1094

Thank you!

Question about constants.jl

Why the NON-compliant COC speed is (-100.0,100.0) instead of (-∞,-100)U(100,∞),Thank you for your time and assistance.

# Velocity range where aircraft is NON-compliant with advisory 
velRanges = Dict(COC=>(-100.0,100.0),
                DNC=>(0.0,100.0),
                DND=>(-100.0,0.0),
                DES1500=>(-25.0,100.0),
                CL1500=>(-100.0,25.0),
                SDES1500=>(-25.0,100.0),
                SCL1500=>(-100.0,25.0),
                SDES2500=>(-41.67,100.0),
                SCL2500=>(-100.0,41.67))

PolicyViz does not display any images

Hi there,

I'm trying to use PolicyViz to plot the intended geometric regions in the VerticalCAS input space for various advisories, like COC, etc. I was able to make some changes to modernize the code to Julia 1.0+ and install dependencies to get it running, but all I can see is the interactive dialog to change parameters. No images are displayed, but I also don't see any errors logged to the console or elsewhere. How can I visualize the MDP policies (and, perhaps later, the trained neural network policies)?

VerticalCAS fails to reproduce MDP results when training

I'm attempting to retrain VerticalCAS neural networks but my training is proving to be remarkably unstable. I've been working with the VerticalCAS repo for a few months and in the last few weeks have been unable to duplicate the behavior of the MDP (image below, generated by my own code since I was unable to get PolicyViz to work as documented in #3)

I have, in the past, been able to successfully retrain an approximation of the MDP using a neural network using (relatively) standard VerticalCAS code (photo below). This image was the policy after only one epoch, and I recall training this behavior to have been fairly reliable (happened > 50% of the time I tried to train). Additionally, losses were small and accuracy was high from early on in training: one value I saw was loss: 0.0463 - accuracy: 0.9569

I've since been unable to reproduce this behavior and get nonsense policies with terrible loss and accuracy (loss: 61075845120.0000 - accuracy: 0.5324). I think these massive losses indicate some sort of instability when training, but I haven't seen them until a few weeks ago. This happens every time I try to train.

I did have to make some modifications to the code to get it to run on modern Python (3.10.12) and Tensorflow (2.14.0), detailed below. Note we're both using M1 MacBook Pros, but again this wasn't a problem a few months ago.

• changing some indexing in GenTrainingData.py to // from / for integer division
• using the AsymMSE from the dasc branch or HorizontalCAS repo (they appear to be the same) so the Theano dependency goes away
• removing init='uniform' when initializing Dense layers

However, I had made all these changes a few months ago when I was able to successfully retrain, and now nothing I've tried seems to work. Another member of my lab has also tried to retrain these networks and has had the same issues as I have.

Do you have any suggested resolution to these issues? And do others experience these sorts of issues when attempting to retrain?

VerticalCAS fails to learn the score table policy

I'm trying to retrain VerticalCAS on my own machine but the trained behavior seems to converge to much simpler policies that just linearly partition the input space and do not match the score table's original behavior (see images below, one of which is from @kjulian3's DASC '19 paper on VerticalCAS and HorizontalCAS). I couldn't get the provided Julia visualization code to work (see #3), so I wrote my own in Python, which seems to work well since using it to plot the score table does match the image from the DASC '19 paper. Note that training does seem to converge; I'm seeing the values plateau for both loss around 0.0042 and accuracy around 0.9826.

I've tried to train using both the master and dasc branches, though there are issues with tensorflow 1 vs 2 compatibility so I had to modify some code locally. Are there any other versions of this code or alternative training approaches that can reproduce the intended behavior? I can't think of anything else to try.

Score table behavior, from DASC '19:

Score table behavior, plotted with my code:

Trained policy after 100 training epochs on master (dasc branch with tensorflow v2 adaptations is very similar):