sisl / aa120q

AA120Q Course Materials

AA120Q

Establishing Trust in Autonomous Systems

This package supports AA120Q: Establishing Trust in Autonomous Systems, offered at Stanford.

Lecture Notebooks

1. Introduction to Julia ^[Pluto]
2. Scientific Computing ^[Pluto]
3. Statistical Models ^[Pluto]
4. Learning ^[Pluto]
5. Simulation ^[Pluto]
6. Building Autonomous Systems ^[Pluto]
7. Robustness to Sensor Error ^[Pluto]
8. Analysis of Autonomous Systems ^[Pluto]
9. Case Studies of Autonomous Systems ^[Pluto]
10. Societal Impact of Autonomy ^[Pluto]

• Optional: Bayesian Networks ^[Pluto]

Assignments

1. Estimating π ^[Pluto]
2. Encounter Plotting ^[Pluto]
3. Simulation ^[Pluto]
4. Simple Collision Avoidance System ^[Pluto]
5. Collision Avoidance System Design ^[Pluto]
6. Analysis ^[Pluto]

• Optional Assignment: Bayesian Network Learning ^[Pluto]

Installation

1. Install Julia version 1.5.3 from https://julialang.org/downloads
   • Add julia to the command line PATH: This will make julia available anywhere on the command line.
     • Windows

       Follow these instructions (https://helpdeskgeek.com/windows-10/add-windows-path-environment-variable/) and add the Julia `bin` directory to your User PATH environment variable (replacing with your actual Julia installation location).

       C:\<PATH_TO_JULIA>\Julia-1.5.3\bin\
       

     • Linux

       Edit your `~/.bashrc` to add the following line (replacing with your actual Julia installation location):

       export PATH=$PATH:/<PATH_TO_JULIA>/bin/
       

     • Mac OS X

       Open a terminal and run the following (this will create a`julia` alias and place it in `/usr/local/bin` which is already on the terminal path):

       sudo sh -c 'mkdir -p /usr/local/bin && ln -fs "/Applications/Julia-1.5.app/Contents/Resources/julia/bin/julia" /usr/local/bin/julia'
       

2. Install Git from https://git-scm.com/downloads
3. Open a terminal and run:

   git clone https://github.com/sisl/AA120Q
   cd AA120Q
   julia install.jl

4. Test the installation by opening julia and running:

] test AA120Q

Running Pluto

1. Open a terminal and run julia (or you can open the Julia application itself)
2. Inside Julia, run the following commands:

using Pluto
Pluto.run()

• You should see an http://localhost:#### url you can open in a browser (this will open Pluto!)

Troubleshooting

Post issues either here on GitHub or on Piazza.

aa120q's Issues

Cell 3 in 04 Lecture throws error

d = D[:SepalLength]
Plots.Histogram(d)

! LaTeX Error: File `luatex85.sty' not found.

Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: sty)

Enter file name: 
! Emergency stop.
<read *> 

l.2 \documentclass
                [tikz]{standalone}
End of file on the terminal!



Here is how much of LuaTeX's memory you used:
 11 strings out of 494693
 100000,89155 words of node,token memory allocated 188 words of node memory still in use:
   31 glue_spec, 1 dir nodes
   avail lists: 2:12,3:1
 3851 multiletter control sequences out of 65536+600000
 14 fonts using 562767 bytes
 10i,0n,7p,45b,8s stack positions out of 5000i,500n,10000p,200000b,100000s
!  ==> Fatal error occurred, no output PDF file produced!

Error saving as SVG
LaTeX error

Removing L from data table

I am removing L from the initial table because altitude layer can be inferred from airplane altitude in the state traces.
The docs need to be updated to reflect this.

Current TCAS model in AA120Q.jl issues a lot of advisories

When I try "evaluate(TCAS(), encounters)", this is the output:
CASEval(n_encounters: 500, n_advisories: 1366, n_NMACs: 2)

Make AA120Q a separate Julia package

I will be making AA120Q.jl a separate package so that students can easily use it with include AA120Q and so that when they get it with Pkg.add() all dependencies are automatically installed. We can move all assignments to the package repo too (but not the solutions and not the lectures).

Lesson Plans

Add a Lesson.jnbk for each week.

Week 6 Competition - CAS

For the week 6 competition, students will develop their own collision avoidance systems. The collision avoidance systems will be filled out in provided .jl files, which will then be submitted. The objective is to minimize the number of collisions and the number of alerts (with some weighting between them).

The CAS implements reset() and tick(). They cannot use any Julia libraries besides those used for the class (AA120Q.jl, Distributions, BayesNets, etc.)

The pilot response model will be simplistic and assume that the RA is followed exactly and immediately.

TEACHERS provide:

• the API
• a set of pre-generated human traces from RLES-SISLES
• an example CAS that works
• a pilot response model
• code for running a CAS on the pre-generated human traces
• code for running a CAS on the full batch of human traces from which evaluation metrics are extracted
• server runs 100,000 sims for each student using their CAS to compute a score

STUDENTS provide:

• their own implementation of a CAS by extending our provided CAS .jl file

Add advisory marker to encounter plot

So we can see when advisories are issued.

Point out ways to improve models

We should discuss why the provided models are bad and suggest methods for improvement.

Tim:

• start a section for each assignment that does this

Shahram:

• improve & elaborate. Make Tim's text understandable!

Remove Reel

We can mention it in the lesson plan instead.

Switch to Gadfly

Pure julia!

Week 5 Competition

For the week 5 competition, students will use their models to estimate a distribution over the miss distance for both with and without TCAS.

The students will run simulations on their end using their models, from which miss distance distributions can be obtained.

TEACHERS provide:

• the API
• training data
• example implementation
• code for generating a set of trajectories both with and without TCAS using an encounter model
• code for pulling miss distance from trajectory
• code for pulling piecewise uniform distribution over miss distance from set of trajectories
• competition leaderboard

STUDENTS provide:

• their own encounter model that implements sample_initial() and sample_transition()
• submit their distribution estimates (and code, to be graded) to the site

Can't import AA120Q package

Whenever I try to import AA120Q with
Pkg.add(PackageSpec(name="AA120Q", url="https://github.com/sisl/AA120Q.git")) using AA120Q

I get the error:
Updating registry at~/.julia/registries/GeneralUpdating git-repohttps://github.com/JuliaRegistries/General.git`
[1mFetching: [========================================>] 100.0 %.0 %==========> ] 23.5 %] 46.8 %=============================> ] 70.3 %] 93.6 % Resolving package versions...
Installed StatsBase ─ v0.26.0
Updating ~/.julia/environments/v1.0/Project.toml
[no changes]
Updating ~/.julia/environments/v1.0/Manifest.toml
[2913bbd2] ↑ StatsBase v0.25.0 ⇒ v0.26.0
Updating git-repo https://github.com/sisl/AA120Q.git
[1mFetching: [========================================>] 100.0 %.0 % Resolving package versions...
┌ Info: Assigning UUID 91cbf3e6-4e4b-5d64-958a-11875e51d27a to AA120Q
└ @ Pkg.Types /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Types.jl:724
Updating ~/.julia/environments/v1.0/Project.toml
[91cbf3e6] ~ AA120Q v0.0.0 #master (https://github.com/sisl/AA120Q.git)
Updating ~/.julia/environments/v1.0/Manifest.toml
[91cbf3e6] ~ AA120Q v0.0.0 #master (https://github.com/sisl/AA120Q.git)
Building AA120Q → ~/.julia/packages/AA120Q/TWYJ6/deps/build.log
┌ Error: Error building AA120Q:
│ ERROR: LoadError: UndefVarError: Pkg not defined
│ Stacktrace:
│ [1] top-level scope at none:0
│ [2] include at ./boot.jl:317 [inlined]
│ [3] include_relative(::Module, ::String) at ./loading.jl:1038
│ [4] include(::Module, ::String) at ./sysimg.jl:29
│ [5] include(::String) at ./client.jl:388
│ [6] top-level scope at none:0
│ in expression starting at /Users/harper/.julia/packages/AA120Q/TWYJ6/deps/build.jl:1
└ @ Pkg.Operations /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1068
┌ Info: Precompiling AA120Q [91cbf3e6-4e4b-5d64-958a-11875e51d27a]
└ @ Base loading.jl:1186
┌ Warning: Pkg.dir(pkgname, paths...) is deprecated; instead, do import AA120Q; joinpath(dirname(pathof(AA120Q)), "..", paths...).
└ @ Pkg.API /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:454
┌ Warning: Pkg.dir(pkgname, paths...) is deprecated; instead, do import AA120Q; joinpath(dirname(pathof(AA120Q)), "..", paths...).
└ @ Pkg.API /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:454
┌ Warning: Pkg.dir(pkgname, paths...) is deprecated; instead, do import AA120Q; joinpath(dirname(pathof(AA120Q)), "..", paths...).
└ @ Pkg.API /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:454
┌ Warning: Pkg.dir(pkgname, paths...) is deprecated; instead, do import AA120Q; joinpath(dirname(pathof(AA120Q)), "..", paths...).
└ @ Pkg.API /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:454
┌ Warning: Pkg.dir(pkgname, paths...) is deprecated; instead, do import AA120Q; joinpath(dirname(pathof(AA120Q)), "..", paths...).
└ @ Pkg.API /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:454
┌ Warning: Pkg.dir(pkgname, paths...) is deprecated; instead, do import AA120Q; joinpath(dirname(pathof(AA120Q)), "..", paths...).
└ @ Pkg.API /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:454
┌ Warning: Pkg.dir(pkgname, paths...) is deprecated; instead, do import AA120Q; joinpath(dirname(pathof(AA120Q)), "..", paths...).
└ @ Pkg.API /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:454
┌ Warning: Pkg.dir(pkgname, paths...) is deprecated; instead, do import AA120Q; joinpath(dirname(pathof(AA120Q)), "..", paths...).
└ @ Pkg.API /Users/osx/buildbot/slave/package_osx64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:454
ERROR: LoadError: LoadError: LoadError: UndefVarError: @printf not defined
Stacktrace:
[1] top-level scope
[2] include at ./boot.jl:317 [inlined]
[3] include_relative(::Module, ::String) at ./loading.jl:1038
[4] include at ./sysimg.jl:29 [inlined]
[5] include(::String) at /Users/harper/.julia/packages/AA120Q/TWYJ6/src/AA120Q.jl:1
[6] top-level scope at none:0
[7] include at ./boot.jl:317 [inlined]
[8] include_relative(::Module, ::String) at ./loading.jl:1038
[9] include(::Module, ::String) at ./sysimg.jl:29
[10] top-level scope at none:2
[11] eval at ./boot.jl:319 [inlined]
[12] eval(::Expr) at ./client.jl:389
[13] top-level scope at ./none:3
in expression starting at /Users/harper/.julia/packages/AA120Q/TWYJ6/src/cas_eval.jl:12
in expression starting at /Users/harper/.julia/packages/AA120Q/TWYJ6/src/cas_eval.jl:12
in expression starting at /Users/harper/.julia/packages/AA120Q/TWYJ6/src/AA120Q.jl:85
Failed to precompile AA120Q [91cbf3e6-4e4b-5d64-958a-11875e51d27a] to /Users/harper/.julia/compiled/v1.0/AA120Q/ypfPZ.ji.

Stacktrace:
[1] error(::String) at ./error.jl:33
[2] macro expansion at ./logging.jl:313 [inlined]
[3] compilecache(::Base.PkgId, ::String) at ./loading.jl:1184
[4] macro expansion at ./logging.jl:311 [inlined]
[5] _require(::Base.PkgId) at ./loading.jl:941
[6] require(::Base.PkgId) at ./loading.jl:852
[7] macro expansion at ./logging.jl:311 [inlined]
[8] require(::Module, ::Symbol) at ./loading.jl:834
[9] top-level scope at In[1]:4
`

I made some code edits to try to fix this, but they did not resolve this issue. Could someone help me out here?

Final Competition - with Sensors

For the final competition, students will develop their own collision avoidance systems with partial observability. The collision avoidance systems will be filled out in provided .jl files, which will then be submitted. The objective is to minimize the number of collisions and the number of alerts (with some weighting between them).

The CAS implements reset() and tick(), the latter of which will receive observations rather than the true aircraft states (ie, range with Gaussian noise, bearing with Gaussian noise, and relative altitude quantized to a bin).

They cannot use any Julia libraries besides those used for the class (AA120Q.jl, Distributions, BayesNets, etc.)

TEACHERS provide:

• the API
• a set of pre-generated human traces from RLES-SISLES
• an example CAS that works
• code for running a CAS on the pre-generated human traces
• code for running a CAS on the full batch of human traces from which evaluation metrics are extracted
• Alpha-Beta Filters demo
• Outlier removal demo
• server runs 100,000 sims for each student using their CAS to compute a score

STUDENTS provide:

• their own implementation of a CAS by extending our provided CAS .jl file

TCAS has high NMAC rate

TCAS has a somewhat high NMAC rate
(but no longer a high alert rate)

Looks like it didn't alert early enough here:

Reversal penalties and Climb Rate magnitude penalties

Add these

Submission via Canvas

Ensure all assignments mention submission via canvas