dennisss / tansa

Dancing Robotics Platform

Tansa: Dancing Robotics Platform

An open platform for making drones dance! This project includes the following:

1. Reference quadcopter electrical/mechanical designs used for developing the system

   • Around $500 BOM cost. Requires a drill, saw, 3d printer, off the shelf parts, and time to replicate.
   • Features 2 controllable spotlights for theatrical purposes
   • See hardware/ directory

2. Drone control algorithms and drivers

   • Written in C++ with no dependency on ROS
   • Interface to networked MAVLink vehicles
   • Trajectory generation and control
     • Control based on the paper: Minimum Snap Trajectory Generation and Control for Quadrotors by Daniel Mellinger and Vijay Kumar
   • Motion capture support
     • Driver for using with OptiTrack Motive 1.9.0
   • Simulation Environment Integration
     • All simulation integrations are multi-drone capable out of the box
     • Gazebo support via PX4/sitl_gazebo

3. Operator User Interface

   • HTML/JS interface based for visualizing and running scripts

License

Unless otherwise specified, all current files and future contributions in this repository are distributed under the MIT License (see LICENSE file).

Requirements

This platform was primary developed for UNIX based OSes (Linux / Mac OS X), but should be mostly compatible with Windows as well (probably won't compile on windows in it's current state though).

You can optionally skip most of this and try using one of your prebuilt Docker images (see config/docker/ directory).

For running/compiling the core code, the following libraries must be installed:

• CMake
• Eigen3
• Boost
• CGAL

For running the simulations, all dependencies on the PX4 toolchain/gazebo SITL environment must be met:

• See http://dev.px4.io/starting-installing.html
• See http://dev.px4.io/simulation-gazebo.html
  • We test with Gazebo 7

UI Dependencies

• Node.js >6.0.0
• A recent web browser such as Google Chrome that supports WebGL

Recommended/Optional Dependencies:

• The IBM CPLEX library is a recommended addition if you are using advanced trajectory generation. It is required for making collision free trajectories. If not included, it will fallback to the less stable CGAL solvers when possible.

Building

1. Run make build to compile all the core libraries

2. Run make build_firmware to build the nested PX4 code and all code needed for Gazebo simulation

3. Run npm install to download packages needed for the GUI

Alternatively, run make sim to start up a multidrone simulator

Run make run to build the core stuff and start up the controller. Make sure that config.json has the right filepaths, and the right ip addresses if not using simulator.

Running

Once everything is built, you can run ./start.sh to quick start into a simulator with the UI running.

The script does three things

1. Starts the main gcs program via make run

   • This communicates and controls the drones

2. Starts the simulator using make runSim

3. Serves the UI via make server

UI

Viewable at http://127.0.0.1:4000 after starting the server

• The root url will take you to the web-based 3d viewer and control suite

• Each 'choreography' is available to load and correspondences to files in the data/ folder

• Upon loading a file, the drones should automatically spawn into the environment

Documentation

See the doc folder

A full source code documentation page can be generated by running make doc. Then the documentation can be viewed from /build_doc/html/index.html

tansa's Issues

Server should ensure every position path is differentiable

Any discontinuity should result in a fail-safe result if reached. But if #4 is done before running a jocs file, this should never happen.

Should be able to run this check from a specific time along with new "home" locations as well, as described in scenario for #44

Latency Corrects

Not crucial right now, but needed for fast real world flight.

• Compute average latency to the drone via WiFi
• Given mocap position/velocity, forward predict position at the time when the drone should get the message
• In position controllers, use the forward prediction to compute control message

Plan how to handle multiple instances of the software running and sending updates to drones

Should this be responded to with similar results as the kill switch?

This issue also includes implementing this change, or creating another Issue when the implementation is defined.

See more detail in the comment by Dennis on this issue

JOCS: Comments

This is probably a quick fix, but needs testing. It would probably be useful to support C/C++ style comments so we don't get really confused if we strip them out to form valid JSON.

Doing a find and replace of something like //.*$ per line should work reasonably as I don't expect anyone to use double '//' inside of the rest of the JSON.

JOCS v2

Includes completion of #83 and #47 and #59 , testing each, and verifying each jocs file abides by same rules and runs without flaw.

Might also require clearing out the jocs files that are more work to maintain than they're worth.

Cleaning up gcs.cpp

• Scrap old hardcoded trajectories: either move somewher else or make into a test case?
• Single common main for just jocs parsing and other modes
  • maybe have "jocs runner" abstracted out of the main()
• Making sure to clean up all connections/allocated resources
• Preparing for #18

Don't take off if can't find mocap or server

Do not allow drone to takeoff unless all necessary components are found

GUI element(s) for editing a jocs file

Use the same gui defined in #33 and referenced in many other tasks.

This task is for making changes to a jocs file, as discussed in tuesday meetings.

Better Simulation Viewer

• Replace slow Gazebo Client
• Incorporate visualization of lighting
• Visually see the theater floor region as the ground plane
• Option to view from audience perspective (among others)

Eventually may roll into the choreography design GUI.

System checks for differences between simulation and real life

Examples:

• Battery Life
• IR and motion sensor capability to identify drone locations
• Verify speeds can be handled by drones
• Drones don't collide

QLab Integration

For show time, they will be using QLab (https://figure53.com/qlab/) on OS X. Theater techs use it to trigger the start of the music and visuals. Ideally there will be a 'cue' in QLab that triggers the drones to start flying as soon as the music starts.

It can be triggered with and trigger other programs via OSC (https://figure53.com/docs/qlab/v3/scripting/osc-dictionary-v3/).

Convert existing choreography to JOCS

Automatic Latency/Ping Estimator

Part of #11 . Needed for #15 to do any maneuvers at a moderately fast speed.

After #5 is done, we can trigger the IR LED to go on and see how long it takes to for a round trip for it be detected

Dissect that time based on ping to drone and ping to Windows running mocap software.

Initial Implementation of JOCS parser

ASAP to complete other tasks

connect GUI element to config file for choosing/running a JOCS file

This is an additional feature for the SAME gui created in #33

Capabilities involving JOCS subsection:

• Specifying a JOCS file to use
• Starting where in the JOCS file to start. If mid-file, use capabilities of JOCS that are implemented in task #44

Synchronize clock between servers

To ensure there isn't lag in the "dance" done by the drones, we need to ensure clocks are synced between QLab and our server.

(updated in response to Dennis's comments)

Track drones using *active* IR

Motion capture doesn't know which drone is which. IR will uniquely identify drones so paths can be sent to proper drones.

Might also be helpful for avoiding collision.

Engineers will work on making this work, but we have to ensure the code is actually making use of this capability.

Pending status from mechanical&computer engineering team.

Implement block sheets (first 3 minutes)

Prepare the choreographies:

• Lone wolf
• Twins
• Triplets
• Primordial Soup
• Plateu
• Through Puppets

Test mostly in simulation as we may not have enough space to do them all.
Can as many done by first theater time so we can test them for real

Initial CLI/GUI for operating the drones

• Possibly write a simple GUI with some buttons in Qt
• We need to be able to hit 'Go'
• Hit 'Pause' (should stop at the next 'safe' location in the choreography velocity=0)
• If Paused, hit 'Land at current position'
• At all times, hit 'Kill'

All these should be commands while the server is running. Restarting the server to change arguments/configs is not an option.

Depending on our approach, related to #18

@dmz38 , it was suggested that maybe you would be interesting in heading this up so that we have a snappy thing to work with this weekend

JOCS translator

This can be done in parallel to the jocs parser. Purpose is to take parsed json and turn it into respective shape and choreography objects.

Shape: Line

Includes:

• Taking initial velocity into account when entering circle. This might include giving lead time/space to get up to velocity needed
• Taking ending velocity into account when exiting the circle, to maintain continuity with next shape in path

Fix all warnings and clean up code style issues

See title

Make base action class

Shapes will inherit from this

Restructure gcs.cpp to use jocs

Instead of hardcoding, have the gcs file actually use the jocs and the jocs parser

Integrate lighting - jocs

Lighting and Motion should be able to run simultaneously - such that the server can send instructions for each separately without one interrupting the other

Note: Lighting controller doesn’t need to run at 100Hz, but only when things change (limited to human viewable speeds). So, it shouldn’t take away much network bandwidth from the motion controllers

This task is for the code to read from jocs and send commands to drones. jocs changes are defined in #59 so must work alongside that task.

Segments/Sequences in JOCS

Have the ability to start the JOCS from certain manually named/numbered segments representing different separate actions in the choreography

Implement entire song from blocking sheets

Ability to chain jocs files

Add ability in code to read multiple jocs files and execute sequentially.

Integrate lighting in backend code

Update all jocs files to include lighting specifications

Improve failsafe drone bahavior

Use timeout behavior. A subprogram on the drone should auto-kill in the following scenarios.

• When messages are timedout or it doesn't get heartbeats from the server.
• When no instance of the software is running (related to previous bullet point)

Support controlling groups of drones at a time

Drones have memberships to groups, ensure multiple groups can be handled at a time.

Include handing multiple groups as well.

Make the base program run more like a 'server'

• A background process that we can just keep running
• Starts several minutes before the 'show'
  • Allow time to view diagnostics and sync positioning/mocap
• Command interface related to #33 and #16 for loading a jocs file, playing, pausing, etc.
• Keep running until drones are on the ground, disarmed, and all choreographies are done

Define a Choreography File Format

Involves managing motion controllers. Should define .json (possible?) file format to be used to store a choreography.

Must decide what should be stored in this object and how. Ideally, as much as possible will be cache-able.

Note: A good place to put this code is in the ‘src/planning’ folder. This is where Dennis sketched out initial thoughts for this task

Include translating Parsons's dance into instances of this

json should also support group assignments

Streamline jocs files

Ensure all jocs files use the same naming conventions and formatting convenctions.

Verify that the parrser has the right "key" names

GUI element for verifying a jocs file

Use the same GUI referenced in #53 and created in #33

This task is for adding capability to run the verification logic defined in #4

Shape: Spiral

Includes:

• Taking initial velocity into account when entering circle. This might include giving lead time/space to get up to velocity needed
• Taking ending velocity into account when exiting the circle, to maintain continuity with next shape in path

CLI arguments + Global/Vehicle config file

Make everything configurable!!!!

Have CLI arguments for us to be able to start different operating 'modes':

1. just mocap with no JSOC (so we can test mocap with an RC controller in mocap assisted position mode)

2. 1 with a JSOC file

3. 1 or # 2 in simulation

4. 1 with and without an RC as a failsafe (in RC mode, the program wouldn't automatically arm itself, but wait for the RC to do this)

Need to make a JSON or something separate from JSOC to hold global settings and ones unique to each Vehicle so we can quickly change parameters for different setups. Partial listed of needed parameters below.

• Maybe split into 2 files: one for hardware stuff and one for 'world' (# of drones, default positions of drones, etc.)

Global stuff

• Mocap settings: interface IP address and multicast address
• ....

Vehicle specific stuff

• Default position expected (mainly for initialization of simulation), and pre-flight check of initial positions
• IP/Port combinations for connecting to it
• Type of vehicle for future proofing (if we want to lookup calibration parameters maybe?)

Shape: Circle

Includes:

• Taking initial velocity into account when entering circle. This might include giving lead time/space to get up to velocity needed
• Taking ending velocity into account when exiting the circle, to maintain continuity with next shape in path

Kill Switch - From Server / User

Allow user to press kill switch remotely through the server

Make sure you read the comment from Dennis on this post

JOCS: Hover/Pause at position

Support explicitly switching to hover mode at a given point when stated in the file. Should essentially act like a 'pause' for one or more drones when they don't have something else to do.

Add ability to start choreography at random time

If director asks for "start at t=72", we can start the piece here. Alternatively, may choose to give "names" to segments of the dance and be able to start from a named segment.

So this would entail programmatically creating a new transition for each drone from a new "home" starting point. Will require math to calculate non-crossing paths as well as getting to correct velocity at the point in the choreography we want to start at.

Extensive test suite to verify that choreography is continuous

Purpose is to ensure choreography is continuous for time and position.

Things to test

• Speed to ensure drone can handle any given speed
• Collision to ensure drones don't collide with each other
• Collision to ensure drones don't collide with people. Human trajectories will be preset, possible using same trajectory as described in #2
• Ensure no drone can be controlled by more than one conflicting controller at a time. Shouldn't be able to specify to move Group 1 of drones [1,2,3] and Group 2 of drones [3,4,5] simultaneously, as 3 would have dual group identity.
• Dennis added comments as well, be sure to look at them

Make error handling actually give useful information.

Right now we just crash with random exceptions that don't explain the problem when JSON parsing fails in various places. Implement a better system with nice output that doesn't require arcane knowledge.

Standardize JOCS v1 Schema

Finalize all changes to JOCS format ASAP so we don't run into compatibility/parsing issues. May require some thought into #1 .

• Should we add in information about the space of the theather?
• Move all JOCS files to data/ directory (from testing/data/)
• Make sure all JOCS example files in the repo actually parse correctly
  • Adding in auto-test cases.
• Think about #47 for how to allow 'including' files
• Will need some type of segment naming/clue system for #44

Stretch goals

• See if we can get better (more specific) parsing errors out of the JSON parser

Cache data

Whenever applicable, store data on drone rather than on server. Save time on calculations, memory access, etc.

Want server code to run as smoothly as possible

Auto disarm drones on completion

In the circle jocs in particular, the drones are overshooting their landing location

Shape: Transition

JOCS allows transitions to be defined, and the code needs a lot of extra functionality to read the shape before and after the transition in order to create the transition

Shape: Slanted Circle

Circle on a plane that isn't parallel to the ground.

Completely not necessary.

Make presentation

Discuss background

• Dennis and Adam
• Code/design that already existed
  Discuss other senior design
  Discuss what we did
• Testing
• JOCS
• Experiences: rewrite simulator rendering, drones breaking, etc
• When things don't work, drone can break and we have to rebuild it
• Safety issues
  Discuss other things
• Simulator and how it differs from real life, how we got it to work

Auto-convert feet to meters

The blocking sheets are written in feet. We need to convert to meters automatically