copterexpress / clever-show

Software for making the drone show with 🍀

clever-show

Русская версия

Software for making the drone show with drones controlled by Raspberry Pi with COEX Clover package and flight controller with PX4 firmware.

Create animation in Blender, convert it to drone paths, set up the drones and run your own show!

Demo video

12 drones perform in a show in Electrotheatre Stanislavsky, Moscow.

This software includes

• Drone side for remote synchronized control of drones with emergency drone protection module
• Server side for making the drone show with ability of tuning drones, animation and music
• Blender 2.8 addon for exporting animation to drone paths
• Raspberry Pi image for quick launch software on the drones
• Documentation

Positioning systems supported

• Optical flow (indoor)
• ArUco map-based (indoor)
• GPS (outdoor)

Quick start

Start making your own show using this tutorial!

clever-show's Issues

Make failsafe watchdog in standalone node.

Request description
At the current time, there are 2 parts of "Raspberry failsafe" in client's software: delta position detection between setpoint and current position in copter_client.py and visual pose timestamp timeout detection in visual_pose_watchdog.py. In some situations these parts are triggered together and conflict with each other, e.g. when there is error in position and fcu switch to auto.land mode, after that visual_pose_watch try to decrease the motors power in offboard mode, but can't do it, because the fcu is already in auto.land mode.

Describe the solution you'd like
Make standalone node called failsafe.py, join all abilities there.

Task manager is 'overinterrupted'

The bug description
Sometimes the task can be interrupted, when it should not be interrupted.

Log example from real flight
It was the usual flight, when I want to land copter by sending command from server, but the command was executed only for the second time.

Jan 22 13:21:43 clever-05 bash[493]: 2020-01-22 13:21:43,028 [FlightL] [Task process] [INFO ] Going to: | x: 10.811 y: -1.975 z: 3.545 yaw: 3.142
Jan 22 13:21:43 clever-05 bash[493]: 2020-01-22 13:21:43,029 [tasking] [Task process] [INFO ] Execution done
Jan 22 13:21:43 clever-05 bash[493]: 2020-01-22 13:21:43,030 [tasking] [Task process] [INFO ] Executing task execute_frame
Jan 22 13:21:43 clever-05 bash[493]: 2020-01-22 13:21:43,096 [FlightL] [Task process] [INFO ] Going to: | x: 10.875 y: -1.967 z: 3.546 yaw: 3.142
Jan 22 13:21:43 clever-05 bash[493]: 2020-01-22 13:21:43,099 [tasking] [MainThread ] [INFO ] Task queue paused
Jan 22 13:21:43 clever-05 bash[493]: 2020-01-22 13:21:43,105 [tasking] [MainThread ] [INFO ] Task queue resumed with timeshift 0.0
Jan 22 13:21:43 clever-05 bash[493]: 2020-01-22 13:21:43,108 [tasking] [Task process] [WARNI] Timeout checking exception: list index out of range
Jan 22 13:21:43 clever-05 bash[504]: 2020-01-22 13:21:43,519 [_main] [Thread-4 ] [DEBUG] armed: True | mode: OFFBOARD | viz_dt: 0.08 | pos_delta: 0.36 | pos_dt: -0.03 | action: emergency_land | range: 10.00
Jan 22 13:21:43 clever-05 bash[493]: 2020-01-22 13:21:43,732 [FlightL] [Telemetry ge] [WARNI] [Linear velocity estimation]: Err: Z velocity estimation: 0.161 m/s
Jan 22 13:21:44 clever-05 bash[504]: 2020-01-22 13:21:44,019 [_main] [Thread-4 ] [DEBUG] armed: True | mode: OFFBOARD | viz_dt: 0.08 | pos_delta: 0.11 | pos_dt: 0.00 | action: emergency_land | range: 10.00
Jan 22 13:21:44 clever-05 bash[504]: 2020-01-22 13:21:44,519 [_main] [Thread-4 ] [DEBUG] armed: True | mode: OFFBOARD | viz_dt: 0.08 | pos_delta: 0.22 | pos_dt: 0.00 | action: emergency_land | range: 10.00
Jan 22 13:21:44 clever-05 bash[493]: 2020-01-22 13:21:44,805 [FlightL] [Telemetry ge] [WARNI] [Angular velocity estimation]: Err: Pitch rate estimation: 0.109 rad/s
Jan 22 13:21:44 clever-05 bash[493]: 2020-01-22 13:21:44,808 [FlightL] [Telemetry ge] [WARNI] [Angular velocity estimation]: Err: Roll rate estimation: 0.129 rad/s
Jan 22 13:21:45 clever-05 bash[504]: 2020-01-22 13:21:45,020 [_main] [Thread-4 ] [DEBUG] armed: True | mode: OFFBOARD | viz_dt: 0.08 | pos_delta: 0.36 | pos_dt: 0.00 | action: emergency_land | range: 10.00
Jan 22 13:21:45 clever-05 bash[504]: 2020-01-22 13:21:45,518 [_main] [Thread-4 ] [DEBUG] armed: True | mode: OFFBOARD | viz_dt: 0.08 | pos_delta: 0.33 | pos_dt: 0.01 | action: emergency_land | range: 10.00
Jan 22 13:21:45 clever-05 bash[493]: 2020-01-22 13:21:45,795 [FlightL] [Telemetry ge] [WARNI] [Angular velocity estimation]: Err: Pitch rate estimation: 0.094 rad/s
Jan 22 13:21:45 clever-05 bash[493]: 2020-01-22 13:21:45,803 [tasking] [MainThread ] [INFO ] Task queue paused
Jan 22 13:21:45 clever-05 bash[493]: 2020-01-22 13:21:45,805 [tasking] [MainThread ] [INFO ] Task queue resumed with timeshift 0.0
Jan 22 13:21:45 clever-05 bash[493]: 2020-01-22 13:21:45,807 [tasking] [Task process] [INFO ] Executing task land
Jan 22 13:21:45 clever-05 bash[493]: 2020-01-22 13:21:45,814 [tasking] [Thread-4 ] [WARNI] Waiting was interrupted!
Jan 22 13:21:45 clever-05 bash[493]: 2020-01-22 13:21:45,824 [tasking] [Thread-4 ] [WARNI] Waiting was interrupted!
Jan 22 13:21:45 clever-05 bash[492]: #33[33m[ WARN] [1579699305.867162293]: ros.mavros.cmd: CMD: Unexpected command 176, result 0#033[0m
Jan 22 13:21:46 clever-05 bash[504]: 2020-01-22 13:21:46,019 [_main] [Thread-4 ] [DEBUG] armed: True | mode: OFFBOARD | viz_dt: 0.13 | pos_delta: 0.34 | pos_dt: 0.06 | action: emergency_land | range: 10.00
Jan 22 13:21:46 clever-05 bash[493]: 2020-01-22 13:21:46,394 [FlightL] [Task process] [WARNI] Land function interrupted!
Jan 22 13:21:46 clever-05 bash[493]: 2020-01-22 13:21:46,396 [tasking] [Task process] [INFO ] Execution done

Expected behavior
The task should not be interrupted by itself, we need to rework the task manager.

Version
v0.3.

Get copter id from /etc/hostname

Idea is to have one clear copter name, instead of many names in different settings.

Add script to copter's raspberry startup to connect first to wifi router with default settings and if connect fails switch to AP mode

Sort table by ids of drones

Now it looks like:
clever-3
clever-5
clever-1
clever-2
clever-4

This is inconvenient, I think

Add ability to view and edit client config file from selected drone

Request description
At this moment, the user has to use ssh to view the client config file. I suggest to add an option "view and edit client config" on the right-click to the copter row.

Show relevant information in spreadsheet

Fill empty cells with information about copter: animation ID, battery status, checks status and copter time

Self-remove drones from server's table

Request description
If a drone was power-off it will continue to stay in the server's table. It is uncomfortable.

Describe the solution you'd like
The drone must be self removed from table after constant delay time without telemetry. Set the delay time as a parameter in server's config file.

Make feedback to know when client is busy

Request description
When we send any command or file, we don't know what happens with client, only telemetry stops to transmit. So we need to think about the way to know the feedback from copter during some tasks and operations.

Add workaround to test software in simulator

Request description
I think it is necessary feature if we want to develop our software faster.

Describe the solution you'd like
I think that a good solution is a docker-container like described here, but with our software and ability to generate as many drones as we want.

Describe alternatives you've considered
Maybe it can be some setup of sitl and visualization with sitl/qgc/our visualization tool that will be lighter than gazebo visualization.

Add ability to load clever launch files directly to client

Need a function to load all files from directory to /home/pi/catkin_ws/src/clever/clever/launch

Add check for animation file

If there is no animation file on copter, it emits next exception:
Starting animation!
Animation thread activated
Exception in thread Thread-8:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "client.py", line 144, in animation_player
frames = play_animation.read_animation_file()
File "/home/pi/CleverSwarm/Drone/play_animation.py", line 43, in read_animation_file
with open(filepath) as animation_file:
IOError: [Errno 2] No such file or directory: 'animation.csv'

Make preflight check

Make simple preflight check on each of selected copters and show the result in the preflight check status cell

Server: table update rate is slow

The bug description
If you try to increase or decrease the column width, visual response is rather slow. If you try to check the copter, visual response of check is rather slow too

Steps to reproduce the behavior
Try to change any visible item of the table

Expected behavior
Fast response of updating the table

Version
v0.3-alpha.2

Additional information
I think the problem is that table update function contains many checks, maybe we need to put these checks into a separate thread.

Add private and public sections for parameters

To add ability for uploading public parameters on all copters, but private only on each copter apart.

Modify the "Start animation" button

Request description
Now the "Start animation" button is available if all of columns from all drones is correct only. Otherwise the button is inactive. No varyativity this way.

Describe the solution you'd like
Add "Start animation for correct drones" button. This command must begin animation for only fully correct drones, but drones with any problems must be ignored automatically in this mission.

Describe alternatives you've considered
Not add new button, modificate "Start animation" function

Incorrect start position with floor aruco-map

The bug description
The drone doesn't see aruco-map on the floor -->
Current position is displayed incorrect -->
Current position doesn´t match with start position -->
Аnimation can't be started.

Steps to reproduce the behavior

1. Put the drone on floor (floor's arucomap)
2. Power on
3. Try to click "Start animation" buttom
4. See error

Expected behavior
The drone must be execute the animation

Add ability to fly in gps coordinates

We need this enhancement to fly outdoor

Clever service doesn't restart when user send new aruco_map file.

copter_client.py run as daemon in systemd.

Add ability to load full animation with takeoff and land without destroying logic of our animation parser

Request description
At this moment the logic of our animation parser is next: wait start command, takeoff to takeoff_height, wait takeoff_time, fly to first point, wait reach_first_point_time, then fly around every setpoint from animation file with a certain delay. When the points are over, wait land_time and land. So the animation file is supposed to have the next restrictions: first point should be on some height and can't be on the floor so the user can't make animation of takeoff, and the last point also should be on some height, and animation of landing is also impossible.

Describe the solution you'd like
Add option to detect first start point and last land point automatically. The first point is the first point before x or y is changed, the last point is the first point in reverse order from the end when x or y is changed.

Describe alternatives you've considered
Add arming and disarming flags to new animation exporter.

Change copter config file on broadcast

Start time doesn't update after resume animation, so animation after pause is out of sync

Add server ip to /etc/chrony/chrony.conf and restart chrony when receive broadcast ip

Release Server as AppImage for Linux, .app/.dmg for MacOS and .exe for Windows

For easier distribution of our software.

For example, we can use pyinstaller.

Add frame delay option in client settings

Request description
Now the delay is hardcoded to be 0.125 seconds per frame, I think we need to add this delay to config file of client.

Add ability to send public and private parameters from server

To make upload of the parameters more clear.
Also, send file with parameters only on selected copters.

Set up loggers correctly

Loggers now can log information only in main thread in client, can we set them up to collect all information from all threads and libraries?

Edit buttons layout

For now, buttons "land" and "disarm" are dangerously near to "flip" and "takeoff". On the other hand, "emergency land" is far away. This could lead to user inconvinice, confusion and missclicks.

Software version not recognised if the project folder was downloaded as ZIP file instead of git clone

The bug description
In drone list the version column highlights yellow.

The server gives error:
subprocess.CalledProcessError: Command 'git log --pretty=format:'%h' -n 1' returned non-zero exit status 128.

Steps to reproduce the behavior

1. Go to 'https://github.com/CopterExpress/clever-show/tree/qt-gui-update'
2. Click on 'Clone or download'
3. Download ZIP
4. Run server
5. Run drone with latest software of the same branch
6. See error: subprocess.CalledProcessError: Command 'git log --pretty=format:'%h' -n 1' returned non-zero exit status 128.

Expected behavior
No error and green version cell.

Version
6b16499

Check if ROS services and nodes are available when perform any flying commands

When ROS services are not available (when restart clever.service for example), the client emits this exception:

Starting takeoff!
Traceback (most recent call last):
File "client.py", line 297, in
play_animation.takeoff(safe_takeoff=SAFE_TAKEOFF)
File "/home/pi/CleverSwarm/Drone/play_animation.py", line 18, in takeoff
FlightLib.takeoff(z=z, wait=True, emergency_land=False)
File "/home/pi/CleverSwarm/Drone/FlightLib/FlightLib.py", line 243, in takeoff
set_rates(thrust=0.1, auto_arm=True)
File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 435, in call
return self.call(*args, **kwds)
File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 505, in call
raise ServiceException("unable to connect to service: %s"%e)
rospy.service.ServiceException: unable to connect to service: [Errno 111] Connection refused
Segmentation fault

Collect all needed telemetry on the client in the separate thread in special structure

Request description
The goal is to get all needed telemetry as fast as we can. I suggest to make separate thread, where all required telemetry will be received. So when the client gets selfcheck request, it will send immediately the information from the telemetry structure.

Repair auto build

The bug description
Repository folder in current commit state is copied to image in wrong way - not all necessary files (which dirs start with ".") are transferred to image, so if the user wants to work with repo dir as github repository, he can't do it.

Steps to reproduce the behavior
Problem is in copy function of img-tool that is used to build image

Expected behavior
Repository is copied to image fully.

Version
v0.3-alpha.2+

Add docker container for chrony for running on different OS

Add flip function in animation file parser

Think about how to implement flip in Blender and blender-csv-animation addon, and about time synchronization of flips

Server adds new line in table when client is reconnected

Check client's time delta on server

If delta > 1s then mark it with red color and send command to update chrony.

Repair file transfer

File transfer is broken after all data is wrapped to json

Add ability to load parameter files to selected fcu's via mavros

Request description
This option opens ability to setup all copters with equal settings with one click.

Add frame_id parameter to client_config

And add functionality for flying in frame designated in client_config file

Add ability to remove disconnected copter from table

Request description
Some users don't like when disconnected copter is displayed in the table, I think we need to get them an option to hide disconnected copters.

Describe the solution you'd like
UI option to hide disconnected copters

Add ability to rename copters directly from the server table

Request description
Add a clear solution for the user in UI to change the name of the copter.

Describe the solution you'd like
I think that the most convenient way is to double-click to cell with copter name, rename it and then press Enter button

Describe alternatives you've considered
Another way is to add separate button called "Rename copter"

Make unittests

For testing basic functionality of client-server interaction and preformance to be clearly sure that our new code won't break any existing funtionality
Probably, that will require some slight refactoring and additional Travis integration

Use preflight check information to set up logic and permissions to start drone show

For example: make start button disabled if at least one of the selected drones doesn't pass preflight checks

Check that start button works after pressing stop button.

The bug description
There's been report that animation doen't restart after pressing stop button and the user has to restart clever-show service manually.

Steps to reproduce the behavior

1. Press stop
2. Move copter to start position
3. Press start
4. Nothing happens

Expected behavior
Animation should start when start button is pressed after stop button was pressed

Version
v0.3-alpha.5

Add ability to send aruco_map files to connected drones

Client does not reconnect to server if server was restarted [server_qt]

The bug description
[server_qt.py]
In some tests, if server restarts with connected clients, they don't reconnect server again

Steps to reproduce the behavior
Restart server with connected clients.

Expected behavior
A clear and concise description of what you expected to happen.

Version
v0.3-alpha.2+

Chrony doesn't synchronize time if the copter was turned on before wifi router.

Need to add a watchdog on server/client to restart time synchronization if time is not synchronized with server.

Add server start check on other available servers on lan

And kill server, if there is another one working, with error messagebox.

Broken binding with broadcast ip

I get next messages when trying to connect to server with wrong ip:

Recieved broadcast message server_ip 1 9 2 . 1 6 8 . 1 . 1 0 4 from ('192.168.1.104', 47315)
Binding to new IP: ('192.168.1.104', 47315)
Waiting for connection, attempt 19
Waiting for connection: [Errno 111] Connection refused
Too many attempts. Trying to get new server IP
Recieved broadcast message server_ip 1 9 2 . 1 6 8 . 1 . 1 0 4 from ('192.168.1.104', 47315)
Binding to new IP: ('192.168.1.104', 47315)
Waiting for connection, attempt 20
Waiting for connection: [Errno 111] Connection refused
Too many attempts. Trying to get new server IP

Expected reaction is to bind with new ip, but there is an error in the code.

Add broadcast binding timeout

Broadcast binding leads to infinite hangout if no broadcast present in the network