Automatically photograph planes that fly by!
Airbus A321 at 32,000 feet
Building image datasets for machine learning applications can be slow and expensive. SkyScan demonstrates a low-cost system that uses sensors to automate the building of datasets appropriate for computer vision models. SkyScan uses a ground-based camera and software-defined radio to capture images and correlated identity information of commercial aircraft. The SkyScan field kit software runs on a Raspberry Pi and all devices are low-power, allowing the system to run for several hours off of a portable battery.
To enable better tracking, most planes broadcast a signal known as Automatic Dependent Surveillance–Broadcast or ADS-B. This signal is at 1090MHz and can be easily received using a low cost Software Defined Radio (SDR), like the RTL-SDR which repurposes a digital TV chip.
From the ADS-B transmissions, you can get a plane's location and altitude. If you know where a plane is and where you are, you can do some math, point a camera at the plane and take a picture. If you have a Pan/Tilt camera lying around, you can have it automatically track a plane as it flies by and snap photos.
After a dataset has been collected, a model can be trained to build a plane detector and classifier using the labeled plane data.
This project is built around the RaspberryPi 4, an RTL-SDR, and an Axis PTZ security camera. It could be extended to work with other SDRs or cameras.
Axis has a great API for their network cameras, and it should work with any of their PTZ cameras. The m5525 is nice because it supports continuous 360 degree rotation. You can literally have it spin around in circles, giving you complete coverage. The code has been tested with the 10x zoom Axis m5525 and the 30x zoom Axis p5655 cameras.
We are using the Nooelec NESDR SMArt v4 SDR. This is a nice and stable RTL-SDR. It is compact and doesn't block all the other ports on a Pi. Since you are just trying to capture local planes, you can get away with using any antenna you have lying around.
Two configurations were developed for this project, one with AC and one with DC power input.
The major components of the system were modelled in CAD software. A few 3D printed parts were designed to house and mount the things together. These files can be found in the hardware STL directory.
See the Hardware README for additional component details including a BOM.
The different components for this project have been made into Docker containers. This modularity makes it easier to add in new data sources or cameras down the road. We have found that containers work really well on the Pi and help enforce that you have properly documented all of the software requirements.
+-------------+ +-------------+ +---------------+ +--------------+
| | | | | | | |
| | | | | | | Axis+PTZ |
| Pi+Aware +----->+ ADSB+MQTT +---------->+ Tracker +----------->+ |
| | TCP | | MQTT | | MQTT | |
| | | | all | | only | |
+-------------+ +-------------+ planes +---------------+ tracking +-------+------+
plane |
| HTTP API
| Pan/Tilt
v
+--------------------+ +--------+---------+
| | | |
| | | |
| MQTT Broker | | Axis m5525 |
| | | Camera |
| | | |
| | | |
+--------------------+ +------------------+
Here is a brief overview of each component. There are additional details in the component's subdirectory
-
mikenye/piaware - This is a dockerized version of FlightAware's PiAware program. PiAware is actually just a wrapper around dump1090. Dump1090 is a small program that can use an RTL-SDR to receive an ADS-B transmission. The program uses these transmission to track where nearby planes are and display them on a webpage. It also outputs all of the messages it receives on a TCP port for other programs to use. We use this connection to get the plane information. PiAware adds the ability to send the information to FlightAware. You could probably just switch this to only use Dump1090.
-
ADSB-MQTT is a small Python program that reads in information collected by Dump1090 over a TCP port and publishes all the messages it receives onto the MQTT bus.
-
tracker receives all of the plane's location, determines how far away from the camera each one is and then finds the closest plane. The location and relative position of the closest plane is periodically published as an MQTT messages. Tracker needs to know the location and altitude of the camera in order to determine the plane's relative position.
-
axis-ptz receives updates on which plane to track over MQTT and then directs the PTZ camera towards the plane and takes a picture.
There are currently three different machine learning pipelines.
The first pipeline uses a series of Jupyter notebooks to prepare the image dataset, train object detection and classification models, and then evaluate those models. These notebooks can be found here. Instructions for using these notebooks can be found here.
The second pipeline re-packages much of the functionality found in the Jupyter notebooks into a Python script that can be run from the command line. The scripts and directions can be found here.
The third pipeline uses the YOLTv4 model to also perform object detection. The relevant notebooks can be found here.
Follow the steps below for installing the edge software.
After the system has been installed, follow the steps below to set up the hardware and start the software.
Other helpful information about this project:
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent