roboflow / dji-aerial-georeferencing

Detect objects in drone videos and plot them on a map

Detect and Map Objects of Interest from Drone Videos

This project extracts the location of objects of interest from a drone video and plots them on a map. By combining the video with data from its flight log and a computer vision model trained on Roboflow, it demonstrates georeferencing a machine learning model's predictions to GPS coordinates and using them to visualize the location of detected solar panels on a map using Mapbox.

finding-solar-panels-small.mov

Try It in Your Browser

The project is deployed to Github Pages here and you can test it out with this sample video and flight log.

If you have your own Drone video you'd like to use, follow the instructions in the blog post to pull your detailed flight log from Airdata.

Resources

• Accompanying Blog Post: Georeferencing Objects in Drone Videos
• Try the aerial solar panels pre-trained computer vision model in your browser on Roboflow Universe
• Browse other Aerial Imagery Datasets and Pre-Trained Models
• Train Your Own Computer Vision Model to use with this repo

Run It Locally

• Clone this repo
• Run npm install in the main directory
• Run npm run build:dev to start a webpack build with livereload
• Open a new terminal window and run npx serve dist
• Open http://localhost:3000 in your browser

Customize It

This repo can easily be changed to run any custom model trained with Roboflow including the thousands of pre-trained models shared on Roboflow Universe. Simply swap out your publishable_key and the model ID and version in the ROBOFLOW_SETTINGS at the top of main.js.

There are also some additional configuration options available at the top of renderMap.js.

For example, changing the model to swimming-pool-b6pz4 to use this swimming pool computer vision model from Roboflow Universe changes the functionality from plotting solar panels to plotting pools:

pools.mp4

Other ideas for how to use this repo:

• Search & Rescue
• Monitoring swimmer safety in triathlons
• Helping governments identify zoning violations
• Monitoring core infrastructure like pipelines for potential hazards like excavators and construction equipment nearby
• Finding people with bonfires in high risk fire areas
• Tracking wildlife
• Mapping oil wells
• Monitoring land use and tree cover
• Finding boats fishing in restricted areas
• Counting cars in parking lots
• Tracking human rights violations
• Other Aerial Datasets & Models to get your gears turning

Getting Your Flight Log

You can get the detailed flight log from a DJI drone using Airdata. The sample video and flight log were taken from a DJI Mavic Air 2. Full details are in the blog post.

Training a Custom Model

If you can't find a pre-trained model that accurately detects your particular object of interest on Roboflow Universe you can create a dataset and train your own custom model using Roboflow.

Roboflow is an end-to-end computer vision platform that has helped over 100,000 developers use computer vision. The easiest way to get started is to sign up for a free Roboflow account and follow our quickstart guide.

Once you've trained a custom model, update your publishable API Key, model ID, and version in the configuration at the top of main.js.

Contributing

Pull requests are welcome to improve this repo. Ideas for improvements that could be made:

• Taking into account changes in the ground elevation & their impact on the distance calculations
• Intelligently choosing the correct part of the flight log based on the duration of isVideo compared to the duration of the loaded video
• Exporting a JSON file of the detected objects
• Adding a CLI for processing outside of a web browser
• Rendering the flight video and predictions into a single image (patching video frames together)
• Video controls (play/pause, scrubbing)
• Option to show the video in a static position vs flying over the flight path
• Add smoothing to the video positioning to account for the flight-log only having a 100ms resolution
• Allowing dynamically swapping the model endpoint & version in the UI to easily try other models in the UI without having to change the code
• Improve the solar panel model or swimming pool model to make better predictions

dji-aerial-georeferencing's Issues

Can this project implement pixel coordinate conversion in case of UAV with pitch and roll angles?

How to accomodate for camer tilit ?

Thanks for ope-sourcing your work.
I have a question regarding camera movement for your current implementation you are assuming that camera is looking down thus drone's position will always be in the center of image but how can we adjust the position error when our drone is looking at an angle.
If possible can you also recommend any book or resource.

Is this repository working or is it still in development? TAILWIND_MODE errors.

I've read the blog post on roboflow and tried to follow along on the Try it in your Browser and Run it Locally guide in the ReadMe, but am having difficulties getting them to work. More specifically, the deployed GitHub page does not exist, and running the npm run build:dev returns TAILWIND_MODE errors.

I've also included screenshots of my outputs in the Roboflow forums post.

handle cases where the pitch value is different from -90

        center = Point([self.observation["longitude"], self.observation["latitude"]])  # Convert longitude and latitude to GeoJSON Point
        altitude = float(self.observation["ascent(feet)"]) * 0.3048  # Convert ascent to meters
        fov_atan = math.tan(self.fov)
        diagonal_distance = altitude * fov_atan
        distance = diagonal_distance / 2
        bearing = (float(self.observation["compass_heading(degrees)"]) - 90) % 360 # Calculate bearing
        offset = math.atan(self.video_height / self.video_width) * 180 / math.pi  # Calculate offset based on video resolution

        # Calculate GPS coordinates of the video's four corners
        options = {'units': 'm'}
        self.top_left = rhumb_destination(center, distance, (bearing - offset + 180) % 360 - 180, options).geometry.coordinates
        self.top_left = self.top_left[:: -1]

        self.top_right = rhumb_destination(center, distance, (bearing + offset + 180) % 360 - 180,options).geometry.coordinates
        self.top_right = self.top_right[:: -1]

        self.bottom_right = rhumb_destination(center, distance, (bearing - offset) % 360 - 180, options).geometry.coordinates
        self.bottom_right = self.bottom_right[:: -1]

        self.bottom_left = rhumb_destination(center, distance, (bearing + offset) % 360 - 180, options).geometry.coordinates
        self.bottom_left = self.bottom_left[:: -1]

        # print the GPS coordinates of the video's four corners
        self.current_position = [self.observation["latitude"], self.observation["longitude"]]

        normalized = [self.dy - self.video_height / 2, self.dx- self.video_width / 2]
        distanceFromCenterInPixels = math.sqrt((self.video_width / 2 - self.dx) ** 2 + (self.video_height / 2 - self.dy) ** 2)
        diagonalDistanceInPixels = math.sqrt(self.video_width ** 2 + self.video_height ** 2)
        percentOfDiagonal = distanceFromCenterInPixels / diagonalDistanceInPixels
        distance = percentOfDiagonal * diagonal_distance  # in meters
        angle = math.atan(normalized[0] / (normalized[1] or 0.000001)) * 180 / math.pi
        # if the detection is in the right half of the frame, we need to rotate it 180 degrees

        if normalized[1] >= 0:
            angle += 180

        point = rhumb_destination(center, distance, (bearing + angle) % 360, options).geometry.coordinates
        point = point[:: -1]
       

Certainly! Isuccessfully rewritten the code in Python, and it's functioning well when the gimbal pitch is set to -90. If I want to extend the functionality to handle cases where the pitch value is different from -90, what should I do ?