Sound Source Tracking as a Heuristic for Frontier Exploration in Search and Rescue using a Quadruped Mobile Robot
AKA: Teaching a robot dog how to play marco polo 🗣️...🤖...🐕
This repository covers the development of the research performed by Francesco Marrato while completing his MASc in Electrical and Computer Engineering at Queen's University. This research was made possible by Queen's University, Ingenuity Labs, and Professor Joshua Marshall (MASc Supervisor).
This repo outlines the development of a system that integrates autonomous exploration agents for disasters scenarios with robots that have the ability to interpret their surroundings through sound. We developed an autonomous frontier exploration system that uses sound source tracking of human voices as a novel heuristic for search. Our system is capable of exploring previously unknown environments and locating a target sound source which emits human speech. Our system was implemented on a Boston Dynamics Spot quadrupedal robot equipped with a four-microphone array. Field trials were conducted in a laboratory environment and in simulated disaster scenarios in both natural forest and urban environments. This research stands to demonstrate that sound source tracking tuned for the human voice is a suited heuristic to bring autonomous search agents used in disaster scenarios closer in capability to the humans they assist.
-
Condensed conference style paper available here.
-
Full thesis available here.
-
Video demonstration here.
A poster presented at IROS 2023. Accepted as a submission for late breaking results.
A poster submitted to the 2023 NSERC Canadian Robotics Network (NCRN) AGM poster session hosted by Toronto Robotics and AI Laboratory at the University of Toronto.
A poster submitted to the Robotics and AI Symposium 2022 (RAIS2022) hosted by Ingenuity Labs at Queen's University in Kingston, ON.
A short video summary of my research was submitted to the GRADFlix video demonstration. GRADFlix is an event hosted by the School of Graduate Studies and Postdoctoral Affairs at Queen's University to show off what current research is going on at Queen's.
I spoke about Ingenuity Labs and my research on Queen's campus public radio. You can listen to the recording on Spotify or Apple Music.
Provided are short, high level, breakdowns of some of the major sub components that make up the frontier exploration system. For a thorough breakdown, justification, and analysis of the entire system please review the provided thesis.
This research included the development of a custom backpack. The backpack included the following;
- An Intel NUC for ROS compatibility
- A ReSpeaker 4-Mic Array
- A Ublox Neo M9N GPS
- A Microstrain 3dm-gx5-25 imu
- A 12V 150W Power Supply
A ruggedized frame was printed using a continuous carbon fiber embedding process to provide impact and crush resistance. Additionally, the microphone mast was designed to be sacrificial, snapping off in a controlled manner in the event of a rollover. Full Fusion360 files are available here.
The system identifies human speech through a four stage process, providing a unit vector in the estimated direction of the human speaker to the rest of the ROS systems.
Using the ODAS software as a base, we extend the sound source tracking ability by first separating each tracked source. We feed each tracked source through a YAMnet classification model. Each tracked source is given a confidence rating on its likelyhood of containing human speech.
Local occupancy grids are generated by Spot with a range of ~3 m in all directions. Our system stitches occupancy grids togther as Spot explores. Spot begins with no prior knowledge of its environment and progressively maps obstacles as it searches.
Spot builds a network graph of safe travel positions as it explores its environment. This node graph is used to plan safe travel trajectories when exploring complex unknown environments. Each time a new frontier is successfully achieved, it is added to the node graph, expanding Spot's opperating space.
Frontiers are evaluated based on their similarity in direction compared to the estimated vector of the human speaker. The heuristic for frontier rading is demonstrated below.
Historically, frontier exploration exploits lowest cost or highest potential for map expansion as the heuristic for search. This research explores the use of a unique heuristic, utilizing a beyond line of sight modality for applications in search and rescue.
A state machine was implemented using the SMACH library, connecting all the mentioned components. The design philosophy was that the state machine would access individual system nodes through ROS actions and services while each node managed its own data. The state machine consists of Start, Plan, Execute, Recover, Update, and Check states for nominal use, with an eight Error state reserved to perform a safe shutdown under any unforeseen circumstances.
This project is completed:
- Thesis succesfuly defended
- Corrections submitted
- Repository updated
If you wish to get in touch with Francesco Marrato.
Email: [email protected]
Email: [email protected]
LinkedIn: https://www.linkedin.com/in/francesco-marrato
Github: https://github.com/TankyFranky
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent