AI-Powered Drones and Machine Learning Revolutionize Sperm Whale Tracking for Project CETI

New methods for whale tracking and rendezvous using autonomous robots

Project CETI (Cetacean Translation Initiative) aims to collect millions to billions of high-quality, highly contextualized vocalizations in order to understand how sperm whales communicate. But finding the whales and knowing where they will surface to capture the data is challenging -- making it difficult to attach listening devices and collect visual information. Today, a Project CETI research team led by Stephanie Gil, Assistant Professor of Computer Science at the Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS), have proposed a new reinforcement learning framework with autonomous drones to find sperm whales and predict where they will surface. The research is published in Science Robotics. This new study uses various sensing devices, such as Project CETI aerial drones with very high frequency (VHF) signal sensing capability that leverage signal phase along with the drone's motion to emulate an 'antenna array in air' for estimating directionality of received pings from CETI's on-whale tags. It demonstrates that it's possible to predict when and where a whale may surface by using these various sensor data as well as predictive models of sperm whales dive behavior. With that information, Project CETI can now design algorithms for the most efficient route for a drone to rendezvous -- or encounter -- a whale at the surface. This also opens up possible conservation applications to help ships avoid striking whales while at the surface. Presenting the Autonomous Vehicles for whAle Tracking And Rendezvous by remote Sensing, or AVATARS framework, this study jointly develops two interrelated components of autonomy and sensing: autonomy, which determines the positioning commands of the autonomous robots to maximize visual whale encounters; and sensing, which measures the Angle-of-Arrival (AOA) from whale tags to inform the decision-making process. Measurements from our autonomous drone to surfaced tags, acoustic AOA from existing underwater sensors, and whale motion models from previous biological studies of sperm whales are provided as inputs to the AVATARS autonomous decision-making algorithm, which in turn aims to minimize missed rendezvous opportunities with whales. AVATARS is the first co-development of VHF sensing and reinforcement learning decision-making for maximizing rendezvous of robots and whales at sea. A well-known application of time-critical rendezvous is used with rideshare apps, which uses real-time sensing to note the dynamic paths and positions of drivers and potential riders. When a rider requests a ride, it can assign a driver to rendezvous with the rider as efficiently and as timely as possible. Project CETI's case is similar in that they are real-time tracking the whale, with the goal of coordinating the drone's rendezvous to meet the whale at the surface. This research advances Project CETI's goal of obtaining millions to billions of high-quality, highly contextualized whale vocalizations. The addition of diverse types of data will improve location estimates and routing algorithms -- helping Project CETI meet that goal more efficiently. "I'm excited to contribute to this breakthrough for Project CETI. By leveraging autonomous systems and advanced sensor integration, we're able to solve key challenges in tracking and studying whales in their natural habitats. This is not only a technological advancement, but also a critical step in helping us understand the complex communications and behaviors of these creatures," said Gil. "This research is a major milestone for Project CETI's mission. We can now significantly enhance our ability to gather high-quality and large-scale dataset on whale vocalizations and the associated behavioral context, putting us one step closer to better listening to and translating what sperm whales are saying," said David Gruber, Founder and Lead of Project CETI. "'This research was an amazing opportunity to test our systems and algorithms in a challenging marine environment. This interdisciplinary work, that combines wireless sensing, artificial intelligence and marine biology, is a prime example of how robotics can be part of the solution for further deciphering the social behavior of sperm whales," said Ninad Jadhav, Harvard University PhD candidate and first author on the paper. "This project provides an excellent opportunity to test our algorithms in the field, where robotics and artificial intelligence can enrich data collection and expedite research for broader science in language processing and marine biology, ultimately protecting the health and habitat of sperm whales," said Sushmita Bhattacharya, a postdoctoral researcher in Gil's REACT Lab at SEAS.

New method for finding sperm whales kind of works like a rideshare app

Marine biologists are inching closer to understanding the ins and outs of sperm whale communication. But in order to decode what the cetaceans are saying, they must first need to find them and know where they will surface. This is no easy feat, since sperm whales can dive over 10,000 feet andstay way below the surface for up to 60 minutes. Their habitats themselves stretch for thousands of miles. Now, scientists from Project CETI (Cetacean Translation Initiative) and Harvard University are proposing a new method for finding sperm whales and predicting where they will surface using autonomous robots and a rich combination of sensor data. The method is detailed in a study published October 30 in the journal Science Robotics. [Related: Sperm whales may have their own 'alphabet.'] This method is a chance for scientists to test out new algorithms, sensing data, and artificial intelligence in a challenging environment. First launched in 2020, Project CETI is working to collect vocalizations to decipher how sperm whales communicate with one another. They have used tags affixed to the whales to track them in real-time and customized drones to follow their behaviors. The new study uses various sensing devices, including aerial drones equipped with very high frequency (VHF) signal sensing capability. These devices can use leverage signal phase and the drone's motion to imitate an "antenna array in air." This can help estimate which direction pings from one of the tagged whales CETI is tracking is traveling. According to the team, this shows how to predict when and where a sperm whale may surface using sensor data to predict their dive behavior. With that data, Project CETI can now create algorithms for the most efficient route for a drone to or encounter a whale at the ocean's surface. In the future, this could be applied to conservation methods, including helping ships avoid hitting whales when they are surfacing. They call this method the Autonomous Vehicles for Whale Tracking And Rendezvous by remote Sensing (AVATARS) framework. It uses two interrelated components-autonomy and sensing. Autonomy determines the positioning commands for autonomous robots deployed to increase the chances of visually spotting whales. Sensing measures the angle-of-arrival from whale tags to better inform the decision-making process of where to deploy the robots. The data taken from autonomous drones, surfaced tags, underwater sensors, and whale motion models from previous biological studies of sperm whales are all put into the AVATARS autonomous decision-making algorithm, which then aims to minimize missed opportunities to rendezvous with sperm whales. [Related: Sperm whale clans tell each other apart by their accents.] A similar and more well-known application of this kind of time-critical rendezvous method is in rideshare apps. They use real-time sensing to note the changing paths and positions to connect drivers with potential riders. When a rider requests a ride, the app can assign a driver to get to the rider as quickly and efficiently as possible. According to the team, Project CETI's new method works in a similar way, by tracking the whales in real-time with the eventual goal of coordinating the drone's rendezvous to meet the whale at the surface. "I'm excited to contribute to this breakthrough for Project CETI," study co-author and Harvard University computer scientist Stephanie Gil said in a statement. "By leveraging autonomous systems and advanced sensor integration, we're able to solve key challenges in tracking and studying whales in their natural habitats. This is not only a technological advancement, but also a critical step in helping us understand the complex communications and behaviors of these creatures."