PanoRadar: Revolutionizing Robot Vision with Radio Waves and AI

PanoRadar: Giving robots superhuman vision through radio waves - Earth.com

PanoRadar marks a breakthrough in the ever-advancing world of robotics, addressing the challenge of providing effective perception systems for robots that can function in severe weather and unfavorable conditions. Unlike traditional sensors, PanoRadar uses radio waves to penetrate smoke, fog, and even certain materials, offering a robust alternative to light-based systems. This innovative approach paves the way for more resilient robotic applications, from autonomous vehicles to rescue missions in hazardous environments. Current sensor systems, such as cameras or LiDAR, rely on light, which limits their effectiveness in conditions like heavy smoke or fog. Alternatively, they may rely on traditional radar which can 'see' through walls and other obstacles, but which produce images lacking in detail. Nature, however, shows us that perception need not rely solely on light. Bats use sound wave echoes for navigation, while sharks detect electrical fields from their prey's movements. These examples highlight alternative perception methods that can inform technological innovations, like radio wave-based sensing. Radio waves, with their longer wavelengths, can penetrate challenging conditions and even certain materials, surpassing the limitations of human vision and conventional sensors. Enter researchers from the University of Pennsylvania School of Engineering and Applied Science (Penn Engineering). They have initiated a new era in robot vision with their creation, the PanoRadar. This novel invention enables robots to perceive their surroundings in three dimensions with incredible detail, by transforming simple radio waves into detailed, 3-D images of the environment. "Our initial question was whether we could combine the best of both sensing modalities - the robustness of radio signals, which is resilient to fog and other challenging conditions, and the high resolution of visual sensors," explained Mingmin Zhao, assistant professor of computer and information science. The birth of PanoRadar is the novel answer to this complex challenge. It has opened new possibilities for robotic perception in environments where traditional systems fail. Think about how a lighthouse works. It sweeps its beam in a circle, scanning the entire horizon, revealing the presence of ships and coastal features. That's how the PanoRadar operates. It consists of a rotating vertical array of antennae that scan the surroundings. These antennae send out radio waves and listen for their reflections from the environment. But PanoRadar is not just a simple scanner. It intelligently combines measurements from all rotation angles, enhancing its imaging resolution. "The key innovation is in how we process these radio wave measurements. Our signal processing and machine learning algorithms are able to extract rich 3D information from the environment," explained Zhao. This novel approach allows PanoRadar to achieve imaging resolution comparable to that of LiDAR, but at a fraction of the cost. Implementing high-resolution imaging while the robot is moving presented a significant challenge. The team needed to combine measurements from various positions with sub-millimeter accuracy. "Even small motion errors can significantly impact the imaging quality," said Haowen Lai, lead author of the paper. In addition, they had to train their system rigorously to comprehend and interpret the complex data it receives, to ensure that it could accurately identify and understand various objects and environmental features in real time. "Indoor environments have consistent patterns and geometries. We leveraged these patterns to help our AI system interpret the radar signals, similar to how humans learn to make sense of what they see," explained Gaoxiang Luo. Thanks to machine learning algorithms, the model was able to improve its understanding against reality using LiDAR data. PanoRadar's capabilities extend beyond what traditional sensors can perceive. "Our field tests across different buildings showed how radio sensing can excel where traditional sensors struggle," said Liu. "The system maintains precise tracking through smoke and can even map spaces with glass walls." This advantage comes from the ability of radio waves to pass through airborne particles with ease, allowing the system to detect elements that LiDAR often misses, such as glass surfaces. Additionally, PanoRadar's high resolution enables it to identify people accurately, an essential capability for applications like autonomous vehicles and rescue operations in challenging environments. These attributes make PanoRadar a game-changer for robots operating in difficult conditions, where reliability and precision are paramount. The research will be presented at the 2024 International Conference on Mobile Computing and Networking (MobiCom). -- - Like what you read? Subscribe to our newsletter for engaging articles, exclusive content, and the latest updates.

Giving robots superhuman vision using radio signals

In the race to develop robust perception systems for robots, one persistent challenge has been operating in bad weather and harsh conditions. For example, traditional, light-based vision sensors such as cameras or LiDAR (Light Detection And Ranging) fail in heavy smoke and fog. However, nature has shown that vision doesn't have to be constrained by light's limitations -- many organisms have evolved ways to perceive their environment without relying on light. Bats navigate using the echoes of sound waves, while sharks hunt by sensing electrical fields from their prey's movements. Radio waves, whose wavelengths are orders of magnitude longer than light waves, can better penetrate smoke and fog, and can even see through certain materials -- all capabilities beyond human vision. Yet robots have traditionally relied on a limited toolbox: they either use cameras and LiDAR, which provide detailed images but fail in challenging conditions, or traditional radar, which can see through walls and other occlusions but produces crude, low-resolution images. Now, researchers from the University of Pennsylvania School of Engineering and Applied Science (Penn Engineering) have developed PanoRadar, a new tool to give robots superhuman vision by transforming simple radio waves into detailed, 3D views of the environment. "Our initial question was whether we could combine the best of both sensing modalities," says Mingmin Zhao, Assistant Professor in Computer and Information Science. "The robustness of radio signals, which is resilient to fog and other challenging conditions, and the high resolution of visual sensors." In a paper to be presented at the 2024 International Conference on Mobile Computing and Networking (MobiCom), Zhao and his team from the Wireless, Audio, Vision, and Electronics for Sensing (WAVES) Lab and the Penn Research In Embedded Computing and Integrated Systems Engineering (PRECISE) Center, including doctoral student Haowen Lai, recent master's graduate Gaoxiang Luo and undergraduate research assistant Yifei (Freddy) Liu, describe how PanoRadar leverages radio waves and artificial intelligence (AI) to let robots navigate even the most challenging environments, like smoke-filled buildings or foggy roads. PanoRadar is a sensor that operates like a lighthouse that sweeps its beam in a circle to scan the entire horizon. The system consists of a rotating vertical array of antennas that scans its surroundings. As they rotate, these antennas send out radio waves and listen for their reflections from the environment, much like how a lighthouse's beam reveals the presence of ships and coastal features. Thanks to the power of AI, PanoRadar goes beyond this simple scanning strategy. Unlike a lighthouse that simply illuminates different areas as it rotates, PanoRadar cleverly combines measurements from all rotation angles to enhance its imaging resolution. While the sensor itself is only a fraction of the cost of typically expensive LiDAR systems, this rotation strategy creates a dense array of virtual measurement points, which allows PanoRadar to achieve imaging resolution comparable to LiDAR. "The key innovation is in how we process these radio wave measurements," explains Zhao. "Our signal processing and machine learning algorithms are able to extract rich 3D information from the environment." One of the biggest challenges Zhao's team faced was developing algorithms to maintain high-resolution imaging while the robot moves. "To achieve LiDAR-comparable resolution with radio signals, we needed to combine measurements from many different positions with sub-millimeter accuracy," explains Lai, the lead author of the paper. "This becomes particularly challenging when the robot is moving, as even small motion errors can significantly impact the imaging quality." Another challenge the team tackled was teaching their system to understand what it sees. "Indoor environments have consistent patterns and geometries," says Luo. "We leveraged these patterns to help our AI system interpret the radar signals, similar to how humans learn to make sense of what they see." During the training process, the machine learning model relied on LiDAR data to check its understanding against reality and was able to continue to improve itself. "Our field tests across different buildings showed how radio sensing can excel where traditional sensors struggle," says Liu. "The system maintains precise tracking through smoke and can even map spaces with glass walls." This is because radio waves aren't easily blocked by airborne particles, and the system can even "capture" things that LiDAR can't, like glass surfaces. PanoRadar's high resolution also means it can accurately detect people, a critical feature for applications like autonomous vehicles and rescue missions in hazardous environments. Looking ahead, the team plans to explore how PanoRadar could work alongside other sensing technologies like cameras and LiDAR, creating more robust, multi-modal perception systems for robots. The team is also expanding their tests to include various robotic platforms and autonomous vehicles. "For high-stakes tasks, having multiple ways of sensing the environment is crucial," says Zhao. "Each sensor has its strengths and weaknesses, and by combining them intelligently, we can create robots that are better equipped to handle real-world challenges."

Giving robots superhuman vision using radio signals

In the race to develop robust perception systems for robots, one persistent challenge has been operating in bad weather and harsh conditions. For example, traditional, light-based vision sensors such as cameras or LiDAR (Light Detection And Ranging) fail in heavy smoke and fog. However, nature has shown that vision doesn't have to be constrained by light's limitations -- many organisms have evolved ways to perceive their environment without relying on light. Bats navigate using the echoes of sound waves, while sharks hunt by sensing electrical fields from their prey's movements. Radio waves, whose wavelengths are orders of magnitude longer than light waves, can better penetrate smoke and fog, and can even see through certain materials -- all capabilities beyond human vision. Yet robots have traditionally relied on a limited toolbox: They either use cameras and LiDAR, which provide detailed images but fail in challenging conditions, or traditional radar, which can see through walls and other occlusions but produces crude, low-resolution images. A new way to see Now, researchers from the University of Pennsylvania School of Engineering and Applied Science (Penn Engineering) have developed PanoRadar, a new tool to give robots superhuman vision by transforming simple radio waves into detailed, 3D views of the environment. "Our initial question was whether we could combine the best of both sensing modalities," says Mingmin Zhao, Assistant Professor in Computer and Information Science. "The robustness of radio signals, which is resilient to fog and other challenging conditions, and the high resolution of visual sensors." In a paper to be presented at the International Conference on Mobile Computing and Networking (MobiCom 2024), held Nov. 18-22 in Washington D.C., Zhao and his team describe how PanoRadar leverages radio waves and artificial intelligence (AI) to let robots navigate even the most challenging environments, like smoke-filled buildings or foggy roads. The team, from the Wireless, Audio, Vision, and Electronics for Sensing (WAVES) Lab and the Penn Research In Embedded Computing and Integrated Systems Engineering (PRECISE) Center, includes doctoral student Haowen Lai, recent master's graduate Gaoxiang Luo and undergraduate research assistant Yifei (Freddy) Liu. Spinning like a lighthouse PanoRadar is a sensor that operates like a lighthouse that sweeps its beam in a circle to scan the entire horizon. The system consists of a rotating vertical array of antennas that scans its surroundings. As they rotate, these antennas send out radio waves and listen for their reflections from the environment, much like how a lighthouse's beam reveals the presence of ships and coastal features. Thanks to the power of AI, PanoRadar goes beyond this simple scanning strategy. Unlike a lighthouse that simply illuminates different areas as it rotates, PanoRadar cleverly combines measurements from all rotation angles to enhance its imaging resolution. While the sensor itself is only a fraction of the cost of typically expensive LiDAR systems, this rotation strategy creates a dense array of virtual measurement points, which allows PanoRadar to achieve imaging resolution comparable to LiDAR. "The key innovation is in how we process these radio wave measurements," explains Zhao. "Our signal processing and machine learning algorithms are able to extract rich 3D information from the environment." Teaching the AI One of the biggest challenges Zhao's team faced was developing algorithms to maintain high-resolution imaging while the robot moves. "To achieve LiDAR-comparable resolution with radio signals, we needed to combine measurements from many different positions with sub-millimeter accuracy," explains Lai, the lead author of the paper. "This becomes particularly challenging when the robot is moving, as even small motion errors can significantly impact the imaging quality." Another challenge the team tackled was teaching their system to understand what it sees. "Indoor environments have consistent patterns and geometries," says Luo. "We leveraged these patterns to help our AI system interpret the radar signals, similar to how humans learn to make sense of what they see." During the training process, the machine learning model relied on LiDAR data to check its understanding against reality and was able to continue to improve itself. "Our field tests across different buildings showed how radio sensing can excel where traditional sensors struggle," says Liu. "The system maintains precise tracking through smoke and can even map spaces with glass walls." This is because radio waves aren't easily blocked by airborne particles, and the system can even "capture" things that LiDAR can't, like glass surfaces. PanoRadar's high resolution also means it can accurately detect people, a critical feature for applications like autonomous vehicles and rescue missions in hazardous environments. Looking ahead, the team plans to explore how PanoRadar could work alongside other sensing technologies like cameras and LiDAR, creating more robust, multi-modal perception systems for robots. The team is also expanding their tests to include various robotic platforms and autonomous vehicles. "For high-stakes tasks, having multiple ways of sensing the environment is crucial," says Zhao. "Each sensor has its strengths and weaknesses, and by combining them intelligently, we can create robots that are better equipped to handle real-world challenges."

PanoRadar: Robots gain superhuman vision with AI and radio waves

The device improves on the low-resolution images produced by conventional radar by processing radio waves using AI algorithms. According to researchers, this makes it possible for robots to precisely navigate through challenging situations and obstructions like smoke, glass, and walls -- situations in which conventional sensors are inadequate. "This innovation in AI-powered perception has the potential to improve multi-modal systems, helping robots operate more effectively in challenging environments like search and rescue missions or autonomous vehicles," said the team, in a video posted on YouTube. One recurring issue in the quest to create reliable perception systems for robots has been functioning in inclement weather and other challenging environments. For instance, in dense smoke and fog, conventional light-based vision sensors like cameras or LiDAR (Light Detection and Ranging) are ineffective. According to researchers, nature has demonstrated, however, that vision need not be limited by the restrictions of light; numerous animals have developed methods of perceiving their surroundings independently of light. Sharks hunt by detecting electrical fields from the motions of their prey, whereas bats use the echoes of sound waves to navigate.