Centre for Data Science & AI

Hearing loss is a debilitating condition that impacts huge numbers of the population worldwide, with approximately 430 million people suffering from hearing impairments. Unfortunately, this number is projected to increase to 700 million people by 2050. Existing hearing aid devices support both audio and video, however these devices encounter significant challenges:

• Audio-based hearing aids encounter various issues, such as difficulties in distinguishing voices in noisy environments and effectively amplifying sounds.
• The integration of video into hearing aids may raise ethical concerns, as it could be perceived as unauthorized filming. In today's context, hearing aids enhanced with visual information face several limitations, with a notable one being the use of face masks during the COVID-19 era.

In the James Watt School of Engineering, Professors Qammer H. Abbasi, Muhammad Imran and their teams are working on an EPSRC-funded project called COG-MHEAR AI Enabled Future Hearing Aid Devices (https://cogmhear.org/). The project focuses on next-generation hearing aids, achievable through radio frequency (RF) sensing of lip and mouth movements. Lip-reading through RF sensing can provide highly accurate cues to hearing aids by identifying spoken sounds and detecting speech patterns using machine learning (ML) and deep learning (DL) techniques. 

This video provides an overview of the experimental process for radar-based lip reading, showcasing its operation with the assistance of AI.

This approach offers significant advantages for multi-modal hearing aid devices, as it functions effectively in various environments, including those with additional peripheral noise, where privacy-preservation is required, and under various lighting conditions. The proposed lip-reading framework holds potential value in numerous applications, including hearing aid devices, biometric security, and voice-enabled control systems in smart homes and car infotainment systems. COG-MHEAR leverages Wi-Fi and radar signals, which are omnipresent, making it a crucial enabling technology for integrating AI/ML into engineering.

This video provides an overview of the experimental process for Wi-Fi based lip reading, showcasing its operation with the assistance of AI.

In their research, they undertook a series of experiments utilizing RF sensing-based technologies, encompassing radar and Wi-Fi signals. Through these experiments, they successfully demonstrated RF sensing's ability to interpret lip movements, even in scenarios where individuals wear face masks. Initially, the primary objective revolved around capturing vowels, words, and sentences using these RF-based technologies. This accomplishment stands as a notable breakthrough with substantial implications for the advancement of multi-modal hearing aids. We are now working with end users and industrial partners, such as Sonova, to bring this work to reality.

---

First published: 16 October 2023