This is how the new Eario gesture recognition system works, which uses sound like sonar and can work in total darkness.

Cornell University engineers have developed a new wearable device that can track a person’s facial expressions using sonar and recreate them in a digital avatar.

This will take recording cameras out of the equation, potentially reducing current privacy concerns. Computers will be able to use this information.

EarIO, as the team calls the device, consists of a headset with a microphone and speaker on each side and can be connected to any conventional headset. Each speaker emits pulses of sound beyond human hearing into the user’s face, and the echo is picked up by microphones.

Echo profiles change slightly depending on how the wearer’s skin moves, stretches, and wrinkles with different facial expressions or conversations.

Specially trained algorithms can recognize these echo profiles and quickly reconstruct the user’s facial expression.showing her in a digital avatar.

“Using the power of artificial intelligence, the algorithm finds complex relationships between muscle movements and facial expressions that the human eye cannot determine.“, says Ke Li, co-author of the study.

The team tested the EarIO system on 16 participants by running the algorithm on a regular smartphone. And, of course, the device could reconstruct facial expressions just as well as a conventional camera.

Background noise, such as wind, conversations, or street noise, did not prevent them from recognizing faces.

The team claims that sonar has some advantages over using a camera.. Acoustic data requires much less power and processing power, which also means the device can be smaller and lighter.

Cameras can also capture a lot of personal data that the user doesn’t want to share, so sonar can be more private.

Regarding the use that can be given to this technology, it can be a convenient way to replicate your physical expression in a digital avatar for games, virtual reality, or the metaverse.

The team says there is still work to be done to eliminate other hindrances, such as when the user turns their head, and improve the AI ​​algorithm learning system.