by Joel McConvey

February 6, 2024 - Generative AI could be listening to your phone calls and hijacking them with fake biometric audio for fraud or manipulation purposes, according to new research published by Security Intelligence. In the wake of a Hong Kong fraud case that saw an employee transfer $25 million in funds to five bank accounts after a virtual meeting with what turned out to be audio-video deepfakes of senior management, the biometrics and digital identity world is on high alert, and the threats are growing more sophisticated by the day.

A blog post by Chenta Lee, chief architect of threat intelligence at IBM Security, breaks down how researchers from IBM X-Force successfully intercepted and covertly hijacked a live conversation by using large language models (LLM) to understand the conversation and manipulate it for malicious purposes - without the speakers knowing it was happening.

“Alarmingly,” writes Lee, “it was fairly easy to construct this highly intrusive capability, creating a significant concern about its use by an attacker driven by monetary incentives and limited to no lawful boundary.”

By combining LLM, speech-to-text, text-to-speech, and voice cloning tactics, X-Force was able to dynamically modify the context and content of a live phone conversation. The method eschewed the use of generative AI to create a whole fake voice and focused instead on replacing keywords in context - for example, masking a spoken real bank account number with an AI-generated one. Tactics can be deployed through a number of vectors, such as malware or compromised VOIP services. A three-second audio sample is enough to create a convincing voice clone, and the LLM takes care of parsing and semantics.

“It is akin to transforming the people in the conversation into dummy puppets,” writes Lee, “and due to the preservation of the original context, it is difficult to detect.” With advanced social engineering added to the mix, the size of the attack surface only grows. Outside of fraud, Lee also points to the potential for a new kind of real-time censorship, which could have dire implications for political discourse, journalism, and the general fabric of reality.

Considering the ease with which they were able to create a successful proof of concept for dynamic voice hijacking, Lee says it is crucial to recognize that “trusted and secure AI is not confined to the AI models themselves. The broader infrastructure must be a defensive mechanism for our AI models and AI-driven attacks.”

According to Pindrop, a further complication is that humans are not very good at detecting fake speech. Writing on the firm’s blog, Head of Brand and Digital Experience Laura Fitzgerald cites new research showing that humans could only detect artificially generated speech 73% of the time.

“Using generative AI technology, bad actors can inject voice into real-time streams, leading to significant fraud loss, the spread of misinformation, and damaged brand reputation,” writes Fitzgerald. The firm says its biometric voice engine, Pindrop Pulse, can outperform humans at deepfake detection.

“In our lab testing with 11 million sample test data sets, Pindrop Pulse can detect a deepfake 99% of the time,” says Fitzgerald. The tech processes a call’s metadata to generate predictions and risk scores. The Passport software provides additional risk analysis based on multiple inputs. Risk APIs display liveness scores in real time, and policies can be calibrated to filter deepfake calls.

The capabilities of AI and LLMs are increasing at speed. “AI performance on benchmark charts can show that it’s surpassed humans at several tasks,” writes Fitzgerald, “and the rate at which humans are being surpassed at new tasks is increasing.” Defenses must be nimble and adaptable, as the curve trends upward into unknown territory.