Eerily lifelike AI voice demo sparks amazement and discomfort on-line

Gavin Purcell, co-host of the AI for People podcast, posted an instance video on Reddit the place the human pretends to be an embezzler and argues with a boss. It is so dynamic that it is troublesome to inform who the human is and which one is the AI mannequin. Judging by our personal demo, it is completely able to what you see within the video.

“Close to-human high quality”

Underneath the hood, Sesame’s CSM achieves its realism through the use of two AI fashions working collectively (a spine and a decoder) based mostly on Meta’s Llama structure that processes interleaved textual content and audio. Sesame skilled three AI mannequin sizes, with the biggest utilizing 8.3 billion parameters (an 8 billion spine mannequin plus a 300 million parameter decoder) on roughly 1 million hours of primarily English audio.

Sesame’s CSM would not comply with the normal two-stage strategy utilized by many earlier text-to-speech techniques. As a substitute of producing semantic tokens (high-level speech representations) and acoustic particulars (fine-grained audio options) in two separate levels, Sesame’s CSM integrates right into a single-stage, multimodal transformer-based mannequin, collectively processing interleaved textual content and audio tokens to provide speech. OpenAI’s voice mannequin makes use of the same multimodal strategy.

In blind assessments with out conversational context, human evaluators confirmed no clear desire between CSM-generated speech and actual human recordings, suggesting the mannequin achieves near-human high quality for remoted speech samples. Nonetheless, when supplied with conversational context, evaluators nonetheless constantly most popular actual human speech, indicating a spot stays in totally contextual speech era.

Sesame co-founder Brendan Iribe acknowledged present limitations in a touch upon Hacker Information, noting that the system is “nonetheless too keen and infrequently inappropriate in its tone, prosody and pacing” and has points with interruptions, timing, and dialog stream. “As we speak, we’re firmly within the valley, however we’re optimistic we will climb out,” he wrote.