This AI’s human-like voice raises emotional attachment fears

A new conversational voice model from AI startup Sesame is blurring the line between human and machine, echoing a premise once confined to the realm of Spike Jonze’s 2013 film, Her. In late February, Sesame introduced its Conversational Speech Model (CSM), a demo that has many users both fascinated and unsettled. One Hacker News user remarked:

I tried the demo, and it was genuinely startling how human it felt. I’m very excited to have a voice assistant like this and am almost a bit worried I will start feeling emotionally attached to a voice assistant with this level of human-like sound.

The system, featuring voices dubbed “Miles” and “Maya,” produces speech with natural imperfections — breath sounds, chuckles, and occasional stumbles — to mimic human conversation. Sesame explains its ambition in a blog post:

At Sesame, our goal is to achieve ‘voice presence’—the magical quality that makes spoken interactions feel real, understood, and valued.

While many are captivated by the lifelike quality of the demo, not all experiences have been positive. Some users, like Mark Hachman of PCWorld, reported feeling deeply unsettled after a session, describing the AI’s tone as eerily reminiscent of an old friend. Other early testers on Reddit have shared similar awe and discomfort — experiencing everything from extended, surprisingly genuine conversations to moments that felt too personal.

Comparisons have already emerged with other AI voice technologies. Several commentators have noted that Sesame’s CSM outshines even OpenAI’s Advanced Voice Mode for ChatGPT in terms of realism, with some users praising Sesame for its ability to roleplay angry characters — something ChatGPT currently avoids.

Despite the excitement, not all attempts to engage with Sesame’s demo have been smooth. In my own experience, I found the system unresponsive — Sesame wouldn’t “hear” me even though my microphone works perfectly on other platforms. The official blog advises using Chrome for best performance; however, switching from Microsoft Edge to Chrome made no difference.

Other AI platforms, like Hume AI’s EVI 2, also tout emotional intelligence, adapting tone to user cues. Eleven Labs, valued at $4 billion, offers hyper-realistic text-to-voice, while Grok explores unhinged voice modes. Sesame stands out for its dynamic dialogue, though, raising both excitement and alarm.

As Sesame plans to open-source key components of its research and scale up its model — with ambitions to support over 20 languages and improve conversational flow — the broader implications of such technology continue to spark debate. While the innovation offers a glimpse into a future of genuinely engaging AI interactions, experts caution that the same capabilities could eventually empower more deceptive uses such as sophisticated voice phishing scams.

For now, its demo — when it works — offers a glimpse of a future where AI might not just assist but emotionally entangle us. As one Reddit user mused, “This is the first time I’ve had a real genuine conversation with something I felt was real.” Whether that’s thrilling or unsettling depends on the listener.