You pause to think mid-sentence, and the assistant jumps in. You start talking while it is answering, and it ploughs on. These moments feel small, but they are why talking to a machine still feels like talking to a machine.
Human conversation runs on tight timing. We predict when someone is about to finish, we slip in short "mm-hm"s to show we are listening, and we recover gracefully when two people start at once. None of that is in the words. It lives in timing, pitch and overlap.
The old way: take turns, strictly
Traditional voice assistants work like a relay. One component detects that you have stopped speaking, another transcribes, a language model writes a reply, and a final component speaks it. Each hand-off adds delay, and the whole system assumes only one person talks at a time.
The new way: listen and speak at once
In 2024, the French lab Kyutai released Moshi, which its authors describe as the first real-time full-duplex spoken language model. Instead of a relay, Moshi models the user's speech and its own speech as parallel streams, so it can handle overlaps and interruptions. Its authors report a theoretical latency of 160 milliseconds and about 200 milliseconds in practice, close to the gaps people leave for each other.
Others are attacking the same problem from different angles. Sesame's Conversational Speech Model uses the history of a conversation to make an AI's voice sound contextually appropriate, a goal the company calls "voice presence." OpenAI's Realtime API now offers semantic turn detection, which judges whether you have finished speaking from what you said, and waits longer when you sound hesitant, rather than relying on silence alone.
Why it matters for Social AI
All of these approaches learn timing from examples. And the examples they need are not scripted or one-sided. They are real conversations, recorded with each speaker on a separate track, so a model can see exactly when people overlap, hand off the floor and backchannel.
Two-person phone calls are only the start. Customer service calls, sales meetings and group calls involve three, five or ten people, each with their own rhythm. Voice AI that works in those settings needs to learn from them.
Crosstalk provides consented, multi-speaker recordings with per-speaker tracks and turn-level annotation, including overlaps and backchannels. See our conversation datasets or read the paper.
Sources: Moshi paper (arXiv), The Decoder on Moshi, Sesame CSM coverage, OpenAI voice activity detection guide.