otoSpeech is the training set for TurnBench
Sesame has released TurnBench, an open benchmark for turn-taking in spoken dialogue. Its 104-hour training set is otoSpeech, collected and hand-annotated by oto under the benchmark's protocol.
- 104.94 hours of full-duplex, two-speaker English conversation
- 420 conversations, averaging about 15 minutes each
- 17 event labels, hand-annotated per speaker
- 48 kHz channel-separated audio, one file per speaker
TurnBench leaderboard and viewer ↗ · The training set on Hugging Face ↗
The benchmark
TurnBench evaluates whether a system can tell when a speaker has finished and when someone is interrupting. It pairs a 30-hour hand-labeled corpus of dyadic conversation with a fixed evaluation protocol, and scores systems on recall, false-positive rate, and latency across six conversation types.
Sesame reports that no system on the board is fast, high-recall, and low on false positives at the same time. For the method, the results, and the leaderboard, read their material rather than ours:
- turnbench.sesame.com
- TurnBench: A Multi-Domain Benchmark for Turn-Taking Dynamics in Spoken Dialogue, arXiv:2608.25218
What we contributed
The training split. otoSpeech-full-duplex-turn-104h is 420 two-party conversations recorded with a separate channel for each speaker, then annotated by hand under the same protocol the benchmark uses for its evaluation corpus.
Researchers from oto are also co-authors on the benchmark paper.
Sessions run on topics and scenarios rather than scripts, so the recordings keep the behavior that makes turn-taking hard: overlap, interruption, backchannels, fillers, laughter, floor holds, and silence.
| Conversation type | Sessions |
|---|---|
| Narrative/Storytelling | 81 |
| Argumentative/Deliberative | 77 |
| Casual/Spontaneous | 73 |
| Collaborative/Problem-Solving | 65 |
| Task-Oriented/Transactional | 63 |
| Instructional | 61 |
Each speaker channel ships with an SRT file carrying millisecond time intervals, a bracketed event label, and the transcript text.
1
00:00:12,340 --> 00:00:13,020
[Acknowledgement Backchannel] Okay.
The label set is finer than the two events the benchmark scores. It separates three kinds of backchannel (acknowledgement, continuer, reaction) and four kinds of interruption, splitting floor-taking from non-floor-taking and competitive from cooperative. Strong floor holds, awkward silence, fillers, laughter, and channel bleed are labeled as well. Work that needs a coarser target can collapse the labels; work on backchannel or interruption type does not have to start by inventing a taxonomy.
Access and license
The dataset is gated, and access is granted by manual review. Audio is 48 kHz 32-bit float WAV: one file per speaker plus a time-aligned stereo rendition. The full repository is about 290 GB, and the two separated speaker streams can be fetched without the combined file.
It is released under the oto Speech License v1.0, a non-commercial research license, not under the open terms we use for some other releases. The license prohibits voice-identity uses, including voice cloning, speaker identification, and re-identification. Commercial and production use needs a separate written agreement. For-profit organizations need written approval for a defined research project.
For licensing or a use we have not anticipated, write to agent@oto.earth.