| --- |
| license: apache-2.0 |
| license_name: mixed-per-model |
| license_link: LICENSE |
| library_name: onnx |
| pipeline_tag: audio-classification |
| language: |
| - en |
| tags: |
| - end-of-turn-detection |
| - turn-taking |
| - voice-agents |
| - speech |
| - onnx |
| - from-scratch |
| datasets: |
| - pipecat-ai/smart-turn-data-v3.2-train |
| - Scicom-intl/semantic-vad-eot |
| - li2017dailydialog/daily_dialog |
| metrics: |
| - roc_auc |
| - average_precision |
| model-index: |
| - name: TurnWave |
| results: |
| - task: |
| type: audio-classification |
| name: End-of-turn detection |
| dataset: |
| type: livekit/eot-bench-data |
| name: eot-bench (English) |
| split: validation |
| metrics: |
| - type: roc_auc |
| value: 0.77 |
| name: AUC |
| - type: average_precision |
| value: 0.602 |
| name: Average precision |
| - type: false_cutoff_rate |
| value: 42.1 |
| name: False cutoffs @300ms latency budget (%) |
| - type: false_cutoff_rate |
| value: 17.2 |
| name: False cutoffs @600ms latency budget (%) |
| --- |
| |
| # TurnWave — end-of-turn detection for voice agents |
|
|
| Decides whether a caller has **finished speaking** or is only pausing, so a voice |
| agent neither interrupts them nor leaves an awkward silence. It replaces the fixed |
| 300–700 ms silence timeout most pipelines still use. |
|
|
| **Trained from scratch — no pretrained weights anywhere.** A causal transformer |
| (RoPE, RMSNorm, SwiGLU) over the transcript tail, and a CNN over log-mel |
| spectrograms for prosody. Even the log-mel front end is hand-built on `torch.stft`, |
| so there is no torchaudio or librosa dependency. |
|
|
| ## Benchmark |
|
|
| Scored by [LiveKit's eot-bench](https://github.com/livekit/eot-bench) harness on |
| real human-to-agent conversation, using their code and published baselines. Lower |
| is better; **bold marks the best per column.** |
|
|
| | model | false cutoffs @300 ms ↓ | @600 ms ↓ | latency @5% cutoff ↓ | |
| |---|---|---|---| |
| | VAD baseline | 55.6% | 21.7% | 1600 ms | |
| | **TurnWave audio branch (this model)** | 42.1% | 17.2% | 1195 ms | |
| | SmartTurn v3.2 | 35.2% | 14.8% | 1051 ms | |
| | LiveKit Turn Detector v1 | **9.9%** | **4.5%** | **543 ms** | |
|
|
| TurnWave beats the VAD baseline on every metric the harness reports. |
|
|
| ## Models in this repo |
|
|
| | file | licence | training data | |
| |---|---|---| |
| | `audio_eot_v2.onnx` | apache-2.0 | trained on smart-turn conversational clips | |
| | `audio_eot.onnx` | cc-by-4.0 | Phase 4; trained on semantic-vad-eot (CC BY 4.0) | |
| | `text_eot.int8.onnx` | cc-by-nc-sa-4.0 | trained on DailyDialog (CC BY-NC-SA 4.0) — non-commercial | |
| | `fusion_eot_v2.onnx` | cc-by-nc-sa-4.0 | fusion head retrained on conversational clips; contains the DailyDialog text branch, so it inherits its terms | |
|
|
| Each model's licence follows its training data, so they differ. `audio_eot_v2` is |
| the one the benchmark above measures and the one to use. |
|
|
| ## Usage |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| from turnwave.infer import TurnDetector # pip install git+https://github.com/Nikhils-G/turnwave |
| |
| detector = TurnDetector(hf_hub_download("Nikhil-09/turnwave", "audio_eot_v2.onnx")) |
| if detector.predict(audio=wav_16k) > 0.5: |
| respond() |
| ``` |
|
|
| 16 kHz mono. The model reads the last 2 seconds ending at the decision point, which |
| sits 0.2 s into the pause — where a live agent decides, and where eot-bench scores. |
|
|
| | model | variant | CPU latency | size | |
| |---|---|---|---| |
| | audio_eot_v2 | fp32 | 4.81 ms | 14.0 MB | |
| | audio_eot | fp32 | 4.52 ms | 14.0 MB | |
| | text_eot | int8 | 5.02 ms | 7.2 MB | |
| | fusion_eot_v2 | fp32 | 38.19 ms | 42.4 MB | |
|
|
| INT8 is not applied blindly: dynamic quantization rewrites MatMul, so it speeds up |
| the transformer and *slows down* the conv-heavy branches. Each model ships whichever |
| variant measured faster. |
|
|
| ## What this project found |
|
|
| The first version of this model scored **AP 0.945** on its own held-out test set and |
| **AUC 0.563** on eot-bench — barely above random. The policy sweep chose thresholds |
| of 0.0 and 1.0, meaning *ignore the model entirely*. |
|
|
| The cause was the training corpus, not the architecture. It derived from a dataset |
| whose own card declares `task_categories: [text-to-speech]` — read speech, whose |
| pauses are reading hesitations rather than conversational turn-yields. The model had |
| learned *"has this sentence finished being read aloud."* |
|
|
| Retraining on conversational data, changing nothing else, lifted AUC to **0.770**. |
| The in-domain score could never have revealed this; only a benchmark on data we did |
| not build could. |
|
|
| ## Limitations |
|
|
| - **English only.** Other languages are in the training data but untested here. |
| - **Behind the production models**, and not a fair comparison: SmartTurn starts from |
| a pretrained Whisper encoder, LiveKit's is a fine-tuned 0.5B LLM distilled from a |
| 7B teacher. This is 3.49M parameters from random initialisation. |
| - **Fusion wins in-domain, not on the benchmark — measured, not guessed.** The |
| fused model reaches AP 0.959 vs 0.905 audio-only on held-out conversational |
| clips (Whisper transcripts), but on eot-bench the text branch scores AUC 0.485 — |
| chance — on real human-agent speech, so fusion there is the audio branch diluted |
| by noise (AUC 0.751 vs 0.770). Until a text branch trained on real spoken |
| dialogue exists, `audio_eot_v2` is the model to deploy. |
| - **Non-commercial models included.** The text and fusion models derive from |
| DailyDialog (CC BY-NC-SA 4.0). Only the audio branches are permissively licensed. |
|
|
| Code, training scripts, and the full write-up: **https://github.com/Nikhils-G/turnwave** |
|
|
