# How to Handle Audio Interruptions and Manage Turn-Taking in the Speech-to-Speech Pipeline

> Learn to handle audio interruptions and manage turn-taking in the speech-to-speech pipeline. Discover the three-component architecture for seamless voice interaction.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-05

---

**The huggingface/speech-to-speech pipeline handles interruptions and turn-taking through a three-component architecture: `VADHandler` detects voice activity and creates turn identifiers, `SpeculativeTurnTracker` manages thread-safe turn revisions, and `SmartTurnAnalyzer` predicts whether the user has finished speaking using an ONNX model.**

The **speech-to-speech** repository implements production-grade turn management that prevents "ghost" responses when users interrupt themselves or pause mid-thought. Unlike simple VAD systems that treat every silence as a turn boundary, this pipeline uses speculative turn tracking and smart turn prediction to distinguish between meaningful pauses and genuine end-of-turn silence.

## Core Turn-Taking Architecture

The interruption handling system rests on three specialized components that work together in the `VADHandler` process loop.

### VADHandler: Turn Creation and Reopen Logic

Located in [`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py), the `VADHandler` class is the sole module responsible for turn identity. It uses Silero VAD for voice detection and implements three critical methods:

- **`_should_reopen_current_turn`** — checks if elapsed silence falls within `speculative_reopen_ms` or the extended `unanswered_reopen_ms` window
- **`_begin_pending_reopen_if_needed`** — creates a speculative reopen candidate when speech resumes inside the grace window
- **`_ensure_turn_for_speech_start`** — guarantees a turn exists before emitting `SpeechStartedEvent`

When a user pauses, the handler calculates whether the silence permits reopening. If speech resumes within the threshold, the same turn continues with a new revision rather than spawning a fresh turn.

### SpeculativeTurnTracker: Thread-Safe Revision Management

The `SpeculativeTurnTracker` in [`src/speech_to_speech/pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py) provides atomic operations for turn state:

| Method | Purpose |
|--------|---------|
| `begin_reopen_candidate` | Registers revision N+1 for the current turn |
| `confirm_reopen_candidate` | Promotes candidate to active revision, discarding stale LLM output |
| `start_reopen_grace` | Initiates the grace period during which reopening is permitted without committing output |
| `commit` | Finalizes the turn, allowing downstream processing to complete |

This tracker ensures that when a user interrupts their own utterance, any partially generated LLM response associated with the previous revision is discarded rather than played to the user.

### SmartTurnAnalyzer: Neural Turn Completion Prediction

The `SmartTurnAnalyzer` in [`src/speech_to_speech/VAD/smart_turn.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/smart_turn.py) runs an ONNX model to classify utterances as **complete** or **incomplete**. The `_smart_turn_timing_ms` method in `VADHandler` calls `SmartTurnAnalyzer.predict`, which:

1. Resamples audio to the model's expected input rate
2. Runs inference via ONNX Runtime
3. Returns a classification with confidence score

For incomplete predictions, the handler extends the grace period to `smart_turn_max_wait_ms` (default 2000 ms) and optionally applies `smart_turn_incomplete_delay_ms` before processing.

## The Interruption Handling Flow

Here is the precise sequence when a user interrupts themselves:

1. **Speech starts** — `VADHandler` creates `turn_1` with revision 0
2. **User pauses** — VAD detects silence; `_should_reopen_current_turn` evaluates the reopen window
3. **Pending reopen created** — `_begin_pending_reopen_if_needed` registers revision 1 as a candidate in `SpeculativeTurnTracker`
4. **User resumes within grace** — `_confirm_pending_reopen` promotes the candidate; same turn ID, incremented revision; prior LLM output discarded
5. **User actually stops** — `SmartTurnAnalyzer.predict` returns **complete**; normal `speculative_reopen_ms` applies; grace expires; turn commits

If `SmartTurnAnalyzer` instead returns **incomplete**, step 5 uses the extended grace, keeping the turn uncommitted and reopenable.

## Configuring Turn-Taking Behavior

All timing parameters are defined in [`src/speech_to_speech/arguments_classes/vad_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/vad_arguments.py) and passed through `s2s_pipeline`.

### Key CLI Arguments

| Argument | Description | Default |
|----------|-------------|---------|
| `--smart_turn` | Enable Smart Turn inference (requires `onnxruntime`) | enabled |
| `--no_smart_turn` | Disable neural turn prediction | — |
| `--smart_turn_max_wait_ms` | Maximum grace after incomplete prediction | 2000 |
| `--smart_turn_incomplete_delay_ms` | Processing delay while waiting for completion | 600 |
| `--speculative_reopen_ms` | Grace when Smart Turn is disabled | 800 |
| `--unanswered_reopen_ms` | Upper bound when no assistant output committed | 7000 |
| `--min_speech_ms` | Minimum active speech to start a turn | 384 |
| `--min_speech_continuation_ms` | Minimum speech to continue reopenable turn | 192 |

### Command-Line Example

Enable Smart Turn with custom timing in realtime mode:

```bash
python -m speech_to_speech.main \
  --mode realtime \
  --smart_turn \
  --smart_turn_model_path /models/smart-turn.onnx \
  --smart_turn_threshold 0.6 \
  --smart_turn_max_wait_ms 2500 \
  --smart_turn_incomplete_delay_ms 800

```

## Programmatic Pipeline Construction

For custom integrations, instantiate the components directly:

```python
from speech_to_speech.VAD.vad_handler import VADHandler
from speech_to_speech.VAD.smart_turn import SmartTurnAnalyzer
from speech_to_speech.pipeline.speculative_turns import SpeculativeTurnTracker
from threading import Event
from queue import Queue

should_listen = Event()
audio_q = Queue()
output_q = Queue()
spec_turns = SpeculativeTurnTracker()

vad = VADHandler(
    should_listen,
    audio_q,
    output_q,
    setup_args=(should_listen,),
    setup_kwargs={
        "sample_rate": 16000,
        "smart_turn": True,
        "smart_turn_model_path": "/models/smart-turn.onnx",
        "smart_turn_threshold": 0.6,
        "smart_turn_max_wait_ms": 2500,
        "smart_turn_incomplete_delay_ms": 800,
        "speculative_turns": spec_turns,
        "text_output_queue": None,
    },
)

# Process raw PCM chunks

vad.process(b'\x00\x01\x02...')

```

The `VADHandler.process` method runs Silero VAD on each chunk. When a segment finalizes, `_smart_turn_timing_ms` determines the grace period. If incomplete, the extended timing prevents premature commitment.

## Pipeline Data Flow

Downstream handlers receive `VADAudio` events carrying `turn_id` and `turn_revision` metadata. The `LMOutputProcessor` and TTS stages use these fields to discard output from superseded revisions, ensuring only the final confirmed revision reaches the user.

Key integration files:

- [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) — orchestrates handlers and injects `SpeculativeTurnTracker`
- [`src/speech_to_speech/api/openai_realtime/runtime_config.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/runtime_config.py) — per-session runtime configuration
- [`src/speech_to_speech/LLM/audio_input_notifier.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/audio_input_notifier.py) — emits explicit turn signals for LLMs requiring them

## Summary

- **VADHandler** in [`vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/vad_handler.py) is the single source of truth for turn identity and reopen decisions
- **SpeculativeTurnTracker** in [`speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/speculative_turns.py) provides atomic revision management with grace period control
- **SmartTurnAnalyzer** in [`smart_turn.py`](https://github.com/huggingface/speech-to-speech/blob/main/smart_turn.py) uses ONNX inference to distinguish incomplete from complete utterances
- Reopen candidates prevent ghost responses by discarding LLM output from superseded revisions
- All timing thresholds are configurable via CLI arguments in [`vad_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/vad_arguments.py)

## Frequently Asked Questions

### What triggers a turn reopen versus a new turn creation?

A turn reopens when speech resumes during the grace period after a silence. The `VADHandler._should_reopen_current_turn` method checks if elapsed time is within `speculative_reopen_ms` (or `unanswered_reopen_ms` if no assistant response exists). If outside this window, fresh speech creates a new turn with a new ID.

### How does the pipeline prevent playing outdated LLM responses?

`SpeculativeTurnTracker` assigns monotonic revisions per turn. When a reopen is confirmed, the revision increments. Downstream processors check the revision attached to each `VADAudio` event and discard any LLM output generated for prior revisions before it reaches TTS.

### Can Smart Turn be disabled for faster inference?

Yes. Pass `--no_smart_turn` to use fixed `speculative_reopen_ms` timing instead of neural prediction. This removes the ONNX Runtime dependency but provides less accurate turn boundaries for conversational speech patterns.

### What is the difference between `speculative_reopen_ms` and `smart_turn_max_wait_ms`?

`speculative_reopen_ms` (default 800 ms) applies when Smart Turn predicts **complete** or is disabled. `smart_turn_max_wait_ms` (default 2000 ms) extends the grace when the model predicts **incomplete**, giving users more time to finish complex utterances without triggering a false turn end.