# How the VAD Pipeline Handles Turn-Taking and Interruption Detection in Speech-to-Speech

> Discover how the VAD pipeline manages turn-taking and interruption detection in speech-to-speech conversations. Learn about VADHandler, SpeculativeTurnTracker, and RuntimeConfig.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: internals
- Published: 2026-08-03

---

**The VAD pipeline manages conversational dynamics through `VADHandler` and `SpeculativeTurnTracker`, which coordinate turn boundaries using speculative reopen logic, while interruption detection is governed by `RuntimeConfig` flags and enforced by downstream audio handlers.**

The `huggingface/speech-to-speech` repository implements a real-time voice activity detection system designed for natural conversational AI. Understanding how the VAD pipeline handles turn-taking and interruption detection reveals how the system achieves low-latency barge-in capabilities while maintaining coherent session state across user utterances.

## Turn-Taking Mechanics and Speculative Reopening

The **VADHandler** class in [`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py) orchestrates turn management through a state machine that distinguishes between new utterances and continuations of existing speech. Rather than treating every silence as a turn boundary, the system employs speculative logic to handle natural pauses within single thoughts.

### Starting and Ensuring Turns

When the **Silero VAD** detects speech onset via `VADIterator`, the pipeline must first guarantee an active turn context exists. The `_ensure_turn_for_speech_start()` method (lines 18-27) implements a resolution hierarchy: it checks for active turns, confirms pending reopens, attempts to reopen recent speculative turns, or finally invokes `_start_new_turn()` to generate a fresh `turn_id` with revision 0.

```python

# Conceptual flow based on vad_handler.py

def _ensure_turn_for_speech_start(self):
    if self.current_turn is None:
        if self._pending_reopen:
            self._confirm_pending_reopen()
        elif self._should_reopen_current_turn():
            self._begin_pending_reopen_if_needed()
        else:
            self._start_new_turn()  # Creates turn_<counter> with revision 0

```

The `_start_new_turn()` method (lines 76-87) instantiates a new turn identifier, clears pending reopen states, and registers the turn with the `SpeculativeTurnTracker`.

### Speculative Reopen Logic

The pipeline treats brief pauses as potential turn continuations rather than definitive boundaries. The `_should_reopen_current_turn()` method (lines 20-48) evaluates reopen eligibility by comparing elapsed audio time against the `speculative_reopen_ms` threshold while verifying the turn is not yet committed.

When a pause qualifies for reopening, `_begin_pending_reopen_if_needed()` (lines 50-66) creates a candidate revision through `SpeculativeTurnTracker.begin_reopen_candidate()`, storing it for confirmation. Once sufficient new audio arrives, `_confirm_pending_reopen()` (lines 80-95) commits the revision via `confirm_reopen_candidate()`, seamlessly continuing the prior turn rather than fragmenting the conversation.

### The SpeculativeTurnTracker

Located in [`src/speech_to_speech/pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py), the **SpeculativeTurnTracker** maintains an ordered mapping of turn revisions and enforces the `unanswered_reopen_ms` grace period. This component ensures turns remain reopenable until the assistant produces a response, preventing race conditions during concurrent utterance processing through its `observe()`, `begin_reopen_candidate()`, and `confirm_reopen_candidate()` methods.

## Interruption Detection and Barge-In Control

Interruption handling (barge-in) operates through a coordination between the VAD pipeline's event emission and downstream handler enforcement, controlled by session-specific runtime flags.

### Runtime Configuration

The **RuntimeConfig** class in [`src/speech_to_speech/api/openai_realtime/runtime_config.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/runtime_config.py) exposes the interruption capability via `interrupt_response_enabled` (lines 58-76). This property maps to the `turn_detection.interrupt_response` flag, which defaults to `True` but can be toggled per session to disable barge-in behavior.

```python

# From runtime_config.py conceptual implementation

@property
def interrupt_response_enabled(self) -> bool:
    return self._config.get("turn_detection", {}).get("interrupt_response", True)

```

### Event-Driven Interruption Flow

When speech starts, the VAD handler emits a **SpeechStartedEvent** carrying an `interrupt_response` boolean. While the current VAD implementation explicitly sets this flag to `False`—delegating real-time interrupt handling to other components—the architecture supports direct VAD-level interruption signaling.

The actual enforcement occurs in [`src/speech_to_speech/api/openai_realtime/handlers/audio.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/handlers/audio.py) (lines 113-115), where the audio handler inspects `event.interrupt_response` alongside the runtime configuration. If both indicate interruption is enabled, the handler aborts the current assistant response. Similarly, the WebSocket router in [`websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_router.py) (lines 744-769) uses `interrupt_response_enabled` to trigger "cancel-and-flush" operations on the assistant output stream.

```python

# Conceptual handling in audio.py

if event.interrupt_response and runtime_config.interrupt_response_enabled:
    await self.abort_current_response()
    await self.flush_playback_buffer()

```

## Summary

- **Turn Management**: The `VADHandler` creates logical turn units via `_start_new_turn()` and manages pauses through speculative reopen logic that evaluates `speculative_reopen_ms` windows before committing boundaries.

- **Speculative Continuation**: The `SpeculativeTurnTracker` maintains revision history and enforces `unanswered_reopen_ms` grace periods, allowing turns to reopen until the assistant responds, preventing fragmentation of user utterances.

- **Interruption Control**: Barge-in capability is governed by `RuntimeConfig.interrupt_response_enabled` and enforced by downstream handlers in [`audio.py`](https://github.com/huggingface/speech-to-speech/blob/main/audio.py) and [`websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_router.py), which cancel assistant responses when new user speech is detected while the flag is active.

- **Separation of Concerns**: The VAD pipeline detects speech and manages turn state, while actual interruption enforcement occurs in audio handling layers that consume `SpeechStartedEvent` signals.

## Frequently Asked Questions

### How does the VAD pipeline distinguish between a new turn and a continuation of the current utterance?

The pipeline uses `_should_reopen_current_turn()` to evaluate whether the elapsed silence falls within the `speculative_reopen_ms` window and whether the turn is already committed. If conditions are met, `SpeculativeTurnTracker` creates a pending revision rather than a fresh turn ID, allowing the system to treat the new speech as a continuation rather than a separate conversational turn.

### What determines whether a user can interrupt the assistant mid-response?

Interruption capability is controlled by the `turn_detection.interrupt_response` flag in `RuntimeConfig`, exposed via `interrupt_response_enabled`. When enabled, downstream handlers in [`audio.py`](https://github.com/huggingface/speech-to-speech/blob/main/audio.py) and the WebSocket router monitor `SpeechStartedEvent` signals and abort ongoing assistant responses to allow the new user utterance to take precedence.

### Where is the speculative turn reopen logic implemented?

The speculative reopen mechanism spans two files: [`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py) contains the decision logic in `_should_reopen_current_turn()`, `_begin_pending_reopen_if_needed()`, and `_confirm_pending_reopen()`, while the state tracking is managed by `SpeculativeTurnTracker` in [`src/speech_to_speech/pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py).

### Why does the VAD handler set `interrupt_response` to False in SpeechStartedEvent?

According to the source implementation in [`vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/vad_handler.py), the VAD handler explicitly sets the `interrupt_response` flag to `False` because real-time interrupt handling is performed elsewhere in the architecture. The audio handler and WebSocket router check the runtime configuration directly to enforce barge-in behavior, separating the concerns of speech detection from response cancellation.