# How Turn-Taking and Interruption Handling Work in Real-Time Mode for Speech-to-Speech

> Understand real-time speech-to-speech turn-taking and interruption handling. Learn how VAD events, active responses, and session configurations enable seamless interaction.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: internals
- Published: 2026-08-06

---

**Turn-taking in real-time mode uses a Voice Activity Detector (VAD) that emits pipeline events with turn IDs and revision numbers, while interruption handling depends on three conditions: an active response must exist, the VAD event must allow interruption, and the session configuration must have `interrupt_response` enabled.**

The `huggingface/speech-to-speech` repository implements a low-latency conversational system that mimics natural human dialogue. Turn-taking and interruption handling are orchestrated by three cooperating components: the **VADHandler** for speech boundary detection, the **RealtimeService** for state management, and the **AudioHandler** for protocol translation.

---

## The Three Core Components of Real-Time Turn-Taking

| Component | Primary Responsibility | Source File |
|-----------|------------------------|-------------|
| **VADHandler** | Detects speech start/end, allocates turn IDs, manages speculative turn reopening | [`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py) |
| **RealtimeService** | Maintains per-connection speculative turn state, routes events, filters stale revisions | [`src/speech_to_speech/api/openai_realtime/service.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/service.py) |
| **AudioHandler** | Translates VAD events to OpenAI Realtime protocol, executes response cancellation | [`src/speech_to_speech/api/openai_realtime/handlers/audio.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/handlers/audio.py) |

---

## How Speech Detection Triggers Turn-Taking

The VAD operates on 512-sample audio chunks. When voice activity exceeds the `_active_speech_min_ms` threshold, the handler invokes `_ensure_turn_for_speech_start()` in [`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py).

This method performs two critical actions:

1. **Allocates a new turn ID** (format: `turn_<n>`) with revision `0`
2. **Checks for speculative reopening** via `_should_reopen_current_turn()` when `_uses_realtime_turn_handling()` returns `True`

When reopened, the same `turn_id` is retained but `turn_revision` increments. The VAD emits a `SpeechStartedEvent` carrying these fields:

```python

# src/speech_to_speech/VAD/vad_handler.py

self.text_output_queue.put(
    SpeechStartedEvent(
        audio_start_ms=effective_start_ms,
        turn_id=turn_id,
        turn_revision=turn_revision,
        reopened=reopened,
        interrupt_response=False,   # default; True for natural speech starts

    )
)

```

The `interrupt_response` flag defaults to `True` for natural speech detection but can be explicitly set to `False` for synthetic starts that should not cancel ongoing responses.

---

## The Interruption Decision Logic

Interruption handling occurs in `AudioHandler.on_speech_started()` at [`src/speech_to_speech/api/openai_realtime/handlers/audio.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/handlers/audio.py). The code evaluates three conditions before cancelling a response:

```python

# src/speech_to_speech/api/openai_realtime/handlers/audio.py

if st.in_response and event.interrupt_response and st.runtime_config.interrupt_response_enabled:
    events.extend(response.finish_response(
        conn_id, 
        status="cancelled", 
        reason="turn_detected"
    ))

```

All three must be satisfied:

- `st.in_response` — A response is currently streaming to the client
- `event.interrupt_response` — The VAD event permits interruption (set in `SpeechStartedEvent`)
- `st.runtime_config.interrupt_response_enabled` — Session configuration allows barge-in

When triggered, the client receives `response.done` with `status="cancelled"` and `reason="turn_detected"`, and a new input audio item begins for transcription.

---

## Configuring Interruption via Session Settings

The `interrupt_response` behavior is client-configurable through the OpenAI Realtime protocol. In [`src/speech_to_speech/api/openai_realtime/runtime_config.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/runtime_config.py), the `interrupt_response_enabled` property resolves the session setting:

```python

# src/speech_to_speech/api/openai_realtime/runtime_config.py

@property
def interrupt_response_enabled(self) -> bool:
    td = self.session.audio.turn_detection if self.session.audio else None
    if td is None:
        return True
    if hasattr(td, "interrupt_response"):
        return td.interrupt_response
    return td.get("interrupt_response", True)

```

### Enabling Barge-In (Default Behavior)

```python
session_update = {
    "type": "session.update",
    "session": {
        "turn_detection": {
            "type": "server_vad",
            "interrupt_response": True   # User speech cancels assistant output

        }
    }
}
await ws.send(json.dumps(session_update))

```

### Disabling Interruption

```python
session_update = {
    "type": "session.update",
    "session": {
        "turn_detection": {
            "type": "server_vad",
            "interrupt_response": False  # User speech queues without cancelling

        }
    }
}
await ws.send(json.dumps(session_update))

```

---

## Speculative Turns and Turn Reopening

The system implements **speculative turns** to handle rapid user responses before the assistant finishes. The `SpeculativeTurnTracker` in [`src/speech_to_speech/pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py) manages this state.

When `_should_reopen_current_turn()` detects speech within `unanswered_reopen_ms` of the last final audio, `_reopen_current_turn()` increments `turn_revision` while preserving `turn_id`. The `SpeechStartedEvent` carries `reopened=True`.

This design provides two benefits:

- **State continuity** — The same logical turn continues, avoiding fragmented conversation history
- **Stale event suppression** — `RealtimeService._is_stale_turn_event()` drops `AssistantTextEvent` and `TokenUsageEvent` from superseded revisions

---

## End-of-Turn Processing Flow

When silence is detected, the VAD emits a `SpeechStoppedEvent`. The processing chain completes as follows:

1. `AudioHandler.on_speech_stopped()` emits `input_audio_buffer.speech_stopped` to the client
2. Final VAD audio (mode `"final"`) passes to the STT component
3. `TranscriptionCompletedEvent` triggers `RealtimeService._on_transcription_completed()`
4. Transcript is added to chat history and `GenerateResponseRequest` is queued
5. LLM generation streams `AssistantTextEvent` and `TokenUsageEvent` to the client

If a newer turn reopens during this process, stale events are discarded based on revision comparison.

---

## Complete Turn-Taking Flow Diagram

```

Microphone PCM → AudioHandler.append_pcm → VADHandler.process
                    │
                    ├─ Speech detected ──► SpeechStartedEvent
                    │                         │
                    │    ├─ turn_id, turn_revision, interrupt_response
                    │    │
                    │    └─► RealtimeService.dispatch_pipeline_event()
                    │              │
                    │              └─► AudioHandler.on_speech_started()
                    │                    │
                    │                    ├─ Check: in_response?
                    │                    ├─ Check: event.interrupt_response?
                    │                    ├─ Check: interrupt_response_enabled?
                    │                    │
                    │                    └─ If all true: response.finish_response(
                    │                           status="cancelled",
                    │                           reason="turn_detected"
                    │                       )
                    │                    │
                    │                    └─ Emit: input_audio_buffer.speech_started
                    │
                    └─ Silence detected ──► SpeechStoppedEvent
                                              │
                                              └─► AudioHandler.on_speech_stopped()
                                                    │
                                                    └─ Emit: input_audio_buffer.speech_stopped
                                                              │
                                                              └─► STT → TranscriptionCompletedEvent
                                                                        │
                                                                        └─► LLM response generation

```

---

## Key Source Files for Turn-Taking Implementation

| File | Critical Functionality |
|------|------------------------|
| [`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py) | Turn allocation, revision management, speculative reopening logic, `SpeechStartedEvent`/`SpeechStoppedEvent` emission |
| [`src/speech_to_speech/pipeline/events.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/events.py) | Event dataclass definitions including `SpeechStartedEvent.interrupt_response` |
| [`src/speech_to_speech/api/openai_realtime/service.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/service.py) | Per-connection state, event routing, stale event detection via `_is_stale_turn_event()` |
| [`src/speech_to_speech/api/openai_realtime/handlers/audio.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/handlers/audio.py) | Protocol translation, interruption execution in `on_speech_started()` |
| [`src/speech_to_speech/api/openai_realtime/runtime_config.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/runtime_config.py) | Session configuration access, `interrupt_response_enabled` property |
| [`src/speech_to_speech/pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py) | `SpeculativeTurnTracker` for turn lifecycle and reopening decisions |

---

## Summary

- **Turn-taking state machine** uses `turn_id` and `turn_revision` to track conversation state across speculative reopening
- **Interruption requires three conditions**: active response, VAD permission flag, and session configuration enabled
- **Barge-in is configurable** via `turn_detection.interrupt_response` in session updates
- **Speculative turns** prevent conversation fragmentation when users respond quickly
- **Stale revision filtering** ensures only current-turn events reach the client

---

## Frequently Asked Questions

### What controls whether user speech interrupts the assistant?

Three factors control interruption: `st.in_response` (response streaming), `event.interrupt_response` from the VAD, and `st.runtime_config.interrupt_response_enabled` from session configuration. All must be true. The session setting defaults to `True` but can be disabled by sending a `session.update` with `"interrupt_response": false`.

### How does the system handle rapid back-and-forth conversation?

The **speculative turn** mechanism in [`src/speech_to_speech/pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/speculative_turns.py) reopens an existing turn when speech occurs within `unanswered_reopen_ms` of the assistant's last audio. This preserves `turn_id` while incrementing `turn_revision`, allowing natural interruptions without fragmenting the conversation history.

### What happens to assistant responses that get interrupted?

Interrupted responses terminate with `response.finish_response(status="cancelled", reason="turn_detected")`. The client receives a `response.done` event with these fields. Any in-flight tokens from that response are discarded, and the LLM processes the new user input instead.

### Can synthetic speech starts avoid triggering interruption?

Yes. The VAD can emit `SpeechStartedEvent` with `interrupt_response=False` for synthetic starts, such as after extended silence. This prevents unnecessary cancellation of ongoing assistant speech when the system itself initiates listening, as tested in `test_vad_final_synthetic_start_does_not_interrupt_response`.