How Turn-Taking and Interruption Handling Work in Real-Time Mode for Speech-to-Speech

Turn-taking in real-time mode uses a Voice Activity Detector (VAD) that emits pipeline events with turn IDs and revision numbers, while interruption handling depends on three conditions: an active response must exist, the VAD event must allow interruption, and the session configuration must have interrupt_response enabled.

The huggingface/speech-to-speech repository implements a low-latency conversational system that mimics natural human dialogue. Turn-taking and interruption handling are orchestrated by three cooperating components: the VADHandler for speech boundary detection, the RealtimeService for state management, and the AudioHandler for protocol translation.

---

The Three Core Components of Real-Time Turn-Taking

Component	Primary Responsibility	Source File
VADHandler	Detects speech start/end, allocates turn IDs, manages speculative turn reopening	src/speech_to_speech/VAD/vad_handler.py
RealtimeService	Maintains per-connection speculative turn state, routes events, filters stale revisions	src/speech_to_speech/api/openai_realtime/service.py
AudioHandler	Translates VAD events to OpenAI Realtime protocol, executes response cancellation	src/speech_to_speech/api/openai_realtime/handlers/audio.py

---

How Speech Detection Triggers Turn-Taking

The VAD operates on 512-sample audio chunks. When voice activity exceeds the _active_speech_min_ms threshold, the handler invokes _ensure_turn_for_speech_start() in src/speech_to_speech/VAD/vad_handler.py.

This method performs two critical actions:

1. Allocates a new turn ID (format: turn_<n>) with revision 0
2. Checks for speculative reopening via _should_reopen_current_turn() when _uses_realtime_turn_handling() returns True

When reopened, the same turn_id is retained but turn_revision increments. The VAD emits a SpeechStartedEvent carrying these fields:

# src/speech_to_speech/VAD/vad_handler.py

self.text_output_queue.put(
    SpeechStartedEvent(
        audio_start_ms=effective_start_ms,
        turn_id=turn_id,
        turn_revision=turn_revision,
        reopened=reopened,
        interrupt_response=False,   # default; True for natural speech starts

    )
)

The interrupt_response flag defaults to True for natural speech detection but can be explicitly set to False for synthetic starts that should not cancel ongoing responses.

---

The Interruption Decision Logic

Interruption handling occurs in AudioHandler.on_speech_started() at src/speech_to_speech/api/openai_realtime/handlers/audio.py. The code evaluates three conditions before cancelling a response:

# src/speech_to_speech/api/openai_realtime/handlers/audio.py

if st.in_response and event.interrupt_response and st.runtime_config.interrupt_response_enabled:
    events.extend(response.finish_response(
        conn_id, 
        status="cancelled", 
        reason="turn_detected"
    ))

All three must be satisfied:

• st.in_response — A response is currently streaming to the client
• event.interrupt_response — The VAD event permits interruption (set in SpeechStartedEvent)
• st.runtime_config.interrupt_response_enabled — Session configuration allows barge-in

When triggered, the client receives response.done with status="cancelled" and reason="turn_detected", and a new input audio item begins for transcription.

---

Configuring Interruption via Session Settings

The interrupt_response behavior is client-configurable through the OpenAI Realtime protocol. In src/speech_to_speech/api/openai_realtime/runtime_config.py, the interrupt_response_enabled property resolves the session setting:

# src/speech_to_speech/api/openai_realtime/runtime_config.py

@property
def interrupt_response_enabled(self) -> bool:
    td = self.session.audio.turn_detection if self.session.audio else None
    if td is None:
        return True
    if hasattr(td, "interrupt_response"):
        return td.interrupt_response
    return td.get("interrupt_response", True)

Enabling Barge-In (Default Behavior)

session_update = {
    "type": "session.update",
    "session": {
        "turn_detection": {
            "type": "server_vad",
            "interrupt_response": True   # User speech cancels assistant output

        }
    }
}
await ws.send(json.dumps(session_update))

Disabling Interruption

session_update = {
    "type": "session.update",
    "session": {
        "turn_detection": {
            "type": "server_vad",
            "interrupt_response": False  # User speech queues without cancelling

        }
    }
}
await ws.send(json.dumps(session_update))

---

Speculative Turns and Turn Reopening

The system implements speculative turns to handle rapid user responses before the assistant finishes. The SpeculativeTurnTracker in src/speech_to_speech/pipeline/speculative_turns.py manages this state.

When _should_reopen_current_turn() detects speech within unanswered_reopen_ms of the last final audio, _reopen_current_turn() increments turn_revision while preserving turn_id. The SpeechStartedEvent carries reopened=True.

This design provides two benefits:

• State continuity — The same logical turn continues, avoiding fragmented conversation history
• Stale event suppression — RealtimeService._is_stale_turn_event() drops AssistantTextEvent and TokenUsageEvent from superseded revisions

---

End-of-Turn Processing Flow

When silence is detected, the VAD emits a SpeechStoppedEvent. The processing chain completes as follows:

1. AudioHandler.on_speech_stopped() emits input_audio_buffer.speech_stopped to the client
2. Final VAD audio (mode "final") passes to the STT component
3. TranscriptionCompletedEvent triggers RealtimeService._on_transcription_completed()
4. Transcript is added to chat history and GenerateResponseRequest is queued
5. LLM generation streams AssistantTextEvent and TokenUsageEvent to the client

If a newer turn reopens during this process, stale events are discarded based on revision comparison.

---

Complete Turn-Taking Flow Diagram

Microphone PCM → AudioHandler.append_pcm → VADHandler.process
                    │
                    ├─ Speech detected ──► SpeechStartedEvent
                    │                         │
                    │    ├─ turn_id, turn_revision, interrupt_response
                    │    │
                    │    └─► RealtimeService.dispatch_pipeline_event()
                    │              │
                    │              └─► AudioHandler.on_speech_started()
                    │                    │
                    │                    ├─ Check: in_response?
                    │                    ├─ Check: event.interrupt_response?
                    │                    ├─ Check: interrupt_response_enabled?
                    │                    │
                    │                    └─ If all true: response.finish_response(
                    │                           status="cancelled",
                    │                           reason="turn_detected"
                    │                       )
                    │                    │
                    │                    └─ Emit: input_audio_buffer.speech_started
                    │
                    └─ Silence detected ──► SpeechStoppedEvent
                                              │
                                              └─► AudioHandler.on_speech_stopped()
                                                    │
                                                    └─ Emit: input_audio_buffer.speech_stopped
                                                              │
                                                              └─► STT → TranscriptionCompletedEvent
                                                                        │
                                                                        └─► LLM response generation

---

Key Source Files for Turn-Taking Implementation

File	Critical Functionality
src/speech_to_speech/VAD/vad_handler.py	Turn allocation, revision management, speculative reopening logic, SpeechStartedEvent/SpeechStoppedEvent emission
src/speech_to_speech/pipeline/events.py	Event dataclass definitions including SpeechStartedEvent.interrupt_response
src/speech_to_speech/api/openai_realtime/service.py	Per-connection state, event routing, stale event detection via _is_stale_turn_event()
src/speech_to_speech/api/openai_realtime/handlers/audio.py	Protocol translation, interruption execution in on_speech_started()
src/speech_to_speech/api/openai_realtime/runtime_config.py	Session configuration access, interrupt_response_enabled property
src/speech_to_speech/pipeline/speculative_turns.py	SpeculativeTurnTracker for turn lifecycle and reopening decisions

---

Summary

• Turn-taking state machine uses turn_id and turn_revision to track conversation state across speculative reopening
• Interruption requires three conditions: active response, VAD permission flag, and session configuration enabled
• Barge-in is configurable via turn_detection.interrupt_response in session updates
• Speculative turns prevent conversation fragmentation when users respond quickly
• Stale revision filtering ensures only current-turn events reach the client

---

Frequently Asked Questions

What controls whether user speech interrupts the assistant?

Three factors control interruption: st.in_response (response streaming), event.interrupt_response from the VAD, and st.runtime_config.interrupt_response_enabled from session configuration. All must be true. The session setting defaults to True but can be disabled by sending a session.update with "interrupt_response": false.

How does the system handle rapid back-and-forth conversation?

The speculative turn mechanism in src/speech_to_speech/pipeline/speculative_turns.py reopens an existing turn when speech occurs within unanswered_reopen_ms of the assistant's last audio. This preserves turn_id while incrementing turn_revision, allowing natural interruptions without fragmenting the conversation history.

What happens to assistant responses that get interrupted?

Interrupted responses terminate with response.finish_response(status="cancelled", reason="turn_detected"). The client receives a response.done event with these fields. Any in-flight tokens from that response are discarded, and the LLM processes the new user input instead.

Can synthetic speech starts avoid triggering interruption?

Yes. The VAD can emit SpeechStartedEvent with interrupt_response=False for synthetic starts, such as after extended silence. This prevents unnecessary cancellation of ongoing assistant speech when the system itself initiates listening, as tested in test_vad_final_synthetic_start_does_not_interrupt_response.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}