Response Cancellation and Barge-In in Hugging Face Speech-to-Speech: A Complete Guide

The Hugging Face speech-to-speech library handles user interruptions through a generation-aware CancelScope that increments a counter when a SpeechStartedEvent is detected, flushing stale audio and text queues while allowing the pipeline to abort in-flight LLM and TTS generation.

Response cancellation and barge-in when user interrupts is a critical feature for real-time voice agents, enabling natural conversation flow by stopping the assistant when the user starts speaking. In the huggingface/speech-to-speech repository, this behavior is implemented through a sophisticated pipeline architecture that tracks generations and discards stale work. The system uses a combination of session configuration flags, event-driven cancellation scopes, and queue management to ensure low-latency interruption handling.

Understanding the Four-Stage Pipeline Architecture

The library operates a four-stage pipeline where each stage runs in its own thread and communicates through thread-safe queues:

• VAD (Voice Activity Detection): Uses Silero VAD v5 in vad_handler.py to detect speech and emit SpeechStartedEvent
• STT (Speech-to-Text): Transcribes audio via stt_handler.py and pushes text to text_prompt_queue
• LLM: Generates responses in lm_handler.py, consuming from text_prompt_queue and writing to text_output_queue
• TTS (Text-to-Speech): Synthesizes audio in tts_handler.py, reading from text_output_queue and writing to output_queue

The Realtime API (WebSocket or WebRTC) feeds raw audio into this pipeline. All interruption logic centers on two components: the CancelScope primitive in src/speech_to_speech/pipeline/cancel_scope.py and the RuntimeConfig in src/speech_to_speech/api/openai_realtime/runtime_config.py.

Detecting User Interruptions with SpeechStartedEvent

When the VAD detects incoming speech, it emits a SpeechStartedEvent defined in src/speech_to_speech/pipeline/events.py:

class SpeechStartedEvent(PipelineEvent):
    type: Literal["speech_started"] = "speech_started"
    interrupt_response: bool = Field(default=True, exclude=True)

The interrupt_response field defaults to True, matching OpenAI's default behavior for server-side VAD. This event travels through the text_output_queue to the WebSocket router, which evaluates whether to trigger cancellation based on the current session's runtime configuration.

The CancelScope Mechanism for Generation Tracking

At the heart of response cancellation lies the CancelScope class in src/speech_to_speech/pipeline/cancel_scope.py. This primitive maintains a generation counter that acts as a version identifier for each assistant response cycle:

from speech_to_speech.pipeline.cancel_scope import CancelScope

cs = CancelScope()
print(cs.generation)        # → 0

cs.cancel()                 # Increments to 1, sets discarding=True

print(cs.is_stale(0))       # → True (generation 0 is now stale)

When cancel() is invoked, it increments the generation counter and sets a discarding flag. Pipeline handlers check cancel_scope.is_stale(generation) to determine if their current work belongs to an abandoned response. The response_done() method clears the discarding flag once the pipeline resets for the next turn.

Runtime Configuration for Barge-In Control

Whether barge-in is permitted depends on RuntimeConfig.interrupt_response_enabled in src/speech_to_speech/api/openai_realtime/runtime_config.py. This property reads the client-supplied session configuration:

# Conceptual usage inside the router

active_cfg = unit.service._state(session_id).runtime_config
interrupt_enabled = (
    text_msg.interrupt_response and 
    (active_cfg is None or active_cfg.interrupt_response_enabled)
)

By default, interrupt_response_enabled returns True, allowing interruptions. When set to False via a client session update, the router ignores SpeechStartedEvent signals and continues playing the current response.

The Cancellation Flow in the WebSocket Router

The _send_loop_for function in src/speech_to_speech/api/openai_realtime/websocket_router.py (lines 61-78) implements the actual cancellation logic. When interrupt_enabled is true and a response is active, the router executes:

unit.cancel_scope.cancel()                     # New generation, discarding=True

unit.service._state(session_id).response_pending = False
_flush_queue(unit.output_queue, preserve=_keep_audio_sentinel)
_flush_queue(unit.text_output_queue, preserve=_keep_user_text_event)
if unit.response_playing.is_set():
    unit.response_playing.clear()

This sequence performs four critical actions:

1. Increments the generation via cancel_scope.cancel(), marking all previous work as stale
2. Flushes the output queues, removing queued audio and text while preserving sentinel events like AUDIO_RESPONSE_DONE and SESSION_END from src/speech_to_speech/pipeline/control.py
3. Clears the response flag to indicate no response is currently playing
4. Signals handlers to abort via the stale generation check

Generation-Aware Event Discarding

Both AssistantTextEvent and AudioOutput events carry a cancel_generation identifier. The helper _generation_is_discardable in websocket_router.py (lines 21-36) filters these:

if generation is not None and unit.cancel_scope.is_stale(generation):
    return True
if unit.cancel_scope.discarding and generation != unit.cancel_scope.generation:
    return True

If the event's generation is older than the current scope generation, or if the scope is in a discarding state and the event doesn't match the current generation, the event is dropped. This prevents "bleed-through" of audio from cancelled responses.

Controlling Barge-In Behavior from the Client

Clients disable response cancellation and barge-in by sending a session.update event:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8765/v1",
    websocket_base_url="ws://localhost:8765/v1",
    api_key="not-needed",
)

with client.realtime.connect(model="local") as conn:
    conn.send({
        "type": "session.update",
        "session": {
            "type": "realtime",
            "audio": {
                "input": {
                    "turn_detection": {
                        "type": "server_vad",
                        "interrupt_response": False  # Disable barge-in

                    }
                }
            }
        }
    })

Setting interrupt_response to false updates the RuntimeConfig, causing the router to skip the cancellation block even when SpeechStartedEvent occurs.

Post-Cancellation Cleanup and Pipeline Recovery

After cancellation, the pipeline must reset cleanly. When the TTS handler emits the AUDIO_RESPONSE_DONE sentinel, the send-loop calls _drain_pending_response_events to flush remaining assistant text, sends a response.done event to the client, and invokes unit.cancel_scope.response_done(generation). This final call clears the discarding flag in cancel_scope.py, allowing the next user turn to proceed with a fresh generation counter.

Summary

• CancelScope in src/speech_to_speech/pipeline/cancel_scope.py provides generation-aware cancellation through a monotonic counter and is_stale() checks.
• SpeechStartedEvent in src/speech_to_speech/pipeline/events.py carries the interrupt_response flag that triggers the cancellation flow.
• RuntimeConfig in src/speech_to_speech/api/openai_realtime/runtime_config.py exposes interrupt_response_enabled to control whether barge-in is permitted per session.
• WebSocket Router in src/speech_to_speech/api/openai_realtime/websocket_router.py coordinates cancellation by flushing queues and clearing flags when interruptions are detected.
• Queue flushing preserves sentinel events while removing stale audio and text, ensuring the pipeline remains in a consistent state.
• Client configuration via session.update allows dynamic enabling or disabling of barge-in without restarting the connection.

Frequently Asked Questions

How does the speech-to-speech pipeline know when to cancel a response?

The pipeline monitors SpeechStartedEvent emitted by the VAD handler in src/speech_to_speech/VAD/vad_handler.py. When this event occurs, the WebSocket router checks RuntimeConfig.interrupt_response_enabled and whether a response is currently active. If both conditions are met, the router calls CancelScope.cancel() to increment the generation counter and flushes the output queues, causing downstream handlers to discard stale work when they detect the generation mismatch.

Can I disable barge-in after the connection is established?

Yes. Send a session.update event with turn_detection.interrupt_response set to false. The RuntimeConfig class reads this value dynamically, and the router will subsequently ignore interruption signals. This takes effect immediately for the active session without requiring a reconnection.

What happens to audio that was already queued when the user interrupts?

The router calls _flush_queue on both output_queue (audio) and text_output_queue (text), removing all pending items except sentinel markers like AUDIO_RESPONSE_DONE. Any in-flight audio generation in the TTS handler is abandoned when the handler checks unit.cancel_scope.is_stale(generation) and finds that its current generation identifier is older than the scope's current counter.

Is the cancellation logic thread-safe across all four pipeline stages?

Yes. The CancelScope uses thread-safe primitives to manage its generation counter and discarding flag. Each pipeline unit maintains its own CancelScope instance, and handlers check the scope state before processing events. The queue flushing operations in the WebSocket router ensure that once cancellation is triggered, no stale events from previous generations can reach the client.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}