# Response Cancellation and Barge-In in Hugging Face Speech-to-Speech: A Complete Guide

> Learn how Hugging Face speech-to-speech handles user interruptions with CancelScope for response cancellation and barge-in. Detects SpeechStartedEvent to abort generation.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-03

---

**The Hugging Face speech-to-speech library handles user interruptions through a generation-aware `CancelScope` that increments a counter when a `SpeechStartedEvent` is detected, flushing stale audio and text queues while allowing the pipeline to abort in-flight LLM and TTS generation.**

Response cancellation and barge-in when user interrupts is a critical feature for real-time voice agents, enabling natural conversation flow by stopping the assistant when the user starts speaking. In the `huggingface/speech-to-speech` repository, this behavior is implemented through a sophisticated pipeline architecture that tracks generations and discards stale work. The system uses a combination of session configuration flags, event-driven cancellation scopes, and queue management to ensure low-latency interruption handling.

## Understanding the Four-Stage Pipeline Architecture

The library operates a **four-stage pipeline** where each stage runs in its own thread and communicates through thread-safe queues:

- **VAD (Voice Activity Detection)**: Uses Silero VAD v5 in [`vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/vad_handler.py) to detect speech and emit `SpeechStartedEvent`
- **STT (Speech-to-Text)**: Transcribes audio via [`stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/stt_handler.py) and pushes text to `text_prompt_queue`
- **LLM**: Generates responses in [`lm_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/lm_handler.py), consuming from `text_prompt_queue` and writing to `text_output_queue`
- **TTS (Text-to-Speech)**: Synthesizes audio in [`tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/tts_handler.py), reading from `text_output_queue` and writing to `output_queue`

The Realtime API (WebSocket or WebRTC) feeds raw audio into this pipeline. All interruption logic centers on two components: the **`CancelScope`** primitive in [`src/speech_to_speech/pipeline/cancel_scope.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/cancel_scope.py) and the **`RuntimeConfig`** in [`src/speech_to_speech/api/openai_realtime/runtime_config.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/runtime_config.py).

## Detecting User Interruptions with SpeechStartedEvent

When the VAD detects incoming speech, it emits a `SpeechStartedEvent` defined in [`src/speech_to_speech/pipeline/events.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/events.py):

```python
class SpeechStartedEvent(PipelineEvent):
    type: Literal["speech_started"] = "speech_started"
    interrupt_response: bool = Field(default=True, exclude=True)

```

The `interrupt_response` field defaults to **True**, matching OpenAI's default behavior for server-side VAD. This event travels through the `text_output_queue` to the WebSocket router, which evaluates whether to trigger cancellation based on the current session's runtime configuration.

## The CancelScope Mechanism for Generation Tracking

At the heart of response cancellation lies the `CancelScope` class in [`src/speech_to_speech/pipeline/cancel_scope.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/cancel_scope.py). This primitive maintains a **generation counter** that acts as a version identifier for each assistant response cycle:

```python
from speech_to_speech.pipeline.cancel_scope import CancelScope

cs = CancelScope()
print(cs.generation)        # → 0

cs.cancel()                 # Increments to 1, sets discarding=True

print(cs.is_stale(0))       # → True (generation 0 is now stale)

```

When `cancel()` is invoked, it increments the generation counter and sets a `discarding` flag. Pipeline handlers check `cancel_scope.is_stale(generation)` to determine if their current work belongs to an abandoned response. The `response_done()` method clears the discarding flag once the pipeline resets for the next turn.

## Runtime Configuration for Barge-In Control

Whether barge-in is permitted depends on `RuntimeConfig.interrupt_response_enabled` in [`src/speech_to_speech/api/openai_realtime/runtime_config.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/runtime_config.py). This property reads the client-supplied session configuration:

```python

# Conceptual usage inside the router

active_cfg = unit.service._state(session_id).runtime_config
interrupt_enabled = (
    text_msg.interrupt_response and 
    (active_cfg is None or active_cfg.interrupt_response_enabled)
)

```

By default, `interrupt_response_enabled` returns `True`, allowing interruptions. When set to `False` via a client session update, the router ignores `SpeechStartedEvent` signals and continues playing the current response.

## The Cancellation Flow in the WebSocket Router

The `_send_loop_for` function in [`src/speech_to_speech/api/openai_realtime/websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/websocket_router.py) (lines 61-78) implements the actual cancellation logic. When `interrupt_enabled` is true and a response is active, the router executes:

```python
unit.cancel_scope.cancel()                     # New generation, discarding=True

unit.service._state(session_id).response_pending = False
_flush_queue(unit.output_queue, preserve=_keep_audio_sentinel)
_flush_queue(unit.text_output_queue, preserve=_keep_user_text_event)
if unit.response_playing.is_set():
    unit.response_playing.clear()

```

This sequence performs four critical actions:
1. **Increments the generation** via `cancel_scope.cancel()`, marking all previous work as stale
2. **Flushes the output queues**, removing queued audio and text while preserving sentinel events like `AUDIO_RESPONSE_DONE` and `SESSION_END` from [`src/speech_to_speech/pipeline/control.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/control.py)
3. **Clears the response flag** to indicate no response is currently playing
4. **Signals handlers** to abort via the stale generation check

## Generation-Aware Event Discarding

Both `AssistantTextEvent` and `AudioOutput` events carry a `cancel_generation` identifier. The helper `_generation_is_discardable` in [`websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_router.py) (lines 21-36) filters these:

```python
if generation is not None and unit.cancel_scope.is_stale(generation):
    return True
if unit.cancel_scope.discarding and generation != unit.cancel_scope.generation:
    return True

```

If the event's generation is older than the current scope generation, or if the scope is in a discarding state and the event doesn't match the current generation, the event is dropped. This prevents "bleed-through" of audio from cancelled responses.

## Controlling Barge-In Behavior from the Client

Clients disable response cancellation and barge-in by sending a `session.update` event:

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8765/v1",
    websocket_base_url="ws://localhost:8765/v1",
    api_key="not-needed",
)

with client.realtime.connect(model="local") as conn:
    conn.send({
        "type": "session.update",
        "session": {
            "type": "realtime",
            "audio": {
                "input": {
                    "turn_detection": {
                        "type": "server_vad",
                        "interrupt_response": False  # Disable barge-in

                    }
                }
            }
        }
    })

```

Setting `interrupt_response` to `false` updates the `RuntimeConfig`, causing the router to skip the cancellation block even when `SpeechStartedEvent` occurs.

## Post-Cancellation Cleanup and Pipeline Recovery

After cancellation, the pipeline must reset cleanly. When the TTS handler emits the `AUDIO_RESPONSE_DONE` sentinel, the send-loop calls `_drain_pending_response_events` to flush remaining assistant text, sends a `response.done` event to the client, and invokes `unit.cancel_scope.response_done(generation)`. This final call clears the `discarding` flag in [`cancel_scope.py`](https://github.com/huggingface/speech-to-speech/blob/main/cancel_scope.py), allowing the next user turn to proceed with a fresh generation counter.

## Summary

- **CancelScope** in [`src/speech_to_speech/pipeline/cancel_scope.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/cancel_scope.py) provides generation-aware cancellation through a monotonic counter and `is_stale()` checks.
- **SpeechStartedEvent** in [`src/speech_to_speech/pipeline/events.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/events.py) carries the `interrupt_response` flag that triggers the cancellation flow.
- **RuntimeConfig** in [`src/speech_to_speech/api/openai_realtime/runtime_config.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/runtime_config.py) exposes `interrupt_response_enabled` to control whether barge-in is permitted per session.
- **WebSocket Router** in [`src/speech_to_speech/api/openai_realtime/websocket_router.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/websocket_router.py) coordinates cancellation by flushing queues and clearing flags when interruptions are detected.
- **Queue flushing** preserves sentinel events while removing stale audio and text, ensuring the pipeline remains in a consistent state.
- **Client configuration** via `session.update` allows dynamic enabling or disabling of barge-in without restarting the connection.

## Frequently Asked Questions

### How does the speech-to-speech pipeline know when to cancel a response?

The pipeline monitors `SpeechStartedEvent` emitted by the VAD handler in [`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py). When this event occurs, the WebSocket router checks `RuntimeConfig.interrupt_response_enabled` and whether a response is currently active. If both conditions are met, the router calls `CancelScope.cancel()` to increment the generation counter and flushes the output queues, causing downstream handlers to discard stale work when they detect the generation mismatch.

### Can I disable barge-in after the connection is established?

Yes. Send a `session.update` event with `turn_detection.interrupt_response` set to `false`. The `RuntimeConfig` class reads this value dynamically, and the router will subsequently ignore interruption signals. This takes effect immediately for the active session without requiring a reconnection.

### What happens to audio that was already queued when the user interrupts?

The router calls `_flush_queue` on both `output_queue` (audio) and `text_output_queue` (text), removing all pending items except sentinel markers like `AUDIO_RESPONSE_DONE`. Any in-flight audio generation in the TTS handler is abandoned when the handler checks `unit.cancel_scope.is_stale(generation)` and finds that its current generation identifier is older than the scope's current counter.

### Is the cancellation logic thread-safe across all four pipeline stages?

Yes. The `CancelScope` uses thread-safe primitives to manage its generation counter and discarding flag. Each pipeline unit maintains its own `CancelScope` instance, and handlers check the scope state before processing events. The queue flushing operations in the WebSocket router ensure that once cancellation is triggered, no stale events from previous generations can reach the client.