How to Implement Barge-In to Interrupt Assistant Responses in the Hugging Face Speech-to-Speech Repository

Barge-in is implemented through three coordinated mechanisms: a runtime configuration flag (RuntimeConfig.interrupt_response_enabled), VAD-triggered output queue flushing, and PipelineControlMessage signals that guarantee clean turn transitions.

The Hugging Face speech-to-speech repository provides real-time conversational AI with natural barge-in support—the ability for users to interrupt the assistant mid-response. This implementation aligns with the OpenAI Realtime API specification while providing flexible runtime control over interruption behavior.

---

Runtime Configuration: The interrupt_response_enabled Flag

The central control for barge-in resides in src/speech_to_speech/api/openai_realtime/runtime_config.py. The RuntimeConfig class exposes interrupt_response_enabled as a property that reads from the session's turn_detection configuration.

# src/speech_to_speech/api/openai_realtime/runtime_config.py

@property
def interrupt_response_enabled(self) -> bool:
    """Whether barge‑in should cancel an active response."""
    …
    return val if val is not None else True

This property defaults to True, matching the OpenAI Realtime API default. The flag is populated from the interrupt_response field in the session creation or update request.

Client-Side Configuration

Enable barge-in when creating a Realtime session:

{
  "audio": {
    "input": {
      "turn_detection": {
        "type": "server_vad",
        "interrupt_response": true
      }
    }
  }
}

Or configure programmatically in Python:

from speech_to_speech.api.openai_realtime.runtime_config import RuntimeConfig
from openai.types.realtime.realtime_audio_config_input import RealtimeAudioConfigInput
from openai.types.realtime.realtime_audio_config import RealtimeAudioConfig
from openai.types.realtime.realtime_session_create_request import RealtimeSessionCreateRequest

session = RealtimeSessionCreateRequest(
    type="realtime",
    audio=RealtimeAudioConfig(
        input=RealtimeAudioConfigInput(
            turn_detection={"type": "server_vad", "interrupt_response": True}
        )
    ),
)

runtime_cfg = RuntimeConfig()
runtime_cfg.apply_session_update(session)

---

VAD Handling: Detecting and Acting on Interruptions

The voice activity detector in src/speech_to_speech/VAD/vad_handler.py monitors incoming audio and triggers barge-in logic when interrupt_response_enabled is active. When new user speech is detected during an ongoing assistant response, the handler flushes the output queue to immediately stop the text-to-speech generation.

# src/speech_to_speech/VAD/vad_handler.py

if runtime_config.interrupt_response_enabled and new_user_speech:
    output_queue.flush()   # abort current TTS output

This queue flush discards in-flight audio frames without corrupting conversation state. The VAD handler preserves critical session-ending signals to ensure the pipeline remains coherent.

---

Pipeline Control Messages for Clean Turn Transitions

Barge-in requires more than queue flushing—it needs explicit pipeline coordination. The src/speech_to_speech/pipeline/control.py module defines PipelineControlMessage and the SESSION_END constant:

# src/speech_to_speech/pipeline/control.py

SESSION_END = PipelineControlMessage(ControlKind.SESSION_END)

When the VAD handler detects a barge-in, it emits this control message to force the current turn to terminate. This prevents race conditions where partial responses might leak into the next turn.

---

End-to-End Barge-In Flow

The complete interruption mechanism operates as follows:

1. Session initialization – Client sets interrupt_response: true in the turn_detection configuration
2. Assistant response – TTS audio streams to the client through the output queue
3. User interruption – VAD detects new speech while response is active
4. Queue flush – runtime_config.interrupt_response_enabled check passes, output_queue.flush() executes
5. Control signal – SESSION_END message enqueued to finalize the aborted turn
6. New turn begins – Pipeline accepts fresh user audio for the next response

---

Testing Barge-In Behavior

The repository includes tests verifying that barge-in clears state correctly. This pattern validates the flush mechanism preserves essential control messages:

def test_barge_in_flush_preserves_session_end():
    # ...setup pipeline, start a response...

    # Simulate user speaking while response is playing

    pipeline.vad_handler.on_audio_packet(user_speech_packet)
    # VAD will notice `interrupt_response_enabled` and flush the TTS queue

    assert not pipeline.output_queue.has_pending_tts()
    # SESSION_END control message is still present

    assert any(msg.kind == ControlKind.SESSION_END for msg in pipeline.control_messages)

---

Key Implementation Files

File	Purpose
src/speech_to_speech/api/openai_realtime/runtime_config.py	Defines interrupt_response_enabled property and session configuration
src/speech_to_speech/VAD/vad_handler.py	Detects voice activity and conditionally flushes output queue
src/speech_to_speech/pipeline/control.py	Implements PipelineControlMessage and SESSION_END signal
src/speech_to_speech/s2s_pipeline.py	Orchestrates VAD, LLM, and TTS components
tests/openai_realtime/test_websocket_router.py	Validates barge-in state cleanup

---

Summary

• Enable barge-in by setting interrupt_response: true in the turn_detection configuration
• Runtime check occurs via RuntimeConfig.interrupt_response_enabled in runtime_config.py
• Immediate response halt is achieved by output_queue.flush() in vad_handler.py
• State consistency is guaranteed through SESSION_END control messages from control.py
• Default behavior matches OpenAI Realtime API: interruptions enabled unless explicitly disabled

---

Frequently Asked Questions

How do I disable barge-in for my speech-to-speech deployment?

Set interrupt_response: false in the turn_detection configuration when creating or updating the session. The RuntimeConfig.interrupt_response_enabled property will return False, and the VAD handler will ignore incoming speech during active responses.

What happens to audio already queued when a user barges in?

The output_queue.flush() call in vad_handler.py discards all pending TTS frames immediately. Any audio already transmitted to the client may continue playing locally, but no new frames are generated or sent.

Does barge-in affect conversation history or context?

No. The interruption mechanism only affects the current turn's output generation. The SESSION_END control message ensures proper turn boundaries without truncating the conversation history sent to the language model.

Can I customize the VAD sensitivity for barge-in detection?

The repository uses the configured VAD parameters from the OpenAI Realtime session. Adjust silence_duration_ms and other turn_detection fields to tune how quickly speech triggers an interruption.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}