How to Handle Audio Interruptions and Manage Turn-Taking in the Speech-to-Speech Pipeline

The huggingface/speech-to-speech pipeline handles interruptions and turn-taking through a three-component architecture: VADHandler detects voice activity and creates turn identifiers, SpeculativeTurnTracker manages thread-safe turn revisions, and SmartTurnAnalyzer predicts whether the user has finished speaking using an ONNX model.

The speech-to-speech repository implements production-grade turn management that prevents "ghost" responses when users interrupt themselves or pause mid-thought. Unlike simple VAD systems that treat every silence as a turn boundary, this pipeline uses speculative turn tracking and smart turn prediction to distinguish between meaningful pauses and genuine end-of-turn silence.

Core Turn-Taking Architecture

The interruption handling system rests on three specialized components that work together in the VADHandler process loop.

VADHandler: Turn Creation and Reopen Logic

Located in src/speech_to_speech/VAD/vad_handler.py, the VADHandler class is the sole module responsible for turn identity. It uses Silero VAD for voice detection and implements three critical methods:

• _should_reopen_current_turn — checks if elapsed silence falls within speculative_reopen_ms or the extended unanswered_reopen_ms window
• _begin_pending_reopen_if_needed — creates a speculative reopen candidate when speech resumes inside the grace window
• _ensure_turn_for_speech_start — guarantees a turn exists before emitting SpeechStartedEvent

When a user pauses, the handler calculates whether the silence permits reopening. If speech resumes within the threshold, the same turn continues with a new revision rather than spawning a fresh turn.

SpeculativeTurnTracker: Thread-Safe Revision Management

The SpeculativeTurnTracker in src/speech_to_speech/pipeline/speculative_turns.py provides atomic operations for turn state:

Method	Purpose
begin_reopen_candidate	Registers revision N+1 for the current turn
confirm_reopen_candidate	Promotes candidate to active revision, discarding stale LLM output
start_reopen_grace	Initiates the grace period during which reopening is permitted without committing output
commit	Finalizes the turn, allowing downstream processing to complete

This tracker ensures that when a user interrupts their own utterance, any partially generated LLM response associated with the previous revision is discarded rather than played to the user.

SmartTurnAnalyzer: Neural Turn Completion Prediction

The SmartTurnAnalyzer in src/speech_to_speech/VAD/smart_turn.py runs an ONNX model to classify utterances as complete or incomplete. The _smart_turn_timing_ms method in VADHandler calls SmartTurnAnalyzer.predict, which:

1. Resamples audio to the model's expected input rate
2. Runs inference via ONNX Runtime
3. Returns a classification with confidence score

For incomplete predictions, the handler extends the grace period to smart_turn_max_wait_ms (default 2000 ms) and optionally applies smart_turn_incomplete_delay_ms before processing.

The Interruption Handling Flow

Here is the precise sequence when a user interrupts themselves:

1. Speech starts — VADHandler creates turn_1 with revision 0
2. User pauses — VAD detects silence; _should_reopen_current_turn evaluates the reopen window
3. Pending reopen created — _begin_pending_reopen_if_needed registers revision 1 as a candidate in SpeculativeTurnTracker
4. User resumes within grace — _confirm_pending_reopen promotes the candidate; same turn ID, incremented revision; prior LLM output discarded
5. User actually stops — SmartTurnAnalyzer.predict returns complete; normal speculative_reopen_ms applies; grace expires; turn commits

If SmartTurnAnalyzer instead returns incomplete, step 5 uses the extended grace, keeping the turn uncommitted and reopenable.

Configuring Turn-Taking Behavior

All timing parameters are defined in src/speech_to_speech/arguments_classes/vad_arguments.py and passed through s2s_pipeline.

Key CLI Arguments

Argument	Description	Default
--smart_turn	Enable Smart Turn inference (requires onnxruntime)	enabled
--no_smart_turn	Disable neural turn prediction	—
--smart_turn_max_wait_ms	Maximum grace after incomplete prediction	2000
--smart_turn_incomplete_delay_ms	Processing delay while waiting for completion	600
--speculative_reopen_ms	Grace when Smart Turn is disabled	800
--unanswered_reopen_ms	Upper bound when no assistant output committed	7000
--min_speech_ms	Minimum active speech to start a turn	384
--min_speech_continuation_ms	Minimum speech to continue reopenable turn	192

Command-Line Example

Enable Smart Turn with custom timing in realtime mode:

python -m speech_to_speech.main \
  --mode realtime \
  --smart_turn \
  --smart_turn_model_path /models/smart-turn.onnx \
  --smart_turn_threshold 0.6 \
  --smart_turn_max_wait_ms 2500 \
  --smart_turn_incomplete_delay_ms 800

Programmatic Pipeline Construction

For custom integrations, instantiate the components directly:

from speech_to_speech.VAD.vad_handler import VADHandler
from speech_to_speech.VAD.smart_turn import SmartTurnAnalyzer
from speech_to_speech.pipeline.speculative_turns import SpeculativeTurnTracker
from threading import Event
from queue import Queue

should_listen = Event()
audio_q = Queue()
output_q = Queue()
spec_turns = SpeculativeTurnTracker()

vad = VADHandler(
    should_listen,
    audio_q,
    output_q,
    setup_args=(should_listen,),
    setup_kwargs={
        "sample_rate": 16000,
        "smart_turn": True,
        "smart_turn_model_path": "/models/smart-turn.onnx",
        "smart_turn_threshold": 0.6,
        "smart_turn_max_wait_ms": 2500,
        "smart_turn_incomplete_delay_ms": 800,
        "speculative_turns": spec_turns,
        "text_output_queue": None,
    },
)

# Process raw PCM chunks

vad.process(b'\x00\x01\x02...')

The VADHandler.process method runs Silero VAD on each chunk. When a segment finalizes, _smart_turn_timing_ms determines the grace period. If incomplete, the extended timing prevents premature commitment.

Pipeline Data Flow

Downstream handlers receive VADAudio events carrying turn_id and turn_revision metadata. The LMOutputProcessor and TTS stages use these fields to discard output from superseded revisions, ensuring only the final confirmed revision reaches the user.

Key integration files:

• src/speech_to_speech/s2s_pipeline.py — orchestrates handlers and injects SpeculativeTurnTracker
• src/speech_to_speech/api/openai_realtime/runtime_config.py — per-session runtime configuration
• src/speech_to_speech/LLM/audio_input_notifier.py — emits explicit turn signals for LLMs requiring them

Summary

• VADHandler in vad_handler.py is the single source of truth for turn identity and reopen decisions
• SpeculativeTurnTracker in speculative_turns.py provides atomic revision management with grace period control
• SmartTurnAnalyzer in smart_turn.py uses ONNX inference to distinguish incomplete from complete utterances
• Reopen candidates prevent ghost responses by discarding LLM output from superseded revisions
• All timing thresholds are configurable via CLI arguments in vad_arguments.py

Frequently Asked Questions

What triggers a turn reopen versus a new turn creation?

A turn reopens when speech resumes during the grace period after a silence. The VADHandler._should_reopen_current_turn method checks if elapsed time is within speculative_reopen_ms (or unanswered_reopen_ms if no assistant response exists). If outside this window, fresh speech creates a new turn with a new ID.

How does the pipeline prevent playing outdated LLM responses?

SpeculativeTurnTracker assigns monotonic revisions per turn. When a reopen is confirmed, the revision increments. Downstream processors check the revision attached to each VADAudio event and discard any LLM output generated for prior revisions before it reaches TTS.

Can Smart Turn be disabled for faster inference?

Yes. Pass --no_smart_turn to use fixed speculative_reopen_ms timing instead of neural prediction. This removes the ONNX Runtime dependency but provides less accurate turn boundaries for conversational speech patterns.

What is the difference between speculative_reopen_ms and smart_turn_max_wait_ms?

speculative_reopen_ms (default 800 ms) applies when Smart Turn predicts complete or is disabled. smart_turn_max_wait_ms (default 2000 ms) extends the grace when the model predicts incomplete, giving users more time to finish complex utterances without triggering a false turn end.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}