How to Configure Smart Turn Endpointing for Better Turn-Taking

Smart Turn endpointing uses a lightweight ONNX model to predict whether a user has finished speaking, allowing the system to delay assistant responses during incomplete turns and prevent conversational interruptions.

The huggingface/speech-to-speech repository implements an advanced turn-taking mechanism called Smart Turn that reduces premature interruptions in voice conversations. By configuring Smart Turn endpointing parameters, you can fine-tune how the system distinguishes between completed and incomplete user utterances. This guide explains the architecture, configuration options, and tuning strategies based on the actual source code implementation.

Smart Turn Architecture Overview

Smart Turn operates as a secondary analysis layer after the Silero VAD detects the end of a speech segment. The system predicts whether the user's utterance is complete or incomplete, then adjusts the speculative response window accordingly. This flow is exercised in the test suite (tests/test_smart_turn.py lines 44‑62).

Core Components

The implementation consists of three primary components defined across the codebase:

VAD Handler (src/speech_to_speech/VAD/vad_handler.py): The orchestration layer that initializes Smart Turn and executes turn-taking logic. The setup() method (lines 59-84) creates a SmartTurnAnalyzer instance when smart_turn=True and stores configuration values for later use.

# src/speech_to_speech/VAD/vad_handler.py (lines 59-84)

def setup(..., smart_turn: bool = True,
          smart_turn_model_path: str | None = None,
          smart_turn_threshold: float = 0.5,
          smart_turn_max_wait_ms: int = 2000,
          smart_turn_incomplete_delay_ms: int = 600,
          smart_turn_cpu_count: int = 1, ...):

Smart Turn Analyzer (src/speech_to_speech/VAD/smart_turn.py): Wraps the ONNX inference model and handles audio preprocessing. The analyzer pads or trims input to a maximum of 8 seconds (MAX_AUDIO_SECONDS = 8) and resamples to 16 kHz (MODEL_SAMPLE_RATE = 16000) before running prediction.

# src/speech_to_speech/VAD/smart_turn.py

class SmartTurnAnalyzer:
    def __init__(..., model_path: str | None = None,
                 threshold: float = 0.5,
                 cpu_count: int = 1, warmup: bool = True):

Argument Classes (src/speech_to_speech/arguments_classes/vad_arguments.py): Exposes all Smart Turn parameters to the CLI and configuration system through the VADHandlerArguments dataclass.

Configuration Parameters

All Smart Turn settings are accessible via the VADHandlerArguments class in src/speech_to_speech/arguments_classes/vad_arguments.py. The following table details each parameter:

Parameter	Default	Description
smart_turn	True	Master toggle to enable or disable Smart Turn analysis
smart_turn_threshold	0.5	Probability threshold for classifying a turn as "complete"
smart_turn_max_wait_ms	2000	Maximum milliseconds to keep response speculative when turn is incomplete
smart_turn_incomplete_delay_ms	600	Processing delay added after incomplete predictions to allow speech resumption
smart_turn_cpu_count	1	Number of CPU threads allocated to ONNX Runtime inference
smart_turn_model_path	None	Optional path to a custom ONNX model file

Implementation Examples

You can configure Smart Turn endpointing programmatically through the Python API or via command-line arguments when running the pipeline.

Python API Configuration

Instantiate the VADHandler and call setup() with your desired Smart Turn parameters. Note that you must provide a SpeculativeTurnTracker instance when enabling speculative turns.

from speech_to_speech.VAD.vad_handler import VADHandler
from speech_to_speech.pipeline.speculative_turns import SpeculativeTurnTracker
from threading import Event

# Initialize required tracker

tracker = SpeculativeTurnTracker()

handler = VADHandler()
handler.setup(
    should_listen=Event(),
    speculative_turns=tracker,
    smart_turn=True,
    smart_turn_threshold=0.7,          # Require higher confidence for completion

    smart_turn_max_wait_ms=3000,       # Wait up to 3s for incomplete turns

    smart_turn_incomplete_delay_ms=800,
    smart_turn_cpu_count=2,            # Parallel inference threads

)

CLI Configuration

Pass arguments directly when launching the pipeline from the command line:

python -m speech_to_speech.cli \
    --smart_turn \
    --smart_turn_threshold 0.6 \
    --smart_turn_max_wait_ms 2500 \
    --smart_turn_incomplete_delay_ms 500 \
    --smart_turn_cpu_count 2

Turn-Taking Logic and Tuning

The method _smart_turn_timing_ms (lines 509-542 in vad_handler.py) implements the decision logic that translates model predictions into timing adjustments. This function returns a tuple (reopen_grace_ms, processing_delay_ms) that controls the speculative window and downstream processing delays.

Behavior Modes

The system responds differently based on the SmartTurnResult:

• Complete Turn: When result.complete == True, the handler uses the standard speculative reopen window (default 800 ms) with no additional processing delay.
• Incomplete Turn: The system extends the speculative grace period to smart_turn_max_wait_ms and adds smart_turn_incomplete_delay_ms before starting STT/LLM processing. This allows users to resume speaking without the assistant interrupting.
• Inference Failure: Falls back to the default speculative window to maintain conversation flow.

Tuning Strategies

Adjust these parameters to optimize for your specific use case:

• Increase smart_turn_threshold (e.g., to 0.7 or 0.8) to make the system more conservative about declaring turns complete. This prevents cut-offs during short pauses but may increase response latency.
• Raise smart_turn_max_wait_ms for users with slower speech patterns or when running complex downstream pipelines that benefit from longer speculative windows.
• Lower smart_turn_incomplete_delay_ms (e.g., to 300-400 ms) for faster response times in high-latency environments, though this risks processing utterances that the user intends to continue.
• Increase smart_turn_cpu_count on multi-core servers to reduce inference latency, though the default single-threaded configuration is optimal for most client-side deployments.

Summary

• Smart Turn endpointing in huggingface/speech-to-speech uses an ONNX model to classify user utterances as complete or incomplete after VAD detection.
• Configure parameters via VADHandlerArguments in src/speech_to_speech/arguments_classes/vad_arguments.py or through CLI flags.
• Key tuning knobs include smart_turn_threshold for completion confidence and smart_turn_max_wait_ms for speculative window duration.
• The _smart_turn_timing_ms method in vad_handler.py orchestrates turn-taking by adjusting grace periods and processing delays based on model predictions.

Frequently Asked Questions

What is the default Smart Turn threshold and when should I change it?

The default threshold is 0.5, defined in src/speech_to_speech/arguments_classes/vad_arguments.py. Increase this value to 0.6-0.8 when your application serves users who pause frequently mid-thought, as it requires higher model confidence before committing the assistant's response. Lower values (0.3-0.4) create more responsive but potentially interruptive behavior for fast turn-taking scenarios.

How does Smart Turn handle incomplete user utterances?

When the analyzer predicts an incomplete turn (probability below threshold), the system activates two protective mechanisms defined in _smart_turn_timing_ms (lines 509-542 of vad_handler.py). First, it extends the speculative response window to smart_turn_max_wait_ms (default 2000 ms), allowing the user to continue speaking. Second, it introduces a processing delay of smart_turn_incomplete_delay_ms (default 600 ms) before sending audio to the STT/LLM pipeline, preventing wasted computation on partial utterances.

Can I use a custom ONNX model for Smart Turn prediction?

Yes. Pass a file path to the smart_turn_model_path parameter in the VADHandler.setup() method or via the --smart_turn_model_path CLI argument. The custom model must accept audio input preprocessed to 16 kHz sample rate and maximum 8 seconds duration, matching the interface defined in src/speech_to_speech/VAD/smart_turn.py. The SmartTurnAnalyzer handles all resampling and padding automatically.

What audio preprocessing does the Smart Turn Analyzer perform?

According to src/speech_to_speech/VAD/smart_turn.py, the SmartTurnAnalyzer resamples all input to 16 kHz (MODEL_SAMPLE_RATE = 16000) and pads or truncates audio to a maximum of 8 seconds (MAX_AUDIO_SECONDS = 8). This standardization ensures compatibility with the ONNX model's expected input dimensions while maintaining real-time performance characteristics suitable for conversational AI applications.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}