Configuring Smart Turn for End-of-Speech Detection Accuracy in Speech-to-Speech Systems

Smart Turn adds a secondary turn-completion check using an ONNX classifier that analyzes up to 8 seconds of acoustic and linguistic content to confirm whether a user has finished speaking after the primary VAD detects silence.

Configuring Smart Turn correctly is essential for balancing responsiveness against false cutoffs in real-time speech-to-speech pipelines. In the huggingface/speech-to-speech repository, this component operates as a second-level validator on top of the Silero VAD, applying a lightweight neural model to determine final turn boundaries. Understanding how to tune its parameters allows you to optimize end-of-speech detection accuracy for your specific hardware and latency requirements.

Understanding Smart Turn Architecture

The Smart Turn system consists of several coordinated components that process audio after the primary voice activity detector signals a potential speech-to-silence boundary.

SmartTurnAnalyzer and ONNX Model

The SmartTurnAnalyzer class (defined in src/speech_to_speech/VAD/smart_turn.py) loads and executes an ONNX model hosted in the Hugging Face repository pipecat-ai/smart-turn-v3. The specific model file is smart-turn-v3.2-cpu.onnx, loaded using ONNX Runtime with configurable CPU threading options via SessionOptions.

The analyzer returns a SmartTurnResult dataclass containing three critical fields:

• complete (bool): Whether the turn is finalized
• probability (float): Confidence score between 0 and 1
• inference_ms (float): Latency measurement for the prediction

Audio Pre-processing Pipeline

Before inference, the _prepare_audio method processes raw waveforms to meet the model's strict input requirements. The pipeline resamples any input sample rate to 16 kHz using SciPy's resample_poly, then pads or truncates the audio to exactly 8 seconds (MAX_AUDIO_SECONDS * MODEL_SAMPLE_RATE samples).

The normalized mono float32 array feeds into the WhisperFeatureExtractor from 🤗 Transformers, ensuring the feature space remains consistent with the Whisper STT model used elsewhere in the pipeline.

Integration with VAD Handler

The SmartTurnHandler in src/speech_to_speech/VAD/vad_handler.py orchestrates the detection flow. When the primary VAD detects silence, the handler invokes self.smart_turn_analyzer.predict(audio, sample_rate=16000). If SmartTurnResult.complete evaluates to True, the pipeline treats the segment as final; otherwise, it continues listening in speculative turn mode.

Configuration Parameters for Accuracy Tuning

Fine-tuning Smart Turn requires adjusting several initialization parameters exposed through SmartTurnAnalyzer.__init__.

Adjusting the Confidence Threshold

The threshold parameter (default 0.5) determines the cutoff for probability > threshold when deciding turn completion. Lower values (e.g., 0.3) create aggressive detection with earlier turn ends but risk false positives, while higher values (e.g., 0.7) demand stronger acoustic and linguistic evidence before concluding a turn, reducing premature cutoffs at the cost of increased latency.

Optimizing CPU Parallelism

The cpu_count parameter controls intra-op threads for ONNX Runtime. For edge devices requiring low latency, set this to 1 or 2 to prevent thread contention. Server-class CPUs can utilize 4 threads to reduce inference time when processing multiple concurrent streams.

Model Loading and Warm-up Options

Setting warmup=False skips the initial dummy inference, useful when you need the first prediction to represent true cold-start performance. The model_path parameter accepts local file paths (e.g., "/path/to/custom/smart-turn.onnx") to bypass automatic downloading from the Hugging Face Hub.

Implementation Examples

Stand-alone Analyzer Usage

import numpy as np
from speech_to_speech.VAD.smart_turn import SmartTurnAnalyzer

# Initialise with aggressive threshold for earlier detection

analyzer = SmartTurnAnalyzer(threshold=0.35, cpu_count=2)

# Simulated 2-second mono audio at 16 kHz

audio = np.random.randn(2 * 16000).astype(np.float32)

result = analyzer.predict(audio, sample_rate=16000)
print(f"Complete? {result.complete} – prob={result.probability:.3f} (took {result.inference_ms:.1f} ms)")

Custom VAD Handler Integration

from speech_to_speech.VAD.vad_handler import VADHandler
from speech_to_speech.VAD.smart_turn import SmartTurnAnalyzer

# Create analyzer with conservative settings

custom_analyzer = SmartTurnAnalyzer(threshold=0.6, cpu_count=4, warmup=False)

# Inject into handler

handler = VADHandler(...)
handler.smart_turn_analyzer = custom_analyzer

Loading Local Model Files

analyzer = SmartTurnAnalyzer(
    model_path="/path/to/local/smart-turn-v3.2-cpu.onnx",
    threshold=0.5,
    cpu_count=1,
)

Full Pipeline Configuration

from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline

pipeline = SpeechToSpeechPipeline(
    vad_smart_turn_threshold=0.45,
    vad_cpu_count=2,
)

pipeline.run(audio_source="microphone")

Summary

• Smart Turn acts as a secondary validator after Silero VAD, using an ONNX model to analyze 8-second audio windows for turn completion.
• The threshold parameter directly controls the trade-off between false positives (early cutoff) and false negatives (late response).
• cpu_count should be set to 1-2 for low-latency edge devices and up to 4 for server deployments.
• Audio preprocessing occurs in SmartTurnAnalyzer._prepare_audio, utilizing SciPy resampling and Whisper feature extraction.
• Configuration happens through SmartTurnAnalyzer.__init__ or pipeline parameters in SpeechToSpeechPipeline.

Frequently Asked Questions

How does Smart Turn differ from the primary VAD?

The primary Silero VAD detects speech-versus-silence boundaries based on acoustic energy alone. Smart Turn adds a linguistic and acoustic understanding layer by running a classifier on the actual content of the utterance to determine if the user has semantically completed their thought, reducing false turn endings on mid-sentence pauses.

What is the optimal threshold setting for real-time applications?

For real-time conversational AI, start with the default 0.5 and adjust based on user feedback. Use 0.3-0.4 for faster response times when users speak in short commands, or 0.6-0.7 for formal dialogues where interrupting mid-thought is costly. Monitor the inference_ms field to ensure threshold adjustments don't coincide with performance degradation.

Can Smart Turn run on GPU instead of CPU?

The current implementation in smart_turn.py uses ONNX Runtime with CPU-specific SessionOptions and single-threaded configurations. While the model architecture supports GPU execution via ONNX Runtime's CUDA provider, the repository's SmartTurnAnalyzer is optimized for CPU inference to maintain compatibility with edge deployment scenarios.

Why does the analyzer require exactly 8 seconds of audio?

The ONNX model was trained on fixed-length 8-second segments (16 kHz sample rate) to standardize the input tensor shape. The _prepare_audio method enforces this by truncating longer utterances or zero-padding shorter ones, ensuring the Whisper feature extractor outputs consistent dimensions regardless of actual speech duration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}