How to Configure Smart Turn Endpointing Parameters to Reduce False Speech Finalizations in Noisy Environments

Raise the smart_turn_threshold to 0.6–0.8, increase smart_turn_max_wait_ms to 3000 ms, and extend smart_turn_incomplete_delay_ms to 800–1200 ms to make the system less eager to finalize turns in noisy environments.

The Smart Turn module in the huggingface/speech-to-speech repository provides intelligent endpointing that runs after the primary VAD to determine whether a speech segment is truly complete or merely paused. Configuring these Smart Turn endpointing parameters correctly is essential for production deployments where background noise can trigger premature finalizations. This guide explains how to tune the ONNX-based classifier to be more tolerant of acoustic interference while maintaining responsive interactions.

Understanding the Smart Turn Module

Smart Turn is a lightweight ONNX-based classifier that executes after the Silero VAD detects a speech-to-silence boundary. Unlike basic voice activity detection, this module analyzes up to eight seconds of audio context to predict whether the speaker has genuinely finished speaking or is likely to continue after a brief pause.

The classifier resamples incoming audio to 16 kHz, pads or truncates it to exactly eight seconds (as defined by MAX_AUDIO_SECONDS = 8 in src/speech_to_speech/VAD/smart_turn.py lines 23–24), and returns a completion probability. When this probability exceeds the configured threshold, the system finalizes the turn; otherwise, it keeps the turn speculative, allowing for resumption without triggering downstream STT or LLM processing.

Key Smart Turn Endpointing Parameters

The system exposes several tunable parameters through VADHandlerArguments in src/speech_to_speech/arguments_classes/vad_arguments.py. Adjusting these values directly impacts how the system behaves in noisy acoustic environments.

smart_turn_threshold

The smart_turn_threshold parameter sets the probability cut-off for declaring a turn complete. When the ONNX model outputs a probability greater than this threshold, the system commits the turn immediately.

• Default: 0.5
• Noisy environment recommendation: 0.6 to 0.8
• Location: src/speech_to_speech/VAD/smart_turn.py lines 47–48

Raising this value forces the model to require higher confidence before closing a turn, which prevents noise-induced spurious "complete" scores from finalizing speech prematurely.

smart_turn_max_wait_ms

The smart_turn_max_wait_ms parameter defines the upper bound on how long the system keeps a speculative turn open after an incomplete Smart Turn prediction. This gives users time to resume speaking after a pause.

• Default: 2000 ms
• Noisy environment recommendation: 3000 ms
• Location: src/speech_to_speech/arguments_classes/vad_arguments.py lines 100–104

Extending this window accommodates noisy pauses where the user might be hesitating or where background silence is intermittent.

smart_turn_incomplete_delay_ms

The smart_turn_incomplete_delay_ms parameter specifies the delay before downstream STT/LLM processing begins when Smart Turn reports incomplete. This pause allows a resumed utterance to invalidate pending work before costly inference starts.

• Default: 600 ms
• Noisy environment recommendation: 800 to 1200 ms
• Location: src/speech_to_speech/arguments_classes/vad_arguments.py lines 106–110

Increasing this delay prevents the system from initiating expensive transcription or language model inference during brief noise-induced silences.

smart_turn_cpu_count

The smart_turn_cpu_count parameter controls the number of CPU threads ONNX Runtime may use for each inference. In high-noise environments with frequent re-evaluations, additional threads reduce latency.

• Default: 1
• Noisy environment recommendation: 2 to 4
• Location: src/speech_to_speech/arguments_classes/vad_arguments.py lines 112–115

smart_turn_model_path

The smart_turn_model_path parameter specifies the path to a custom Smart Turn ONNX model. By default, the system automatically downloads the latest v3.2 CPU model from the Hugging Face Hub.

• Default: None (auto-download)
• Noisy environment recommendation: Path to a model fine-tuned on noisy corpora
• Location: src/speech_to_speech/arguments_classes/vad_arguments.py lines 88–92

speculative_reopen_ms (VAD-level)

The speculative_reopen_ms parameter operates at the VAD level and defines the time window where a soft-ended turn remains reopenable before a response commits it. This provides leeway when Smart Turn is disabled or uncertain.

• Default: 800 ms
• Noisy environment recommendation: 1200 ms
• Location: src/speech_to_speech/arguments_classes/vad_arguments.py lines 64–68

How Smart Turn Endpointing Works Internally

The VADHandler in src/speech_to_speech/VAD/vad_handler.py orchestrates the endpointing logic through the following sequence:

1. Silero VAD marks a speech-to-silence boundary in the audio stream.
2. Smart Turn receives up to eight seconds of the utterance context.
3. The audio is resampled to 16 kHz and fed to the ONNX model defined in smart_turn.py.
4. The model returns a single probability via SmartTurnResult (lines 30–33), indicating completion if probability > threshold.
5. If complete, the VADHandler commits the turn immediately. If incomplete, the handler keeps the turn speculative for smart_turn_max_wait_ms.
6. When the result is incomplete, the handler inserts a smart_turn_incomplete_delay_ms pause before triggering downstream pipelines, allowing users to resume speaking without incurring unnecessary processing costs.

Configuring Parameters for Noisy Environments

CLI Configuration

You can pass these parameters directly via the command-line interface defined in src/speech_to_speech/cli.py:

speech-to-speech \
  --url wss://my-realtime-endpoint/realtime \
  --smart_turn_threshold 0.7 \
  --smart_turn_max_wait_ms 3000 \
  --smart_turn_incomplete_delay_ms 1000 \
  --smart_turn_cpu_count 2 \
  --speculative_reopen_ms 1200

Programmatic Configuration

For Python applications, instantiate VADHandlerArguments and pass it to the VADHandler:

from speech_to_speech.arguments_classes.vad_arguments import VADHandlerArguments
from speech_to_speech.VAD.vad_handler import VADHandler

args = VADHandlerArguments(
    smart_turn=True,
    smart_turn_threshold=0.7,
    smart_turn_max_wait_ms=3000,
    smart_turn_incomplete_delay_ms=1000,
    smart_turn_cpu_count=2,
    speculative_reopen_ms=1200,
)

handler = VADHandler(args)

Using Custom ONNX Models

If you have a Smart Turn model fine-tuned on noisy audio corpora, specify the path via smart_turn_model_path:

handler_args = VADHandlerArguments(
    smart_turn=True,
    smart_turn_model_path="/path/to/custom-smart-turn.onnx",
    smart_turn_threshold=0.65,
)
handler = VADHandler(handler_args)

Summary

• Raise smart_turn_threshold to 0.6–0.8 to require higher model confidence before finalizing turns in noisy environments.
• Increase smart_turn_max_wait_ms to 3000 ms to keep speculative turns open longer during acoustic pauses.
• Extend smart_turn_incomplete_delay_ms to 800–1200 ms to prevent costly downstream processing during brief noise-induced silences.
• Allocate more CPU threads via smart_turn_cpu_count (2–4) to maintain low inference latency when re-evaluating frequently.
• Deploy custom models using smart_turn_model_path if the default ONNX model struggles with your specific noise profile.
• Adjust speculative_reopen_ms at the VAD level to provide additional fallback leeway when Smart Turn predictions are uncertain.

Frequently Asked Questions

What is the difference between Silero VAD and Smart Turn endpointing?

Silero VAD detects raw speech activity and marks boundaries between speech and silence, while Smart Turn is a secondary ONNX-based classifier that determines whether a silence boundary represents a genuine turn completion or merely a pause within an ongoing utterance. According to the huggingface/speech-to-speech source code, Smart Turn runs after Silero VAD and uses up to eight seconds of audio context to make this determination.

Why does raising the smart_turn_threshold reduce false finalizations?

Raising the smart_turn_threshold from the default 0.5 to 0.6–0.8 forces the ONNX model in src/speech_to_speech/VAD/smart_turn.py to require higher confidence before the VADHandler commits a turn. In noisy environments, ambient sounds can produce spurious "complete" predictions with probabilities between 0.5 and 0.6; by elevating the threshold, these uncertain predictions remain classified as incomplete, keeping the turn speculative and reopenable.

How does smart_turn_incomplete_delay_ms prevent wasted computation?

The smart_turn_incomplete_delay_ms parameter inserts a configurable pause (default 600 ms, recommended 800–1200 ms for noise) before triggering downstream STT or LLM pipelines when Smart Turn reports incomplete. As implemented in src/speech_to_speech/VAD/vad_handler.py, this delay allows users to resume speaking immediately after a brief noise-induced pause, invalidating the pending inference before GPU or CPU resources are allocated to process the utterance.

Can I use Smart Turn with a custom-trained model for specific noise environments?

Yes. The smart_turn_model_path parameter in src/speech_to_speech/arguments_classes/vad_arguments.py accepts a file system path to any compatible ONNX model. You can replace the default v3.2 CPU model (auto-downloaded from the Hugging Face Hub) with a version fine-tuned on your specific acoustic environment, such as automotive cabins, call centers, or industrial settings, to improve endpointing accuracy without modifying the inference logic in smart_turn.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}