What Is Smart Turn Endpointing and How to Enable It in Hugging Face Speech-to-Speech

Smart Turn endpointing is an optional post-VAD step that uses a lightweight ONNX classifier to detect whether a user's speech turn is complete or incomplete, preventing premature assistant responses and reducing conversational cut-offs.

Smart Turn endpointing helps real-time speech systems wait for users to finish speaking before generating a response. In the huggingface/speech-to-speech repository, this feature runs automatically after Silero VAD detects a speech-to-silence boundary, using a dedicated neural network to judge turn completion with configurable timing rules.

How Smart Turn Endpointing Works

The Smart Turn system extends the basic VAD pipeline with intelligent turn-completion detection.

The Smart Turn Workflow

After Silero VAD signals a potential end-of-speech boundary, the VADHandler triggers the following process:

1. Audio capture: The handler extracts the last ~8 seconds of audio preceding the silence
2. Model inference: SmartTurnAnalyzer.predict() runs the ONNX classifier on this audio window
3. Decision routing: Based on the prediction, the system applies different timing strategies

The decision logic resides in VADHandler._smart_turn_timing_ms (lines 511–541 of src/speech_to_speech/VAD/vad_handler.py), while the model implementation lives in src/speech_to_speech/VAD/smart_turn.py within the SmartTurnAnalyzer class and SmartTurnResult dataclass.

Timing Behavior Based on Predictions

Prediction	System Behavior
Complete (result.complete == True)	Uses normal short speculative reopen window (speculative_reopen_ms)
Incomplete	Extends speculative grace to smart_turn_max_wait_ms (default 2000ms) and adds smart_turn_incomplete_delay_ms (default 600ms)

The incomplete path gives users extra time to resume speaking without the assistant committing to a premature response, directly reducing cut-off artifacts in real-time conversations.

How to Enable and Configure Smart Turn

Smart Turn endpointing is enabled by default in the speech-to-speech pipeline. You only need to configure it explicitly when customizing behavior or disabling it.

Disabling Smart Turn

To turn the feature off entirely, use either:

speech-to-speech --no_smart_turn

Or in Python:

from speech_to_speech.arguments_classes.vad_arguments import VADArguments

vad_args = VADArguments(smart_turn=False)

CLI Configuration Options

All Smart Turn parameters are defined in src/speech_to_speech/arguments_classes/vad_arguments.py:

Flag	Description	Default
--smart_turn_model_path	Local path to ONNX model; auto-downloads pipecat-ai/smart-turn-v3 if omitted	None
--smart_turn_threshold	Probability threshold for "complete" classification	0.5
--smart_turn_max_wait_ms	Maximum wait time when turn judged incomplete	2000
--smart_turn_incomplete_delay_ms	Extra STT/LLM processing delay after incomplete prediction	600
--smart_turn_cpu_count	ONNX Runtime CPU threads	1

Example: Custom Model with Higher Threshold

speech-to-speech \
  --smart_turn \
  --smart_turn_model_path /models/smart-turn-v3.2-cpu.onnx \
  --smart_turn_threshold 0.7 \
  --smart_turn_max_wait_ms 2500 \
  --smart_turn_incomplete_delay_ms 800

This configuration uses a stricter completion threshold (0.7 vs. 0.5) and allows more wait time for incomplete turns.

Implementing Smart Turn in Python Code

Instantiate VADHandler with Smart Turn

from speech_to_speech.VAD.vad_handler import VADHandler

handler = VADHandler(
    sample_rate=16000,
    smart_turn=True,                     # default, shown explicitly

    smart_turn_model_path=None,          # None triggers auto-download

    smart_turn_threshold=0.6,
    smart_turn_max_wait_ms=2500,
    smart_turn_incomplete_delay_ms=800,
)

The smart_turn_model_path=None setting triggers automatic download of the Smart Turn v3.2 CPU model from the pipecat-ai/smart-turn-v3 Hugging Face hub.

Direct SmartTurnAnalyzer Usage

For debugging or custom pipelines, invoke the predictor directly:

from speech_to_speech.VAD.smart_turn import SmartTurnAnalyzer

analyzer = SmartTurnAnalyzer(threshold=0.6)

# audio: NumPy float32 array, ~8 seconds at 16kHz recommended

result = analyzer.predict(audio)
print(f"Complete: {result.complete}, Probability: {result.probability}")

The SmartTurnResult dataclass exposes complete (boolean) and probability (float) fields for downstream logic.

Key Implementation Files

File	Purpose
src/speech_to_speech/VAD/smart_turn.py	SmartTurnAnalyzer class, model download, audio preprocessing, ONNX inference
src/speech_to_speech/VAD/vad_handler.py	Integration with VAD pipeline; _smart_turn_timing_ms timing decisions
src/speech_to_speech/arguments_classes/vad_arguments.py	CLI/JSON argument definitions
tests/test_smart_turn.py	Unit tests for timing behavior and integration

Why Smart Turn Endpointing Matters

Smart Turn addresses two critical real-time conversation problems:

• Reduces cut-off responses: By detecting incomplete turns and extending wait windows, the system avoids interrupting users who pause mid-thought
• Saves compute and latency: Prevents wasted STT/LLT processing on turns where the user will clearly continue speaking

The lightweight ONNX model (Smart Turn v3.2) adds minimal overhead while significantly improving conversational naturalness.

Summary

• Smart Turn endpointing is a post-VAD classifier that judges speech turn completion using an ONNX model
• The feature is enabled by default; disable with --no_smart_turn or smart_turn=False
• Configuration happens through five CLI arguments controlling model path, threshold, wait times, and compute threads
• Incomplete predictions trigger extended grace periods (default 2000ms max wait + 600ms delay) to prevent premature responses
• Core implementation spans smart_turn.py (inference), vad_handler.py (integration), and vad_arguments.py (configuration)

Frequently Asked Questions

What model does Smart Turn endpointing use?

Smart Turn uses the Smart Turn v3.2 classifier, a lightweight ONNX model provided through the pipecat-ai/smart-turn-v3 Hugging Face hub. The model runs on CPU via ONNX Runtime with configurable thread count.

How much audio does Smart Turn analyze?

The SmartTurnAnalyzer typically processes the last ~8 seconds of audio preceding a VAD-detected silence boundary. This window provides sufficient context for the classifier to judge turn completion patterns.

Can I use a custom Smart Turn model?

Yes. Pass --smart_turn_model_path <path> or set smart_turn_model_path in VADHandler to load a local ONNX file. The system skips automatic hub download when a valid path is provided.

Does Smart Turn add latency to the pipeline?

Smart Turn adds minimal inference latency (lightweight ONNX model) but can reduce perceived latency overall by preventing premature STT/LLM invocations. The trade-off is configurable: lower thresholds and shorter smart_turn_max_wait_ms values reduce waiting time at the cost of more potential cut-offs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}