How to Configure Smart Turn Endpointing for Voice Activity Detection in Hugging Face Speech-to-Speech

Smart Turn endpointing is controlled via six configuration parameters that adjust the completeness threshold, wait times, and inference resources for a post-VAD classifier in speech_to_speech/VAD/smart_turn.py.

What Is Smart Turn Endpointing?

Smart Turn is an optional post-VAD classifier that determines whether a spoken utterance is complete or incomplete. It integrates directly into the voice activity detection pipeline in the huggingface/speech-to-speech repository to improve turn-taking accuracy in real-time conversations.

When enabled, Smart Turn affects two critical timing behaviors:

• Speculative reopen grace — how long the assistant's response remains speculative after speech ends
• Processing delay — an optional pause before STT/LLM processing when the turn is flagged as incomplete

Smart Turn is enabled by default in recent versions of the framework.

Smart Turn Architecture

The system comprises three interconnected components:

Component	File Location	Primary Responsibility
SmartTurnAnalyzer	speech_to_speech/VAD/smart_turn.py	Loads the ONNX model, runs inference, returns SmartTurnResult with complete, probability, and inference_ms fields
VADHandler	speech_to_speech/VAD/vad_handler.py	Orchestrates VAD detection, accumulates audio, invokes Smart Turn timing logic via _smart_turn_timing_ms()
VADHandlerArguments	speech_to_speech/arguments_classes/vad_arguments.py	Exposes all CLI and programmatic configuration knobs

The inference flow works as follows:

1. VAD detects speech end via VADIterator
2. Final segment triggers _smart_turn_timing_ms() in VADHandler (lines 15-22)
3. If SmartTurnAnalyzer exists, predict() runs ONNX inference
4. Based on result.complete, the handler returns timing values:
   • Complete turn: speculative_reopen_ms (default 800 ms)
   • Incomplete turn: smart_turn_max_wait_ms (default 2000 ms) plus processing_delay_ms (clamped to smart_turn_incomplete_delay_ms)

These values schedule the speculative reopen grace (lines 71-74) and optionally delay downstream processing (lines 81-82).

Smart Turn Configuration Parameters

All parameters are defined in vad_arguments.py (lines 82-115) and passed to VADHandler.setup():

Parameter	Type	Default	Description
smart_turn	bool	True	Master enable/disable switch
smart_turn_model_path	str | None	None	Path to custom ONNX model; auto-downloads v3.2 from pipecat-ai/smart-turn-v3 if omitted
smart_turn_threshold	float	0.5	Probability threshold for declaring turn complete (0.0–1.0)
smart_turn_max_wait_ms	int	2000	Maximum grace period for incomplete turns in milliseconds
smart_turn_incomplete_delay_ms	int	600	Additional STT/LLM delay after incomplete prediction; clamped to smart_turn_max_wait_ms
smart_turn_cpu_count	int	1	CPU threads for ONNX Runtime inference

CLI Configuration Examples

Basic Enable with Custom Threshold

python -m speech_to_speech \
  --smart_turn \
  --smart_turn_threshold 0.7 \
  --smart_turn_max_wait_ms 2500

Raises the completeness threshold to reduce false positives in backchanneling scenarios.

Production Tuning with Multi-Threading

python -m speech_to_speech \
  --smart_turn \
  --smart_turn_threshold 0.6 \
  --smart_turn_max_wait_ms 3000 \
  --smart_turn_incomplete_delay_ms 800 \
  --smart_turn_cpu_count 4

smart_turn_cpu_count increases inference throughput on multi-core systems without affecting latency for single concurrent streams.

Programmatic Configuration

Standard Setup via VADHandlerArguments

from speech_to_speech.VAD.vad_handler import VADHandler
from speech_to_speech.arguments_classes.vad_arguments import VADHandlerArguments
import threading

args = VADHandlerArguments(
    smart_turn=True,
    smart_turn_threshold=0.6,
    smart_turn_max_wait_ms=3000,
    smart_turn_incomplete_delay_ms=700,
    smart_turn_cpu_count=4,
)

handler = VADHandler()
handler.setup(
    should_listen=threading.Event(),
    speculative_turns=speculative_tracker,
    thresh=args.thresh,
    sample_rate=args.sample_rate,
    smart_turn=args.smart_turn,
    smart_turn_model_path=args.smart_turn_model_path,
    smart_turn_threshold=args.smart_turn_threshold,
    smart_turn_max_wait_ms=args.smart_turn_max_wait_ms,
    smart_turn_incomplete_delay_ms=args.smart_turn_incomplete_delay_ms,
    smart_turn_cpu_count=args.smart_turn_cpu_count,
)

The setup() signature mirrors lines 77-84 in vad_handler.py, ensuring all timing parameters propagate correctly.

Direct SmartTurnAnalyzer Instantiation

from speech_to_speech.VAD.smart_turn import SmartTurnAnalyzer

handler.smart_turn_analyzer = SmartTurnAnalyzer(
    model_path="/path/to/custom-smart-turn.onnx",
    threshold=0.55,
    cpu_count=2,
)

Useful for A/B testing model variants. The analyzer loads weights at initialization (lines 60-73 in smart_turn.py).

Disabling Smart Turn Entirely

handler.setup(
    should_listen=should_listen_event,
    speculative_turns=tracker,
    thresh=0.5,
    sample_rate=16000,
    smart_turn=False,  # Disables analyzer creation

)

When smart_turn=False, lines 103-108 in vad_handler.py skip analyzer instantiation and fall back to fixed speculative_reopen_ms timing.

Selecting Appropriate Threshold Values

Scenario	Recommended Threshold	Rationale
Fast-paced dialogue	0.3–0.4	Aggressive turn completion reduces perceived latency
Professional meetings	0.5–0.6	Balanced handling of pauses and interruptions
Technical support calls	0.6–0.8	Conservative completion avoids cutting off thinking time
Children's speech	0.4–0.5	Accounts for irregular pause patterns

Higher thresholds reduce premature turn-taking but may increase perceived response latency.

Custom Model Deployment

To use a fine-tuned Smart Turn model without modifying source:

1. Train or export to ONNX format compatible with the expected input tensor shape
2. Specify path via --smart_turn_model_path or VADHandlerArguments
3. Validate with SmartTurnAnalyzer.predict() on sample audiochunks before production deployment

The default v3.2 model is fetched from Hugging Face Hub at pipecat-ai/smart-turn-v3 when no custom path is provided.

Summary

• Smart Turn endpointing improves VAD accuracy by classifying turn completeness using an ONNX neural network
• Configuration happens through six parameters in VADHandlerArguments, defined in vad_arguments.py
• Enablement is controlled by the boolean smart_turn flag; all other parameters are ignored when False
• Threshold tuning (smart_turn_threshold) directly trades off latency against interruption risk
• Resource allocation (smart_turn_cpu_count) scales inference without pipeline restructuring

Frequently Asked Questions

What happens when Smart Turn predicts an incomplete turn?

The VADHandler extends the speculative reopen window to smart_turn_max_wait_ms and adds processing_delay_ms (the lesser of smart_turn_incomplete_delay_ms and smart_turn_max_wait_ms) before starting STT/LLM processing. This gives the user time to continue speaking without the assistant interrupting.

Can I use Smart Turn without an internet connection?

Yes. Download the default v3.2 model once, then reference it locally via --smart_turn_model_path /local/path/to/model.onnx. The SmartTurnAnalyzer loads entirely from local filesystem after initialization.

How does smart_turn_cpu_count affect performance?

It controls intra-op parallelism in ONNX Runtime. Values above 4 rarely benefit single-stream inference; instead, increase this parameter when running multiple concurrent VADHandler instances on the same host. The default of 1 minimizes thread contention for typical deployments.

Where is the Smart Turn model stored when auto-downloaded?

The pipecat-ai/smart-turn-v3 model is cached via the Hugging Face Hub library, typically in ~/.cache/huggingface/hub/. Set HF_HOME environment variable to relocate this cache if disk space is constrained.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}