# Configuring Smart Turn for End-of-Speech Detection Accuracy in Speech-to-Speech Systems

> Improve speech-to-speech accuracy using Smart Turn for end-of-speech detection. Learn how this ONNX classifier enhances turn completion with acoustic and linguistic analysis.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: tutorial
- Published: 2026-08-09

---

**Smart Turn adds a secondary turn-completion check using an ONNX classifier that analyzes up to 8 seconds of acoustic and linguistic content to confirm whether a user has finished speaking after the primary VAD detects silence.**

Configuring Smart Turn correctly is essential for balancing responsiveness against false cutoffs in real-time speech-to-speech pipelines. In the `huggingface/speech-to-speech` repository, this component operates as a second-level validator on top of the Silero VAD, applying a lightweight neural model to determine final turn boundaries. Understanding how to tune its parameters allows you to optimize end-of-speech detection accuracy for your specific hardware and latency requirements.

## Understanding Smart Turn Architecture

The Smart Turn system consists of several coordinated components that process audio after the primary voice activity detector signals a potential speech-to-silence boundary.

### SmartTurnAnalyzer and ONNX Model

The **`SmartTurnAnalyzer`** class (defined in [`src/speech_to_speech/VAD/smart_turn.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/smart_turn.py)) loads and executes an ONNX model hosted in the Hugging Face repository `pipecat-ai/smart-turn-v3`. The specific model file is `smart-turn-v3.2-cpu.onnx`, loaded using ONNX Runtime with configurable CPU threading options via `SessionOptions`.

The analyzer returns a **`SmartTurnResult`** dataclass containing three critical fields:
- `complete` (bool): Whether the turn is finalized
- `probability` (float): Confidence score between 0 and 1
- `inference_ms` (float): Latency measurement for the prediction

### Audio Pre-processing Pipeline

Before inference, the `_prepare_audio` method processes raw waveforms to meet the model's strict input requirements. The pipeline resamples any input sample rate to 16 kHz using **SciPy's** `resample_poly`, then pads or truncates the audio to exactly 8 seconds (`MAX_AUDIO_SECONDS * MODEL_SAMPLE_RATE` samples).

The normalized mono float32 array feeds into the **WhisperFeatureExtractor** from 🤗 Transformers, ensuring the feature space remains consistent with the Whisper STT model used elsewhere in the pipeline.

### Integration with VAD Handler

The **`SmartTurnHandler`** in [`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py) orchestrates the detection flow. When the primary VAD detects silence, the handler invokes `self.smart_turn_analyzer.predict(audio, sample_rate=16000)`. If `SmartTurnResult.complete` evaluates to `True`, the pipeline treats the segment as final; otherwise, it continues listening in speculative turn mode.

## Configuration Parameters for Accuracy Tuning

Fine-tuning Smart Turn requires adjusting several initialization parameters exposed through `SmartTurnAnalyzer.__init__`.

### Adjusting the Confidence Threshold

The **`threshold`** parameter (default `0.5`) determines the cutoff for `probability > threshold` when deciding turn completion. Lower values (e.g., `0.3`) create aggressive detection with earlier turn ends but risk false positives, while higher values (e.g., `0.7`) demand stronger acoustic and linguistic evidence before concluding a turn, reducing premature cutoffs at the cost of increased latency.

### Optimizing CPU Parallelism

The **`cpu_count`** parameter controls intra-op threads for ONNX Runtime. For edge devices requiring low latency, set this to `1` or `2` to prevent thread contention. Server-class CPUs can utilize `4` threads to reduce inference time when processing multiple concurrent streams.

### Model Loading and Warm-up Options

Setting **`warmup=False`** skips the initial dummy inference, useful when you need the first prediction to represent true cold-start performance. The **`model_path`** parameter accepts local file paths (e.g., `"/path/to/custom/smart-turn.onnx"`) to bypass automatic downloading from the Hugging Face Hub.

## Implementation Examples

### Stand-alone Analyzer Usage

```python
import numpy as np
from speech_to_speech.VAD.smart_turn import SmartTurnAnalyzer

# Initialise with aggressive threshold for earlier detection

analyzer = SmartTurnAnalyzer(threshold=0.35, cpu_count=2)

# Simulated 2-second mono audio at 16 kHz

audio = np.random.randn(2 * 16000).astype(np.float32)

result = analyzer.predict(audio, sample_rate=16000)
print(f"Complete? {result.complete} – prob={result.probability:.3f} (took {result.inference_ms:.1f} ms)")

```

### Custom VAD Handler Integration

```python
from speech_to_speech.VAD.vad_handler import VADHandler
from speech_to_speech.VAD.smart_turn import SmartTurnAnalyzer

# Create analyzer with conservative settings

custom_analyzer = SmartTurnAnalyzer(threshold=0.6, cpu_count=4, warmup=False)

# Inject into handler

handler = VADHandler(...)
handler.smart_turn_analyzer = custom_analyzer

```

### Loading Local Model Files

```python
analyzer = SmartTurnAnalyzer(
    model_path="/path/to/local/smart-turn-v3.2-cpu.onnx",
    threshold=0.5,
    cpu_count=1,
)

```

### Full Pipeline Configuration

```python
from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline

pipeline = SpeechToSpeechPipeline(
    vad_smart_turn_threshold=0.45,
    vad_cpu_count=2,
)

pipeline.run(audio_source="microphone")

```

## Summary

- **Smart Turn** acts as a secondary validator after Silero VAD, using an ONNX model to analyze 8-second audio windows for turn completion.
- The **`threshold`** parameter directly controls the trade-off between false positives (early cutoff) and false negatives (late response).
- **`cpu_count`** should be set to `1-2` for low-latency edge devices and up to `4` for server deployments.
- Audio preprocessing occurs in `SmartTurnAnalyzer._prepare_audio`, utilizing SciPy resampling and Whisper feature extraction.
- Configuration happens through `SmartTurnAnalyzer.__init__` or pipeline parameters in `SpeechToSpeechPipeline`.

## Frequently Asked Questions

### How does Smart Turn differ from the primary VAD?

The primary Silero VAD detects speech-versus-silence boundaries based on acoustic energy alone. Smart Turn adds a linguistic and acoustic understanding layer by running a classifier on the actual content of the utterance to determine if the user has semantically completed their thought, reducing false turn endings on mid-sentence pauses.

### What is the optimal threshold setting for real-time applications?

For real-time conversational AI, start with the default `0.5` and adjust based on user feedback. Use `0.3-0.4` for faster response times when users speak in short commands, or `0.6-0.7` for formal dialogues where interrupting mid-thought is costly. Monitor the `inference_ms` field to ensure threshold adjustments don't coincide with performance degradation.

### Can Smart Turn run on GPU instead of CPU?

The current implementation in [`smart_turn.py`](https://github.com/huggingface/speech-to-speech/blob/main/smart_turn.py) uses ONNX Runtime with CPU-specific `SessionOptions` and single-threaded configurations. While the model architecture supports GPU execution via ONNX Runtime's CUDA provider, the repository's `SmartTurnAnalyzer` is optimized for CPU inference to maintain compatibility with edge deployment scenarios.

### Why does the analyzer require exactly 8 seconds of audio?

The ONNX model was trained on fixed-length 8-second segments (16 kHz sample rate) to standardize the input tensor shape. The `_prepare_audio` method enforces this by truncating longer utterances or zero-padding shorter ones, ensuring the Whisper feature extractor outputs consistent dimensions regardless of actual speech duration.