# How the VAD Component Works in Hugging Face Speech-to-Speech: A Deep Dive into Voice Activity Detection

> Learn how the VAD component in Hugging Face Speech-to-Speech detects speech and converts audio. Understand voice activity detection for improved processing.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: deep-dive
- Published: 2026-08-11

---

**The VAD subsystem detects when a user starts and stops speaking, converting raw microphone samples into turn-level audio chunks for downstream speech-to-text, LLM, and text-to-speech processing.**

Voice Activity Detection (VAD) is the critical front-end component in the `huggingface/speech-to-speech` pipeline. It determines speech boundaries in real-time streaming audio, enabling natural conversational turn-taking. This article explains how the VAD component works by examining its three-layer architecture: the low-level `VADIterator`, the orchestrating `VADHandler`, and the optional `SmartTurnAnalyzer`.

## VADIterator: Streaming Speech Detection with Silero VAD

The `VADIterator` class in [`src/speech_to_speech/VAD/vad_iterator.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_iterator.py) provides the foundation for speech detection using the **Silero VAD model** loaded via `torch.hub`.

### Model Loading and Chunk Processing

The iterator loads `silero_vad` and processes audio as `torch.Tensor` chunks. It maintains a `_pre_speech_buffer` to capture audio occurring before the speech threshold is crossed, controlled by the `speech_pad_ms` parameter.

### State Machine Operation

The iterator implements a two-state machine:

- **Idle state** – Fills the pre-speech buffer until the model's speech probability exceeds `threshold`
- **Speaking state** – Accumulates chunks in `buffer`; when probability drops below `threshold - 0.15`, a temporary end is marked

Speech segments close when silence persists longer than `min_silence_duration_ms`. The method returns `None` while speaking, then a list of `torch.Tensor` objects representing the complete utterance.

### Key Configuration Parameters

| Parameter | Description | Default |
|-----------|-------------|---------|
| `threshold` | Speech probability trigger | 0.6 |
| `min_silence_duration_ms` | Minimum silence to end segment | 500 |
| `speech_pad_ms` | Audio to prepend before trigger | 100 |
| `sampling_rate` | Audio sample rate (Hz) | 16000 |

## VADHandler: Turn Management and Pipeline Integration

`VADHandler` in [`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py) extends `BaseHandler` to integrate the iterator into the full pipeline. It manages listening state, audio conversion, and sophisticated turn-handling logic.

### Audio Conversion and Invocation

Raw PCM `bytes` are converted through `np.int16` to float32 via `int2float`, then passed to the iterator as `torch.from_numpy(audio_float32)`.

### Speech-Start Detection

When `iterator.triggered` becomes `True`, the handler verifies that **active speech duration** exceeds `min_speech_ms` (or `min_speech_continuation_ms` for reopened turns). Valid starts emit a `SpeechStartedEvent` and allocate a turn identifier with revision tracking.

### Progressive Streaming During Speech

While speech continues, the handler yields `VADAudio` chunks with `mode="progressive"` based on a dynamic `_progressive_processing_pause`. This allows downstream components to begin partial processing before the turn completes.

### Speech-End Handling and Segment Management

Upon receiving a non-empty list from the iterator, the handler applies three strategies:

- **Merge** – Combines previously held short segments with the current one
- **Discard** – Removes noise-only fragments below duration thresholds
- **Hold** – Retains short segments (`_SHORT_SEGMENT_MIN_FRAGMENT_MS`, `short_segment_merge_ms`) for potential future stitching

### Speculative Turn Reopening

The handler coordinates with `SpeculativeTurnTracker` to support rapid turn resumption—when users pause briefly then continue (e.g., "…and also…"). This prevents premature assistant responses.

### Smart Turn Classification

When enabled, `SmartTurnAnalyzer` runs on the final segment to determine turn completeness.

### Optional Audio Enhancement

If `audio_enhancement=True` and `deepfilternet` is available, the final segment passes through `enhance()` for noise suppression.

### Event Emission and State Reset

The handler emits `SpeechStartedEvent` and `SpeechStoppedEvent` with timestamps, turn metadata, and processing delays. `on_session_end()` clears all buffers, counters, and speculative state.

## SmartTurnAnalyzer: ONNX-Based Turn Completion Classification

The `SmartTurnAnalyzer` in [`src/speech_to_speech/VAD/smart_turn.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/smart_turn.py) provides intelligent turn-boundary detection beyond simple silence thresholds.

### Model Architecture

- **ONNX runtime** – Loads `smart-turn-v3.2-cpu.onnx`
- **Feature extraction** – Uses Whisper-compatible feature extraction for consistent audio front-end processing
- **Inference scope** – Processes up to 8 seconds of final utterance audio

### Decision Logic

Returns `SmartTurnResult` containing:

- `complete` – Boolean indicating whether the turn is finished
- `probability` – Confidence score
- `inference_ms` – Processing latency

If `complete` is `False`, the handler can delay the assistant response up to `smart_turn_max_wait_ms` and inject `smart_turn_incomplete_delay_ms` before processing continues.

## Configuration Interface

All VAD parameters are exposed through `VADHandlerArguments` in [`src/speech_to_speech/arguments_classes/vad_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/vad_arguments.py). These can be set via CLI flags, configuration files, or `RuntimeConfig` objects from the OpenAI Realtime API.

## Code Examples

### Using VADIterator Directly

```python
import torch
from speech_to_speech.VAD.vad_iterator import VADIterator
import torchaudio

# Load Silero VAD model

model, _ = torch.hub.load(
    "snakers4/silero-vad:master", "silero_vad", trust_repo=True, skip_validation=True
)

iterator = VADIterator(model, threshold=0.5, sampling_rate=16000)

# Process streaming 16 kHz mono PCM

for wav_chunk in torchaudio.load("sample.wav")[0].split(1600):  # 100 ms chunks

    out = iterator(wav_chunk)
    if out is not None:  # speech segment finished

        utterance = torch.cat(out).numpy()
        print(f"Detected utterance: {len(utterance)} samples")

```

### Integrating VADHandler in a Pipeline

```python
from speech_to_speech.VAD.vad_handler import VADHandler
from speech_to_speech.pipeline.messages import VADAudio
from threading import Event

should_listen = Event()
should_listen.set()

handler = VADHandler()
handler.setup(
    should_listen=should_listen,
    speculative_turns=SpeculativeTurnTracker(),
    thresh=0.6,
    sample_rate=16000,
    min_silence_ms=300,
    min_speech_ms=384,
)

# Process raw PCM from microphone or WebSocket

for out in handler.process(audio_bytes):
    if isinstance(out, VADAudio) and out.mode == "final":
        transcription = stt_service.transcribe(out.audio, sample_rate=16000)

```

### Enabling Smart Turn and Audio Enhancement

```python
handler.setup(
    should_listen=should_listen,
    speculative_turns=SpeculativeTurnTracker(),
    thresh=0.55,
    smart_turn=True,
    smart_turn_threshold=0.7,
    smart_turn_max_wait_ms=1500,
    audio_enhancement=True,  # requires deepfilternet

)

```

## Key Source Files

| File | Purpose |
|------|---------|
| [`src/speech_to_speech/VAD/vad_iterator.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_iterator.py) | Low-level Silero VAD streaming and speech boundary detection |
| [`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py) | High-level handler with turn management, progressive streaming, and smart-turn integration |
| [`src/speech_to_speech/VAD/smart_turn.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/smart_turn.py) | ONNX classifier for turn completion detection |
| [`src/speech_to_speech/arguments_classes/vad_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/vad_arguments.py) | Configuration dataclass for all VAD parameters |
| [`src/speech_to_speech/pipeline/events.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/events.py) | Event definitions (`SpeechStartedEvent`, `SpeechStoppedEvent`) |
| [`src/speech_to_speech/pipeline/messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py) | `VADAudio` message type for audio chunks with mode and metadata |

## Summary

- **VADIterator** provides lightweight, streaming speech detection using the Silero VAD model with configurable thresholds and padding
- **VADHandler** orchestrates the full turn-management pipeline, including progressive streaming, speculative reopening, short-segment handling, and optional audio enhancement
- **SmartTurnAnalyzer** adds intelligent turn-completion classification using an ONNX model to reduce premature responses
- All components are configurable through `VADHandlerArguments` and support both standalone and Realtime API integration

## Frequently Asked Questions

### What VAD model does Speech-to-Speech use?

The pipeline uses **Silero VAD**, loaded via `torch.hub` from the `snakers4/silero-vad` repository. This lightweight PyTorch model runs efficiently on CPU for real-time streaming applications.

### How does the pipeline handle brief pauses within a sentence?

The **speculative turn tracker** and `min_silence_duration_ms` parameter work together. Short pauses below the threshold keep the turn open; the `SmartTurnAnalyzer` can optionally verify that the utterance is incomplete and extend the waiting period.

### Can I adjust how much audio is captured before speech starts?

Yes. The `speech_pad_ms` parameter controls how much pre-trigger audio is prepended to each segment. The `VADIterator` maintains this in `_pre_speech_buffer` and includes it in the returned utterance.

### Is audio enhancement available for noisy environments?

Yes. Setting `audio_enhancement=True` enables **DeepFilterNet** noise suppression on final speech segments, provided the `deepfilternet` package is installed in your environment.