# How Omi Handles Voice Activity Detection in Real-Time Transcription Processing

> Discover how Omi's real-time transcription processing uses cloud STT providers for voice activity detection, optimizing live audio streams.

- Repository: [omi/omi](https://github.com/basedhardware/omi)
- Tags: deep-dive
- Published: 2026-02-26

---

**TLDR:** Omi processes live audio streams without local VAD by delegating voice detection to cloud STT providers (Deepgram, Soniox, Speechmatics), while batch-processing recorded files through a local Silero VAD implementation in [`backend/utils/stt/vad.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/stt/vad.py) that splits audio into speech-only chunks before transcription.

The Omi open-source wearable AI platform (basedhardware/omi) routes audio through two distinct processing pipelines based on latency requirements. While real-time WebSocket streams rely on upstream voice detection, recorded audio undergoes explicit **Voice Activity Detection (VAD)** segmentation using a hybrid local and hosted model architecture. This dual approach optimizes for speed in live conversations while ensuring precision for speech-profile generation and file synchronization.

## Two Processing Modes: Live Streaming vs. Batch

### Live Streaming (No Local VAD)

In the real-time transcription pipeline defined in [`backend/routers/transcribe.py`](https://github.com/basedhardware/omi/blob/main/backend/routers/transcribe.py), incoming PCM audio chunks flow directly through the WebSocket to the selected STT service via the `utils.stt.streaming` module. Because cloud providers like Deepgram, Soniox, and Speechmatics perform internal voice-activity detection on their streaming endpoints, Omi intentionally skips local VAD processing to minimize latency.

```python

# backend/routers/transcribe.py

await websocket.accept()
...
await process_audio_dg(ws, uid, ... )   # Deepgram handles VAD internally

```

### Batch Processing (Explicit VAD)

For recorded *.wav files—used in speech-profile creation, conversation syncing, and post-processing—Omi applies its own VAD layer. The system invokes `vad_is_empty()` from [`backend/utils/stt/vad.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/stt/vad.py) to strip silence and segment speech before sending discrete chunks to the STT provider's batch (pre-recorded) endpoint.

## VAD Implementation in [`backend/utils/stt/vad.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/stt/vad.py)

### Hybrid Model Architecture with Fallback

The core VAD logic resides in `vad_is_empty()`, which implements a tiered detection strategy:

1. **Hosted VAD**: If the environment variable `HOSTED_VAD_API_URL` is configured, Omi submits the audio via HTTP POST to a remote VAD microservice hosting a high-quality model.
2. **Local Silero Fallback**: On hosted service failure or timeout, the system falls back to the **Silero VAD** model loaded via `torch.hub`.
3. **Redis Caching**: When `cache=True`, speech segments are stored in Redis with a 24-hour TTL (`redis_db.set_generic_cache`), eliminating redundant inference on subsequent calls.

```python

# backend/utils/stt/vad.py

def vad_is_empty(file_path, return_segments: bool = False, cache: bool = False):
    """
    Returns True if no speech is detected, otherwise optionally returns the
    list of speech segments.
    """
    # 1. Check Redis cache

    # 2. Try hosted VAD (HTTP POST)

    # 3. Fallback to local Silero VAD

    # 4. Convert sample-index timestamps to seconds

```

### Segment Post-Processing for Speech Profiles

The `apply_vad_for_speech_profile()` function handles audio cleaning for speaker-embedding extraction. It loads the entire file once using **pydub**, merges neighboring segments separated by less than 1 second, trims inter-segment silence while preserving a 1-second buffer, and overwrites the original WAV with the cleaned version.

```python
def apply_vad_for_speech_profile(file_path: str):
    voice_segments = vad_is_empty(file_path, return_segments=True)
    # merge close segments (< 1s apart)

    # load audio once → AudioSegment.from_wav(file_path)

    # trim silence and export back to file_path

```

## Pipeline Integration Points

VAD is invoked at specific architectural boundaries within the backend:

### [`backend/routers/sync.py`](https://github.com/basedhardware/omi/blob/main/backend/routers/sync.py)

The `retrieve_vad_segments()` function processes user-uploaded recordings. It calls `vad_is_empty()` with caching enabled, merges gaps smaller than 120 seconds, filters out segments shorter than 1 second, and exports each valid chunk as a separate WAV file stored in `segmented_paths` for Deepgram batch transcription.

```python
def retrieve_vad_segments(path: str, segmented_paths: set, errors: list = None):
    voice_segments = vad_is_empty(path, return_segments=True, cache=True)
    # Merge gaps < 120s, keep only segments > 1s

    for i, segment in enumerate(segments):
        segment_path = f'{path_dir}/{segment_timestamp}.wav'
        segment_aseg = aseg[segment['start'] * 1000 : segment['end'] * 1000]
        segment_aseg.export(segment_path, format='wav')
        segmented_paths.add(segment_path)

```

### [`backend/routers/speech_profile.py`](https://github.com/basedhardware/omi/blob/main/backend/routers/speech_profile.py)

Generates cleaned speech-profile audio via `apply_vad_for_speech_profile()` to ensure speaker diarization models receive only valid speech data.

### [`backend/utils/conversations/postprocess_conversation.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/conversations/postprocess_conversation.py)

Recomputes exact segment durations after conversation storage using `vad_is_empty(file_path, return_segments=True)` for caching and UI display accuracy.

### [`backend/scripts/stt/j_apply_vad_to_speech_profiles.py`](https://github.com/basedhardware/omi/blob/main/backend/scripts/stt/j_apply_vad_to_speech_profiles.py)

Command-line utility for batch-processing existing speech-profile files through the VAD pipeline.

## Error Handling and Edge Cases

When the hosted VAD service returns an error or timeout, the system logs the failure and seamlessly falls back to the local Silero model. If VAD returns an empty segment list indicating no speech detected, the API raises `HTTPException(400, "Audio is empty")` to prevent wasteful downstream transcription processing on silent files.

## Summary

- Omi uses **dual-mode VAD handling**: cloud providers handle detection for live streams, while local processing manages recorded files.
- The VAD engine resides in [`backend/utils/stt/vad.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/stt/vad.py), implementing a **hosted-first, Silero-fallback** strategy with optional Redis caching.
- Batch processing merges segments separated by less than 1 second (speech profiles) or 120 seconds (sync uploads), discarding sub-second speech chunks.
- Real-time WebSocket routes in [`backend/routers/transcribe.py`](https://github.com/basedhardware/omi/blob/main/backend/routers/transcribe.py) intentionally bypass local VAD to minimize latency.
- Silent files trigger an immediate 400 error before transcription resources are allocated.

## Frequently Asked Questions

### Does Omi perform VAD on live audio streams?

No. The live streaming WebSocket endpoint in [`backend/routers/transcribe.py`](https://github.com/basedhardware/omi/blob/main/backend/routers/transcribe.py) forwards raw PCM chunks directly to STT providers like Deepgram or Speechmatics. These cloud services perform internal voice-activity detection, allowing Omi to avoid the computational overhead of local VAD during real-time transcription.

### What VAD model does Omi use for recorded audio?

Omi primarily uses the **Silero VAD** model loaded via `torch.hub` as the local fallback. If the `HOSTED_VAD_API_URL` environment variable is configured, it attempts a remote high-quality VAD microservice first, falling back to Silero only if the hosted request fails.

### How does Omi handle long pauses in uploaded recordings?

In the sync router ([`backend/routers/sync.py`](https://github.com/basedhardware/omi/blob/main/backend/routers/sync.py)), Omi merges speech segments separated by gaps of up to 120 seconds into continuous chunks, provided the total segment length exceeds 1 second. For speech-profile generation, the threshold is stricter: gaps under 1 second are merged, and 1-second buffers are preserved between chunks.

### Can VAD results be cached to improve performance?

Yes. Passing `cache=True` to `vad_is_empty()` stores the detected speech segments in Redis with a 24-hour TTL. Subsequent calls for the same file path retrieve cached results instantly, avoiding redundant model inference or HTTP requests to hosted services.