How Omi Handles Voice Activity Detection in Real-Time Transcription Processing

TLDR: Omi processes live audio streams without local VAD by delegating voice detection to cloud STT providers (Deepgram, Soniox, Speechmatics), while batch-processing recorded files through a local Silero VAD implementation in backend/utils/stt/vad.py that splits audio into speech-only chunks before transcription.

The Omi open-source wearable AI platform (basedhardware/omi) routes audio through two distinct processing pipelines based on latency requirements. While real-time WebSocket streams rely on upstream voice detection, recorded audio undergoes explicit Voice Activity Detection (VAD) segmentation using a hybrid local and hosted model architecture. This dual approach optimizes for speed in live conversations while ensuring precision for speech-profile generation and file synchronization.

Two Processing Modes: Live Streaming vs. Batch

Live Streaming (No Local VAD)

In the real-time transcription pipeline defined in backend/routers/transcribe.py, incoming PCM audio chunks flow directly through the WebSocket to the selected STT service via the utils.stt.streaming module. Because cloud providers like Deepgram, Soniox, and Speechmatics perform internal voice-activity detection on their streaming endpoints, Omi intentionally skips local VAD processing to minimize latency.

# backend/routers/transcribe.py

await websocket.accept()
...
await process_audio_dg(ws, uid, ... )   # Deepgram handles VAD internally

Batch Processing (Explicit VAD)

For recorded *.wav files—used in speech-profile creation, conversation syncing, and post-processing—Omi applies its own VAD layer. The system invokes vad_is_empty() from backend/utils/stt/vad.py to strip silence and segment speech before sending discrete chunks to the STT provider's batch (pre-recorded) endpoint.

VAD Implementation in backend/utils/stt/vad.py

Hybrid Model Architecture with Fallback

The core VAD logic resides in vad_is_empty(), which implements a tiered detection strategy:

1. Hosted VAD: If the environment variable HOSTED_VAD_API_URL is configured, Omi submits the audio via HTTP POST to a remote VAD microservice hosting a high-quality model.
2. Local Silero Fallback: On hosted service failure or timeout, the system falls back to the Silero VAD model loaded via torch.hub.
3. Redis Caching: When cache=True, speech segments are stored in Redis with a 24-hour TTL (redis_db.set_generic_cache), eliminating redundant inference on subsequent calls.

# backend/utils/stt/vad.py

def vad_is_empty(file_path, return_segments: bool = False, cache: bool = False):
    """
    Returns True if no speech is detected, otherwise optionally returns the
    list of speech segments.
    """
    # 1. Check Redis cache

    # 2. Try hosted VAD (HTTP POST)

    # 3. Fallback to local Silero VAD

    # 4. Convert sample-index timestamps to seconds

Segment Post-Processing for Speech Profiles

The apply_vad_for_speech_profile() function handles audio cleaning for speaker-embedding extraction. It loads the entire file once using pydub, merges neighboring segments separated by less than 1 second, trims inter-segment silence while preserving a 1-second buffer, and overwrites the original WAV with the cleaned version.

def apply_vad_for_speech_profile(file_path: str):
    voice_segments = vad_is_empty(file_path, return_segments=True)
    # merge close segments (< 1s apart)

    # load audio once → AudioSegment.from_wav(file_path)

    # trim silence and export back to file_path

Pipeline Integration Points

VAD is invoked at specific architectural boundaries within the backend:

backend/routers/sync.py

The retrieve_vad_segments() function processes user-uploaded recordings. It calls vad_is_empty() with caching enabled, merges gaps smaller than 120 seconds, filters out segments shorter than 1 second, and exports each valid chunk as a separate WAV file stored in segmented_paths for Deepgram batch transcription.

def retrieve_vad_segments(path: str, segmented_paths: set, errors: list = None):
    voice_segments = vad_is_empty(path, return_segments=True, cache=True)
    # Merge gaps < 120s, keep only segments > 1s

    for i, segment in enumerate(segments):
        segment_path = f'{path_dir}/{segment_timestamp}.wav'
        segment_aseg = aseg[segment['start'] * 1000 : segment['end'] * 1000]
        segment_aseg.export(segment_path, format='wav')
        segmented_paths.add(segment_path)

backend/routers/speech_profile.py

Generates cleaned speech-profile audio via apply_vad_for_speech_profile() to ensure speaker diarization models receive only valid speech data.

backend/utils/conversations/postprocess_conversation.py

Recomputes exact segment durations after conversation storage using vad_is_empty(file_path, return_segments=True) for caching and UI display accuracy.

backend/scripts/stt/j_apply_vad_to_speech_profiles.py

Command-line utility for batch-processing existing speech-profile files through the VAD pipeline.

Error Handling and Edge Cases

When the hosted VAD service returns an error or timeout, the system logs the failure and seamlessly falls back to the local Silero model. If VAD returns an empty segment list indicating no speech detected, the API raises HTTPException(400, "Audio is empty") to prevent wasteful downstream transcription processing on silent files.

Summary

• Omi uses dual-mode VAD handling: cloud providers handle detection for live streams, while local processing manages recorded files.
• The VAD engine resides in backend/utils/stt/vad.py, implementing a hosted-first, Silero-fallback strategy with optional Redis caching.
• Batch processing merges segments separated by less than 1 second (speech profiles) or 120 seconds (sync uploads), discarding sub-second speech chunks.
• Real-time WebSocket routes in backend/routers/transcribe.py intentionally bypass local VAD to minimize latency.
• Silent files trigger an immediate 400 error before transcription resources are allocated.

Frequently Asked Questions

Does Omi perform VAD on live audio streams?

No. The live streaming WebSocket endpoint in backend/routers/transcribe.py forwards raw PCM chunks directly to STT providers like Deepgram or Speechmatics. These cloud services perform internal voice-activity detection, allowing Omi to avoid the computational overhead of local VAD during real-time transcription.

What VAD model does Omi use for recorded audio?

Omi primarily uses the Silero VAD model loaded via torch.hub as the local fallback. If the HOSTED_VAD_API_URL environment variable is configured, it attempts a remote high-quality VAD microservice first, falling back to Silero only if the hosted request fails.

How does Omi handle long pauses in uploaded recordings?

In the sync router (backend/routers/sync.py), Omi merges speech segments separated by gaps of up to 120 seconds into continuous chunks, provided the total segment length exceeds 1 second. For speech-profile generation, the threshold is stricter: gaps under 1 second are merged, and 1-second buffers are preserved between chunks.

Can VAD results be cached to improve performance?

Yes. Passing cache=True to vad_is_empty() stores the detected speech segments in Redis with a 24-hour TTL. Subsequent calls for the same file path retrieve cached results instantly, avoiding redundant model inference or HTTP requests to hosted services.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}