# How Omi Processes Conversation Transcripts with Whisper and Other STT Engines

> Discover how Omi processes conversation transcripts using Whisper and other STT engines. Learn about its fallback and normalization pipeline.

- Repository: [omi/omi](https://github.com/basedhardware/omi)
- Tags: deep-dive
- Published: 2026-02-26

---

**Omi processes conversation transcripts by first attempting transcription with Deepgram, falling back to Fal AI Whisper if the output is incomplete, then normalizing the word-level results into structured TranscriptSegment objects through a dedicated post-processing pipeline.**

When a user completes a voice interaction in the Omi open-source wearable (basedhardware/omi), the raw audio enters a sophisticated backend pipeline designed to handle multiple speech-to-text (STT) backends. This system converts audio recordings into diarized, timestamped conversation transcripts that power the application's memory and analysis features.

## Entry Point: The postprocess_conversation Orchestrator

The transcription workflow begins in [`backend/utils/conversations/postprocess_conversation.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/conversations/postprocess_conversation.py) within the `postprocess_conversation` function. This orchestrator manages the entire lifecycle from audio storage to final persistence.

First, the function uploads the audio file to Cloud Storage and generates a signed URL. It then invokes the primary STT provider:

```python

# backend/utils/conversations/postprocess_conversation.py

words = deepgram_prerecorded(signed_url, speakers_count=speakers_count)
fal_segments = postprocess_words(words, aseg.duration_seconds)

```

If the Deepgram result is dramatically shorter than expected—indicating a potential transcription failure—the pipeline automatically triggers a fallback to **Fal AI Whisper** (`fal_whisperx`) and re-runs the post-processing steps on the new output.

## Primary STT Backend: Deepgram Pre-recorded

The default transcription service is implemented in [`backend/utils/stt/pre_recorded.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/stt/pre_recorded.py) via the `deepgram_prerecorded` function. This wrapper interacts with the Deepgram cloud API using the `nova-3` model (or `nova-2-general` in some configurations) and provides built-in speaker diarization.

```python

# backend/utils/stt/pre_recorded.py

def deepgram_prerecorded(
    audio_url: str,
    speakers_count: int = None,
    attempts: int = 0,
    return_language: bool = False,
    diarize: bool = True,
    language: Optional[str] = None,
    model: str = "nova-3",
) -> Union[List[dict], Tuple[List[dict], str]]:
    response = _deepgram_client.listen.rest.v("1").transcribe_url(
        {"url": audio_url}, options
    )
    # Convert Deepgram → Fal-WhisperX compatible format

    for w in dg_words:
        words.append({
            "timestamp": [w["start"], w["end"]],
            "speaker": f"SPEAKER_{w.get('speaker', 0):02d}",
            "text": w.get("punctuated_word", w["word"]),
        })

```

Key capabilities include:

- **Language detection**: Passes a `language` parameter or uses `detect_language` for automatic detection; returns the detected language when `return_language=True`
- **Speaker diarization**: Deepgram's built-in diarization produces numeric speaker IDs that the code converts to the `SPEAKER_XX` string format used throughout Omi
- **Retry logic**: Implements up to three attempts (`attempts < 2`) for resilience
- **Output normalization**: Returns a list of dictionaries matching the shape expected by `fal_whisperx`, ensuring downstream compatibility

## Fal AI Whisper Fallback Processing

When Deepgram fails or returns insufficient text, the pipeline activates the `fal_whisperx` function from the same [`pre_recorded.py`](https://github.com/basedhardware/omi/blob/main/pre_recorded.py) module. This alternative uses Fal AI's hosted Whisper implementation (version 3) to generate transcripts.

```python

# backend/utils/stt/pre_recorded.py

def fal_whisperx(
    audio_url: str,
    speakers_count: int = None,
    attempts: int = 0,
    return_language: bool = False,
    diarize: bool = True,
    chunk_level: str = 'word',
) -> List[dict]:
    handler = fal_client.submit(
        "fal-ai/whisper",
        arguments={
            "audio_url": audio_url,
            "task": "transcribe",
            "diarize": diarize,
            "chunk_level": chunk_level,
            "version": "3",
            "batch_size": 64,
            "num_speakers": speakers_count,
        },
    )
    result = handler.get()
    words = result.get('chunks', [])

```

**Deepgram vs. Fal AI Whisper**:

- **Provider**: Deepgram uses proprietary cloud APIs while Fal AI hosts open-source Whisper models
- **Model versions**: Deepgram uses `nova-3`; Fal AI uses `fal-ai/whisper` v3
- **Speaker format**: Deepgram returns numeric IDs requiring conversion to `SPEAKER_XX` strings, whereas Fal AI returns the formatted speaker labels directly when `diarize=True`
- **Language info**: Deepgram requires explicit flags for language detection; Fal AI returns inferred languages in the `inferred_languages` field

Both functions return identical word-list structures, allowing the rest of the pipeline to remain agnostic to the transcription source.

## Normalizing Output with postprocess_words

Regardless of which STT engine generates the raw data, the `postprocess_words` function transforms the word-level output into coherent conversation segments. This critical normalization step resides in [`backend/utils/stt/pre_recorded.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/stt/pre_recorded.py).

```python

# backend/utils/stt/pre_recorded.py

def postprocess_words(words: List[dict], duration: int, skip_n_seconds: int = 0) -> List[TranscriptSegment]:
    words = _words_cleaning(words)                      # normalize timestamps & fill gaps

    user_speaker_id = _retrieve_user_speaker_id(words, skip_n_seconds)
    segments = _merge_segments(words, skip_n_seconds, user_speaker_id)
    return _segments_as_objects(segments)              # → List[TranscriptSegment]

```

The process follows four distinct stages:

1. **`_words_cleaning`**: Rounds timestamps, guarantees every word has a speaker label, and initializes the `is_user` flag to `False`
2. **`_retrieve_user_speaker_id`**: Analyzes the first *skip_n_seconds* of audio to determine which speaker corresponds to the device owner
3. **`_merge_segments`**: Collapses consecutive words spoken by the same speaker into continuous segments, respecting a maximum 30-second gap threshold
4. **`_segments_as_objects`**: Instantiates Pydantic models defined in [`backend/models/transcript_segment.py`](https://github.com/basedhardware/omi/blob/main/backend/models/transcript_segment.py), creating the final `TranscriptSegment` objects used by the application

## Speaker Profile Matching for User Identification

After generic STT processing completes, the pipeline optionally enhances transcript accuracy through speaker identification. This step only executes when the audio sampling rate is 16 kHz:

```python

# backend/utils/conversations/postprocess_conversation.py

if aseg.frame_rate == 16000:
    matches = get_speech_profile_matching_predictions(
        uid, file_path, [s.dict() for s in segments]
    )
    for i, segment in enumerate(segments):
        segment.is_user = matches[i]['is_user']
        segment.person_id = matches[i].get('person_id')

```

This speech-profile matching compares segment embeddings against the user's voice profile, updating the `is_user` boolean and assigning `person_id` values to distinguish between multiple speakers in the conversation.

## Persisting Processed Conversation Transcripts

Finally, the completed segments are stored in both Firestore and Redis for retrieval by the mobile and web clients:

```python

# backend/utils/conversations/postprocess_conversation.py

conversations_db.store_model_segments_result(
    uid, conversation.id, streaming_model, conversation.transcript_segments
)
conversations_db.store_model_segments_result(
    uid, conversation.id, 'fal_whisperx', fal_segments
)

```

The system retains results from both the original streaming model (e.g., Deepgram live streaming used during recording) and the post-processed model, enabling comparison, debugging, and quality assurance.

## Summary

- **Dual-engine reliability**: Omi uses Deepgram as the primary STT provider with Fal AI Whisper as an automatic fallback when transcripts are incomplete
- **Unified data model**: Both `deepgram_prerecorded` and `fal_whisperx` output standardized word lists that `postprocess_words` converts into `TranscriptSegment` objects
- **Speaker intelligence**: The pipeline includes diarization, user identification through speech profiles (at 16 kHz), and gap-aware segment merging
- **Implementation locations**: Core logic resides in [`backend/utils/stt/pre_recorded.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/stt/pre_recorded.py) and [`backend/utils/conversations/postprocess_conversation.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/conversations/postprocess_conversation.py)

## Frequently Asked Questions

### What STT engines does Omi use for processing conversation transcripts?

Omi primarily uses **Deepgram** (specifically the `nova-3` model) for transcription, with **Fal AI Whisper** (version 3) serving as a fallback mechanism. The system attempts Deepgram first and automatically switches to Fal AI Whisper if the initial transcript is significantly shorter than expected, ensuring robust coverage even when one service encounters issues.

### How does Omi handle speaker diarization in conversation transcripts?

Both STT engines support speaker diarization. Deepgram returns numeric speaker IDs that Omi converts to `SPEAKER_XX` format, while Fal AI Whisper returns pre-formatted speaker labels when `diarize=True`. After initial transcription, the `postprocess_words` function merges consecutive words from the same speaker into continuous segments and uses speech-profile matching (at 16 kHz sample rates) to identify which speaker is the device owner.

### When does Omi fall back to Whisper instead of using Deepgram?

The fallback triggers in `postprocess_conversation` when the Deepgram result is "dramatically shorter than the original transcript." This heuristic indicates potential transcription failure or audio quality issues, prompting the system to re-process the audio using `fal_whisperx` and replace the insufficient Deepgram output with the Whisper-generated segments.

### What format does Omi use for storing transcript segments?

Omi stores transcripts as **TranscriptSegment** objects defined in [`backend/models/transcript_segment.py`](https://github.com/basedhardware/omi/blob/main/backend/models/transcript_segment.py). These Pydantic models contain timestamps, speaker identifiers, transcribed text, `is_user` boolean flags, and optional `person_id` fields. The segments persist in both Firestore and Redis, with separate storage for streaming model results and post-processed Fal WhisperX results to maintain audit trails.