How Omi Processes Conversation Transcripts with Whisper and Other STT Engines

Omi processes conversation transcripts by first attempting transcription with Deepgram, falling back to Fal AI Whisper if the output is incomplete, then normalizing the word-level results into structured TranscriptSegment objects through a dedicated post-processing pipeline.

When a user completes a voice interaction in the Omi open-source wearable (basedhardware/omi), the raw audio enters a sophisticated backend pipeline designed to handle multiple speech-to-text (STT) backends. This system converts audio recordings into diarized, timestamped conversation transcripts that power the application's memory and analysis features.

Entry Point: The postprocess_conversation Orchestrator

The transcription workflow begins in backend/utils/conversations/postprocess_conversation.py within the postprocess_conversation function. This orchestrator manages the entire lifecycle from audio storage to final persistence.

First, the function uploads the audio file to Cloud Storage and generates a signed URL. It then invokes the primary STT provider:

# backend/utils/conversations/postprocess_conversation.py

words = deepgram_prerecorded(signed_url, speakers_count=speakers_count)
fal_segments = postprocess_words(words, aseg.duration_seconds)

If the Deepgram result is dramatically shorter than expected—indicating a potential transcription failure—the pipeline automatically triggers a fallback to Fal AI Whisper (fal_whisperx) and re-runs the post-processing steps on the new output.

Primary STT Backend: Deepgram Pre-recorded

The default transcription service is implemented in backend/utils/stt/pre_recorded.py via the deepgram_prerecorded function. This wrapper interacts with the Deepgram cloud API using the nova-3 model (or nova-2-general in some configurations) and provides built-in speaker diarization.

# backend/utils/stt/pre_recorded.py

def deepgram_prerecorded(
    audio_url: str,
    speakers_count: int = None,
    attempts: int = 0,
    return_language: bool = False,
    diarize: bool = True,
    language: Optional[str] = None,
    model: str = "nova-3",
) -> Union[List[dict], Tuple[List[dict], str]]:
    response = _deepgram_client.listen.rest.v("1").transcribe_url(
        {"url": audio_url}, options
    )
    # Convert Deepgram → Fal-WhisperX compatible format

    for w in dg_words:
        words.append({
            "timestamp": [w["start"], w["end"]],
            "speaker": f"SPEAKER_{w.get('speaker', 0):02d}",
            "text": w.get("punctuated_word", w["word"]),
        })

Key capabilities include:

• Language detection: Passes a language parameter or uses detect_language for automatic detection; returns the detected language when return_language=True
• Speaker diarization: Deepgram's built-in diarization produces numeric speaker IDs that the code converts to the SPEAKER_XX string format used throughout Omi
• Retry logic: Implements up to three attempts (attempts < 2) for resilience
• Output normalization: Returns a list of dictionaries matching the shape expected by fal_whisperx, ensuring downstream compatibility

Fal AI Whisper Fallback Processing

When Deepgram fails or returns insufficient text, the pipeline activates the fal_whisperx function from the same pre_recorded.py module. This alternative uses Fal AI's hosted Whisper implementation (version 3) to generate transcripts.

# backend/utils/stt/pre_recorded.py

def fal_whisperx(
    audio_url: str,
    speakers_count: int = None,
    attempts: int = 0,
    return_language: bool = False,
    diarize: bool = True,
    chunk_level: str = 'word',
) -> List[dict]:
    handler = fal_client.submit(
        "fal-ai/whisper",
        arguments={
            "audio_url": audio_url,
            "task": "transcribe",
            "diarize": diarize,
            "chunk_level": chunk_level,
            "version": "3",
            "batch_size": 64,
            "num_speakers": speakers_count,
        },
    )
    result = handler.get()
    words = result.get('chunks', [])

Deepgram vs. Fal AI Whisper:

• Provider: Deepgram uses proprietary cloud APIs while Fal AI hosts open-source Whisper models
• Model versions: Deepgram uses nova-3; Fal AI uses fal-ai/whisper v3
• Speaker format: Deepgram returns numeric IDs requiring conversion to SPEAKER_XX strings, whereas Fal AI returns the formatted speaker labels directly when diarize=True
• Language info: Deepgram requires explicit flags for language detection; Fal AI returns inferred languages in the inferred_languages field

Both functions return identical word-list structures, allowing the rest of the pipeline to remain agnostic to the transcription source.

Normalizing Output with postprocess_words

Regardless of which STT engine generates the raw data, the postprocess_words function transforms the word-level output into coherent conversation segments. This critical normalization step resides in backend/utils/stt/pre_recorded.py.

# backend/utils/stt/pre_recorded.py

def postprocess_words(words: List[dict], duration: int, skip_n_seconds: int = 0) -> List[TranscriptSegment]:
    words = _words_cleaning(words)                      # normalize timestamps & fill gaps

    user_speaker_id = _retrieve_user_speaker_id(words, skip_n_seconds)
    segments = _merge_segments(words, skip_n_seconds, user_speaker_id)
    return _segments_as_objects(segments)              # → List[TranscriptSegment]

The process follows four distinct stages:

1. _words_cleaning: Rounds timestamps, guarantees every word has a speaker label, and initializes the is_user flag to False
2. _retrieve_user_speaker_id: Analyzes the first skip_n_seconds of audio to determine which speaker corresponds to the device owner
3. _merge_segments: Collapses consecutive words spoken by the same speaker into continuous segments, respecting a maximum 30-second gap threshold
4. _segments_as_objects: Instantiates Pydantic models defined in backend/models/transcript_segment.py, creating the final TranscriptSegment objects used by the application

Speaker Profile Matching for User Identification

After generic STT processing completes, the pipeline optionally enhances transcript accuracy through speaker identification. This step only executes when the audio sampling rate is 16 kHz:

# backend/utils/conversations/postprocess_conversation.py

if aseg.frame_rate == 16000:
    matches = get_speech_profile_matching_predictions(
        uid, file_path, [s.dict() for s in segments]
    )
    for i, segment in enumerate(segments):
        segment.is_user = matches[i]['is_user']
        segment.person_id = matches[i].get('person_id')

This speech-profile matching compares segment embeddings against the user's voice profile, updating the is_user boolean and assigning person_id values to distinguish between multiple speakers in the conversation.

Persisting Processed Conversation Transcripts

Finally, the completed segments are stored in both Firestore and Redis for retrieval by the mobile and web clients:

# backend/utils/conversations/postprocess_conversation.py

conversations_db.store_model_segments_result(
    uid, conversation.id, streaming_model, conversation.transcript_segments
)
conversations_db.store_model_segments_result(
    uid, conversation.id, 'fal_whisperx', fal_segments
)

The system retains results from both the original streaming model (e.g., Deepgram live streaming used during recording) and the post-processed model, enabling comparison, debugging, and quality assurance.

Summary

• Dual-engine reliability: Omi uses Deepgram as the primary STT provider with Fal AI Whisper as an automatic fallback when transcripts are incomplete
• Unified data model: Both deepgram_prerecorded and fal_whisperx output standardized word lists that postprocess_words converts into TranscriptSegment objects
• Speaker intelligence: The pipeline includes diarization, user identification through speech profiles (at 16 kHz), and gap-aware segment merging
• Implementation locations: Core logic resides in backend/utils/stt/pre_recorded.py and backend/utils/conversations/postprocess_conversation.py

Frequently Asked Questions

What STT engines does Omi use for processing conversation transcripts?

Omi primarily uses Deepgram (specifically the nova-3 model) for transcription, with Fal AI Whisper (version 3) serving as a fallback mechanism. The system attempts Deepgram first and automatically switches to Fal AI Whisper if the initial transcript is significantly shorter than expected, ensuring robust coverage even when one service encounters issues.

How does Omi handle speaker diarization in conversation transcripts?

Both STT engines support speaker diarization. Deepgram returns numeric speaker IDs that Omi converts to SPEAKER_XX format, while Fal AI Whisper returns pre-formatted speaker labels when diarize=True. After initial transcription, the postprocess_words function merges consecutive words from the same speaker into continuous segments and uses speech-profile matching (at 16 kHz sample rates) to identify which speaker is the device owner.

When does Omi fall back to Whisper instead of using Deepgram?

The fallback triggers in postprocess_conversation when the Deepgram result is "dramatically shorter than the original transcript." This heuristic indicates potential transcription failure or audio quality issues, prompting the system to re-process the audio using fal_whisperx and replace the insufficient Deepgram output with the Whisper-generated segments.

What format does Omi use for storing transcript segments?

Omi stores transcripts as TranscriptSegment objects defined in backend/models/transcript_segment.py. These Pydantic models contain timestamps, speaker identifiers, transcribed text, is_user boolean flags, and optional person_id fields. The segments persist in both Firestore and Redis, with separate storage for streaming model results and post-processed Fal WhisperX results to maintain audit trails.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}