How Speech Profile Creation and Recognition Works in Omi: A Complete Technical Guide

Omi creates a per-user speech profile by storing a 30-second voice sample in Google Cloud Storage, training a Soniox speaker model, and prepending this audio to live transcription streams so the STT service can identify the user's voice; after transcription, a hosted Speech-Profile API matches each segment to the user.

Omi is an open-source AI wearable that relies on speech profile creation and recognition to distinguish the device owner from other speakers in multi-party conversations. The system combines cloud storage, speaker diarization, and real-time audio streaming to build a voice fingerprint and apply it during live transcription sessions.

Creating a Speech Profile in Omi

Uploading and Storing Profile Audio

The process begins when a user records approximately 30 seconds of speech through the client application. The backend receives this WAV file and immediately uploads it to a private Google Cloud Storage bucket using upload_profile_audio() in utils/other/storage.py.

from utils.other.storage import upload_profile_audio

def create_profile(uid: str, wav_path: str) -> str:
    # Stores file at <uid>/speech_profile.wav in GCS

    public_url = upload_profile_audio(wav_path, uid)
    return public_url

The file is stored under the path <uid>/speech_profile.wav within the bucket defined by BUCKET_SPEECH_PROFILES. This centralized storage ensures the profile audio is accessible to any backend instance handling the user's transcription sessions.

Training the Speaker Model with Soniox

After successful storage, the system registers the user with Soniox, a speech-to-text service that supports speaker identification. The create_user_speech_profile() function in utils/stt/soniox_util.py orchestrates this by first removing any existing speaker entry, then creating a fresh speaker via manage_speakers --add_speaker, and finally adding the uploaded audio via manage_speakers --add_audio.

from utils.stt.soniox_util import create_user_speech_profile

def train_profile(uid: str) -> bool:
    # Returns True on successful training

    return create_user_speech_profile(uid)

Upon successful training, the system sets a Redis flag user_has_soniox_speech_profile to enable fast lookups during subsequent transcription sessions. This flag prevents unnecessary database queries and speeds up session initialization.

Detecting Speech Profiles During Live Transcription

Before establishing a WebSocket connection at /transcribe, the router checks whether the user has a valid speech profile using get_user_has_speech_profile() in utils/other/storage.py. This function verifies the Redis flag and can optionally check the profile's age to ensure freshness.

If the flag is True, the router reserves SPEECH_PROFILE_FIXED_DURATION (30 seconds) plus padding time in the audio buffer. It then initializes a secondary "profile" STT socket alongside the primary transcription socket. This dual-socket approach ensures the STT service receives the profile audio first, allowing it to adapt to the user's voice characteristics before processing live conversation audio.

Streaming Profile Audio to STT Services

To prime the STT engine, Omi prepends the stored profile audio to the live stream using send_initial_file_path() in utils/stt/streaming.py. This function reads the profile WAV file from Google Cloud Storage and streams it to the STT socket up to the target duration, padding any remaining time with silence to maintain continuous audio flow.

from utils.stt.streaming import send_initial_file_path

async def prime_stt_engine(file_path: str, socket_send, is_active):
    await send_initial_file_path(
        file_path=file_path,
        send_to_socket=socket_send,
        is_active=is_active,
        sample_rate=16000,
        target_duration=30.0,  # SPEECH_PROFILE_FIXED_DURATION

        padding_seconds=2.0    # SPEECH_PROFILE_PADDING_DURATION

    )

This priming mechanism works with both Deepgram and Soniox STT providers, ensuring the service has learned the user's voice patterns before the actual conversation begins.

Matching Segments to Users: Speech Recognition

After transcription completes, Omi must determine which speaker is the device owner. The system calls the hosted Speech-Profile API via get_speech_profile_matching_predictions() in utils/stt/speech_profile.py. This function POSTs the original audio file and a list of transcript segments with timestamps to the external API.

from utils.stt.speech_profile import get_speech_profile_matching_predictions

def label_segments(uid: str, wav_path: str, segments):
    # Returns list of {"is_user": bool, "person_id": str|null}

    matches = get_speech_profile_matching_predictions(
        uid, wav_path, [s.dict() for s in segments]
    )
    
    for seg, match in zip(segments, matches):
        seg.is_user = match["is_user"]
        seg.person_id = match.get("person_id")
    return segments

The API returns a list of prediction objects containing is_user boolean flags and optional person_id values for other speakers. In utils/conversations/postprocess_conversation.py, the postprocess_conversation() function applies these flags to each TranscriptSegment object, definitively labeling which portions of the conversation belong to the device owner.

End-to-End Flow: Live Transcription with Speech Profiles

The complete pipeline from profile creation to speaker recognition follows this sequence:

1. Profile Creation: User uploads 30-second voice sample → stored in GCS at <uid>/speech_profile.wav → trained with Soniox → Redis flag set
2. Session Initialization: WebSocket connects to /transcribe → router checks get_user_has_speech_profile() → reserves pre-seconds if profile exists
3. Audio Priming: _create_speech_profile_loader_task() calls send_initial_file_path() to stream profile audio to STT socket before live audio
4. Live Transcription: STT service processes conversation with adapted speaker model
5. Post-Processing: get_speech_profile_matching_predictions() calls hosted API to get is_user flags
6. Segment Labeling: postprocess_conversation() applies flags to transcript segments

Key Implementation Files

Purpose	File Path	Key Functions
Profile storage (GCS)	backend/utils/other/storage.py	upload_profile_audio(), get_user_has_speech_profile()
Soniox speaker training	backend/utils/stt/soniox_util.py	create_user_speech_profile(), _train_user_speech_profile()
Profile audio streaming	backend/utils/stt/streaming.py	send_initial_file_path()
Speech recognition API	backend/utils/stt/speech_profile.py	get_speech_profile_matching_predictions()
Transcription router	backend/routers/transcribe.py	WebSocket handler, _create_speech_profile_loader_task()
Post-processing	backend/utils/conversations/postprocess_conversation.py	postprocess_conversation()
Public API endpoint	backend/routers/speech_profile.py	Upload endpoint

Summary

• Speech profile creation requires uploading a 30-second voice sample to Google Cloud Storage and training a Soniox speaker model, with state tracked via Redis flags.
• Profile detection happens at the start of each transcription session, determining whether to prepend profile audio to the stream.
• Audio priming uses send_initial_file_path() to stream the stored profile to the STT service before live audio begins, enabling speaker adaptation.
• Speech recognition relies on a hosted Speech-Profile API that analyzes segments after transcription, returning is_user flags that are applied to transcript segments in post-processing.

Frequently Asked Questions

How long should the speech profile audio sample be?

The Omi system expects approximately 30 seconds of clear speech, as defined by SPEECH_PROFILE_FIXED_DURATION. This duration provides sufficient acoustic data for Soniox to build a reliable speaker model while minimizing storage costs and transmission latency during the priming phase.

Can Omi use speech profiles with both Deepgram and Soniox?

Yes. While the profile creation and training specifically uses Soniox's speaker management system via manage_speakers commands in soniox_util.py, the profile audio prepending mechanism in send_initial_file_path() works with any STT service that accepts streaming audio, including Deepgram. However, the definitive speaker matching after transcription relies on the hosted Speech-Profile API regardless of which STT provider processed the stream.

What happens if speech profile training fails?

If create_user_speech_profile() encounters an error during the Soniox training process—whether due to network issues, invalid audio formats, or insufficient audio quality—the function returns False and the Redis flag user_has_soniox_speech_profile is not set. Consequently, subsequent transcription sessions will skip the profile prepending step, and the system will treat all speakers as unknown until a successful profile is created.

How does the hosted Speech-Profile API determine if a segment belongs to the user?

The API receives the original audio recording and a list of transcript segments with their timestamps. It performs speaker diarization and voice biometric analysis, comparing the acoustic characteristics of each segment against the user's stored speech profile. The API returns a boolean is_user flag for each segment, along with an optional person_id for other detected speakers, which Omi then attaches to the transcript segments in postprocess_conversation.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}