Omi Backend Modal Deployment Setup: Serverless GPU Architecture Explained

The Omi backend deploys three independent Modal services—speech_profile, vad, and job—that run GPU-accelerated speaker recognition, voice-activity detection, and hourly cron jobs inside reproducible container images.

The Modal deployment setup for the basedhardware/omi repository provides a fully serverless backend architecture. Instead of maintaining always-on servers, Omi uses Modal's serverless platform to spin up GPU instances on demand for heavy inference tasks while keeping background jobs running on lightweight scheduled containers.

Overview of the Modal Architecture

The deployment consists of three distinct Modal Apps defined in separate files under backend/modal/:

• speech_profile – GPU-powered speaker identification service
• vad – Voice-activity detection endpoint
• job – Hourly cron job for notifications

Each service declares its own container image, resource requirements, and secrets, allowing independent scaling and versioning.

Core Modal Services

Speech Profile Service

The speech_profile service handles speaker recognition using the SpeechBrain library. Defined in [backend/modal/speech_profile_modal.py](https://github.com/basedhardware/omi/blob/main/backend/modal/speech_profile_modal.py), this service requires GPU acceleration to run embedding extraction models efficiently.

Resource configuration:

• GPU: NVIDIA T4 (modal.gpu.T4(count=1))
• CPU: 4 cores
• Memory: 1–2 GB (memory=(1024, 2048))
• Concurrency: 2 concurrent inputs (allow_concurrent_inputs=2)
• Warm pool: 1 always-warm container (keep_warm=1)

The service exposes a POST endpoint that accepts a user ID, audio file, and transcription segments, returning classification results indicating whether each segment belongs to the user or another speaker.

Voice Activity Detection (VAD) Service

The vad service provides speech-segment timestamps for uploaded audio files. Unlike the standalone speech_profile app, the VAD endpoint is mounted on the main FastAPI server defined in [backend/modal/main.py](https://github.com/basedhardware/omi/blob/main/backend/modal/main.py).

The actual inference logic resides in [backend/modal/vad_modal.py](https://github.com/basedhardware/omi/blob/main/backend/modal/vad_modal.py), which implements the vad_endpoint function. The main application imports this function and exposes it via the /v1/vad route, inheriting the same secret configuration (huggingface-token, envs, gcp-credentials) as the primary deployment.

Background Job Service

The job service handles hourly background tasks such as sending notification emails and push messages. Defined in [backend/modal/job_modal.py](https://github.com/basedhardware/omi/blob/main/backend/modal/job_modal.py), this service uses Modal's Cron scheduler rather than HTTP endpoints.

Key characteristics:

• Schedule: Cron('0 * * * *') (runs at the top of every hour)
• Image: Debian slim base with FFmpeg, Git, and Unzip
• Entry point: start_cron_job() from utils.other.notifications

This service requires fewer resources than the GPU-enabled inference services, using only standard CPU and memory allocations without GPU support.

Container Image and Resource Configuration

Each Modal service defines a reproducible container image using Modal's declarative Image API. The build process starts from a minimal Debian slim base and layers only the dependencies required for that specific service.

Image Definition Pattern

# From backend/modal/speech_profile_modal.py

image = (
    Image.debian_slim()
    .apt_install('ffmpeg')
    .pip_install("torch")
    .pip_install("speechbrain")
    # Additional dependencies...

)

This approach ensures that the speech_profile image contains PyTorch and SpeechBrain for GPU inference, while the job image remains lightweight with only notification utilities.

Resource Specification

Modal functions declare hardware requirements through the @app.function() decorator:

@app.function(
    image=image,
    keep_warm=1,
    memory=(1024, 2048),
    allow_concurrent_inputs=2,
    cpu=4,
    gpu=modal.gpu.T4(count=1),
    secrets=[
        Secret.from_name('huggingface-token'),
        Secret.from_name('envs'),
        Secret.from_name("gcp-credentials")
    ],
)
@web_endpoint(method='POST')
def endpoint(uid: str, audio_file: UploadFile = File(...), segments: str = Form(...)):
    # Implementation...

The keep_warm=1 parameter maintains one idle container to minimize cold-start latency for user-facing inference, while allow_concurrent_inputs=2 enables request batching within the same container instance.

FastAPI Routing and Endpoints

The [backend/modal/main.py](https://github.com/basedhardware/omi/blob/main/backend/modal/main.py) file serves as the entry point for the Modal deployment, creating a FastAPI application that routes HTTP requests to the appropriate Modal functions.

Route Definitions

from fastapi import FastAPI, File, UploadFile, Form

app = FastAPI()

@app.post('/v1/speaker-identification')
def speaker_identification(uid: str, audio_file: UploadFile = File, segments: str = Form(...)):
    return speaker_identification_endpoint(uid, audio_file, segments)

@app.post('/v1/vad')
def vad(file: UploadFile = File):
    return vad_endpoint(file)

This architecture separates the HTTP interface (FastAPI routes in main.py) from the compute implementation (Modal functions in speech_profile_modal.py and vad_modal.py). When deployed, Modal automatically provisions the FastAPI server and scales the underlying GPU/CPU resources based on request volume.

Deployment Workflow

Deploying the Omi backend to Modal requires the Modal CLI and appropriate authentication tokens.

Deployment Commands

Execute the deployment using the Modal CLI from the repository root:

modal deploy backend/modal/main.py

Alternatively, the repository includes a pre-commit script that automates this process:

scripts/pre-commit

This script invokes the Modal CLI to build the container images, push them to Modal's internal registry, and spin up the FastAPI server with the configured GPU and cron resources.

Endpoint Access

Once deployed, Modal generates a public URL for the service (e.g., https://<project>.modal.run). The FastAPI endpoints become immediately accessible at:

• POST /v1/speaker-identification – Speaker recognition
• POST /v1/vad – Voice activity detection

The cron job runs automatically on the specified schedule without requiring external triggers.

Summary

• Three independent Modal apps handle distinct workloads: speech_profile (GPU inference), vad (HTTP endpoint), and job (hourly cron).
• Declarative container images built from Image.debian_slim() ensure reproducible environments with specific Python and system dependencies.
• GPU acceleration using NVIDIA T4 cards with 4 CPUs and 1–2 GB memory enables real-time speaker recognition while minimizing costs through serverless scaling.
• FastAPI routing in backend/modal/main.py separates HTTP interface logic from Modal compute functions, exposing /v1/speaker-identification and /v1/vad endpoints.
• Automated deployment via modal deploy backend/modal/main.py or the scripts/pre-commit hook builds images and provisions infrastructure without manual server management.

Frequently Asked Questions

How does the Omi backend handle GPU resources for speaker recognition?

The speech_profile service in backend/modal/speech_profile_modal.py requests an NVIDIA T4 GPU through Modal's modal.gpu.T4(count=1) parameter. It allocates 4 CPU cores and 1–2 GB of RAM with keep_warm=1 to maintain one idle container for low-latency inference. The GPU only spins up when processing speaker identification requests, reducing costs compared to always-on GPU servers.

What is the difference between the VAD and speech profile Modal services?

The VAD (voice activity detection) service identifies speech segments in audio files and runs as a standard CPU-based endpoint mounted in backend/modal/main.py. The speech_profile service performs computationally intensive speaker recognition using deep learning models, requiring GPU acceleration and dedicated memory allocation. While both expose HTTP endpoints, only speech_profile requires the T4 GPU and higher CPU count specified in its Modal function decorator.

How do I deploy the Omi backend to Modal?

Deploy the backend by running modal deploy backend/modal/main.py from the repository root directory. This command builds the declarative container images defined in speech_profile_modal.py and job_modal.py, pushes them to Modal's internal registry, and provisions the FastAPI server with configured GPU and cron resources. Alternatively, execute scripts/pre-commit to trigger the same deployment workflow through the repository's automation script.

What triggers the background job service in the Modal deployment?

The job service defined in backend/modal/job_modal.py runs automatically on a fixed schedule using Modal's Cron('0 * * * *') decorator, which triggers the function at the top of every hour. Unlike the HTTP endpoints for speech processing, this cron job requires no external HTTP request to initiate. It executes start_cron_job() from the notifications utility module to process pending emails and push messages without maintaining a persistent server.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}