# Omi Backend Modal Deployment Setup: Serverless GPU Architecture Explained

> Discover the Omi backend Modal deployment setup, featuring serverless GPU architecture for speaker recognition, VAD, and cron jobs. Learn how it utilizes reproducible containers for efficient processing.

- Repository: [omi/omi](https://github.com/basedhardware/omi)
- Tags: architecture
- Published: 2026-02-26

---

**The Omi backend deploys three independent Modal services—`speech_profile`, `vad`, and `job`—that run GPU-accelerated speaker recognition, voice-activity detection, and hourly cron jobs inside reproducible container images.**

The **Modal deployment setup** for the [basedhardware/omi](https://github.com/basedhardware/omi) repository provides a fully serverless backend architecture. Instead of maintaining always-on servers, Omi uses Modal's serverless platform to spin up GPU instances on demand for heavy inference tasks while keeping background jobs running on lightweight scheduled containers.

## Overview of the Modal Architecture

The deployment consists of three distinct Modal **Apps** defined in separate files under `backend/modal/`:

- **`speech_profile`** – GPU-powered speaker identification service
- **`vad`** – Voice-activity detection endpoint
- **`job`** – Hourly cron job for notifications

Each service declares its own container image, resource requirements, and secrets, allowing independent scaling and versioning.

## Core Modal Services

### Speech Profile Service

The `speech_profile` service handles speaker recognition using the **SpeechBrain** library. Defined in [[`backend/modal/speech_profile_modal.py`](https://github.com/basedhardware/omi/blob/main/backend/modal/speech_profile_modal.py)](https://github.com/basedhardware/omi/blob/main/backend/modal/speech_profile_modal.py), this service requires GPU acceleration to run embedding extraction models efficiently.

**Resource configuration:**
- **GPU:** NVIDIA T4 (`modal.gpu.T4(count=1)`)
- **CPU:** 4 cores
- **Memory:** 1–2 GB (`memory=(1024, 2048)`)
- **Concurrency:** 2 concurrent inputs (`allow_concurrent_inputs=2`)
- **Warm pool:** 1 always-warm container (`keep_warm=1`)

The service exposes a `POST` endpoint that accepts a user ID, audio file, and transcription segments, returning classification results indicating whether each segment belongs to the user or another speaker.

### Voice Activity Detection (VAD) Service

The `vad` service provides speech-segment timestamps for uploaded audio files. Unlike the standalone `speech_profile` app, the VAD endpoint is mounted on the **main FastAPI server** defined in [[`backend/modal/main.py`](https://github.com/basedhardware/omi/blob/main/backend/modal/main.py)](https://github.com/basedhardware/omi/blob/main/backend/modal/main.py).

The actual inference logic resides in [[`backend/modal/vad_modal.py`](https://github.com/basedhardware/omi/blob/main/backend/modal/vad_modal.py)](https://github.com/basedhardware/omi/blob/main/backend/modal/vad_modal.py), which implements the `vad_endpoint` function. The main application imports this function and exposes it via the `/v1/vad` route, inheriting the same secret configuration (`huggingface-token`, `envs`, `gcp-credentials`) as the primary deployment.

### Background Job Service

The `job` service handles hourly background tasks such as sending notification emails and push messages. Defined in [[`backend/modal/job_modal.py`](https://github.com/basedhardware/omi/blob/main/backend/modal/job_modal.py)](https://github.com/basedhardware/omi/blob/main/backend/modal/job_modal.py), this service uses Modal's **Cron** scheduler rather than HTTP endpoints.

**Key characteristics:**
- **Schedule:** `Cron('0 * * * *')` (runs at the top of every hour)
- **Image:** Debian slim base with FFmpeg, Git, and Unzip
- **Entry point:** `start_cron_job()` from `utils.other.notifications`

This service requires fewer resources than the GPU-enabled inference services, using only standard CPU and memory allocations without GPU support.

## Container Image and Resource Configuration

Each Modal service defines a **reproducible container image** using Modal's declarative `Image` API. The build process starts from a minimal Debian slim base and layers only the dependencies required for that specific service.

### Image Definition Pattern

```python

# From backend/modal/speech_profile_modal.py

image = (
    Image.debian_slim()
    .apt_install('ffmpeg')
    .pip_install("torch")
    .pip_install("speechbrain")
    # Additional dependencies...

)

```

This approach ensures that the `speech_profile` image contains PyTorch and SpeechBrain for GPU inference, while the `job` image remains lightweight with only notification utilities.

### Resource Specification

Modal functions declare hardware requirements through the `@app.function()` decorator:

```python
@app.function(
    image=image,
    keep_warm=1,
    memory=(1024, 2048),
    allow_concurrent_inputs=2,
    cpu=4,
    gpu=modal.gpu.T4(count=1),
    secrets=[
        Secret.from_name('huggingface-token'),
        Secret.from_name('envs'),
        Secret.from_name("gcp-credentials")
    ],
)
@web_endpoint(method='POST')
def endpoint(uid: str, audio_file: UploadFile = File(...), segments: str = Form(...)):
    # Implementation...

```

The `keep_warm=1` parameter maintains one idle container to minimize cold-start latency for user-facing inference, while `allow_concurrent_inputs=2` enables request batching within the same container instance.

## FastAPI Routing and Endpoints

The [[`backend/modal/main.py`](https://github.com/basedhardware/omi/blob/main/backend/modal/main.py)](https://github.com/basedhardware/omi/blob/main/backend/modal/main.py) file serves as the **entry point** for the Modal deployment, creating a FastAPI application that routes HTTP requests to the appropriate Modal functions.

### Route Definitions

```python
from fastapi import FastAPI, File, UploadFile, Form

app = FastAPI()

@app.post('/v1/speaker-identification')
def speaker_identification(uid: str, audio_file: UploadFile = File, segments: str = Form(...)):
    return speaker_identification_endpoint(uid, audio_file, segments)

@app.post('/v1/vad')
def vad(file: UploadFile = File):
    return vad_endpoint(file)

```

This architecture separates the **HTTP interface** (FastAPI routes in [`main.py`](https://github.com/basedhardware/omi/blob/main/main.py)) from the **compute implementation** (Modal functions in [`speech_profile_modal.py`](https://github.com/basedhardware/omi/blob/main/speech_profile_modal.py) and [`vad_modal.py`](https://github.com/basedhardware/omi/blob/main/vad_modal.py)). When deployed, Modal automatically provisions the FastAPI server and scales the underlying GPU/CPU resources based on request volume.

## Deployment Workflow

Deploying the Omi backend to Modal requires the Modal CLI and appropriate authentication tokens.

### Deployment Commands

Execute the deployment using the Modal CLI from the repository root:

```bash
modal deploy backend/modal/main.py

```

Alternatively, the repository includes a pre-commit script that automates this process:

```bash
scripts/pre-commit

```

This script invokes the Modal CLI to build the container images, push them to Modal's internal registry, and spin up the FastAPI server with the configured GPU and cron resources.

### Endpoint Access

Once deployed, Modal generates a public URL for the service (e.g., `https://<project>.modal.run`). The FastAPI endpoints become immediately accessible at:
- `POST /v1/speaker-identification` – Speaker recognition
- `POST /v1/vad` – Voice activity detection

The cron job runs automatically on the specified schedule without requiring external triggers.

## Summary

- **Three independent Modal apps** handle distinct workloads: `speech_profile` (GPU inference), `vad` (HTTP endpoint), and `job` (hourly cron).
- **Declarative container images** built from `Image.debian_slim()` ensure reproducible environments with specific Python and system dependencies.
- **GPU acceleration** using NVIDIA T4 cards with 4 CPUs and 1–2 GB memory enables real-time speaker recognition while minimizing costs through serverless scaling.
- **FastAPI routing** in [`backend/modal/main.py`](https://github.com/basedhardware/omi/blob/main/backend/modal/main.py) separates HTTP interface logic from Modal compute functions, exposing `/v1/speaker-identification` and `/v1/vad` endpoints.
- **Automated deployment** via `modal deploy backend/modal/main.py` or the `scripts/pre-commit` hook builds images and provisions infrastructure without manual server management.

## Frequently Asked Questions

### How does the Omi backend handle GPU resources for speaker recognition?

The `speech_profile` service in [`backend/modal/speech_profile_modal.py`](https://github.com/basedhardware/omi/blob/main/backend/modal/speech_profile_modal.py) requests an NVIDIA T4 GPU through Modal's `modal.gpu.T4(count=1)` parameter. It allocates 4 CPU cores and 1–2 GB of RAM with `keep_warm=1` to maintain one idle container for low-latency inference. The GPU only spins up when processing speaker identification requests, reducing costs compared to always-on GPU servers.

### What is the difference between the VAD and speech profile Modal services?

The **VAD** (voice activity detection) service identifies speech segments in audio files and runs as a standard CPU-based endpoint mounted in [`backend/modal/main.py`](https://github.com/basedhardware/omi/blob/main/backend/modal/main.py). The **speech_profile** service performs computationally intensive speaker recognition using deep learning models, requiring GPU acceleration and dedicated memory allocation. While both expose HTTP endpoints, only `speech_profile` requires the T4 GPU and higher CPU count specified in its Modal function decorator.

### How do I deploy the Omi backend to Modal?

Deploy the backend by running `modal deploy backend/modal/main.py` from the repository root directory. This command builds the declarative container images defined in [`speech_profile_modal.py`](https://github.com/basedhardware/omi/blob/main/speech_profile_modal.py) and [`job_modal.py`](https://github.com/basedhardware/omi/blob/main/job_modal.py), pushes them to Modal's internal registry, and provisions the FastAPI server with configured GPU and cron resources. Alternatively, execute `scripts/pre-commit` to trigger the same deployment workflow through the repository's automation script.

### What triggers the background job service in the Modal deployment?

The `job` service defined in [`backend/modal/job_modal.py`](https://github.com/basedhardware/omi/blob/main/backend/modal/job_modal.py) runs automatically on a fixed schedule using Modal's `Cron('0 * * * *')` decorator, which triggers the function at the top of every hour. Unlike the HTTP endpoints for speech processing, this cron job requires no external HTTP request to initiate. It executes `start_cron_job()` from the notifications utility module to process pending emails and push messages without maintaining a persistent server.