Structure of the Plugin System for Omi Apps: FastAPI Architecture Explained

Omi Apps uses a modular FastAPI-based plugin architecture where third-party integrations live as independent Python packages under plugins/, register their own FastAPI routers in plugins/main.py, and deploy via Modal, while the frontend discovers them through both an approved backend endpoint and a community JSON catalog.

The basedhardware/omi repository implements a four-layer plugin system that allows developers to extend the Omi AI wearable's capabilities without modifying core backend code. This architecture separates plugin logic into self-contained modules, exposes them through a unified FastAPI surface, and dynamically injects plugin context into LLM conversations. Understanding the structure of the plugin system for Omi Apps is essential for building custom integrations that interact with conversation data and external services.

Plugin Package Structure

Each Omi plugin resides in its own sub-directory under the plugins/ folder, following a consistent layout that separates concerns between router logic and data models.

The directory structure follows this pattern:

plugins/
  ├─ basic/
  │    └─ conversation_created.py   # Router for basic plugin events

  ├─ oauth/
  │    └─ conversation_created.py   # Router for OAuth-based plugins

  ├─ zapier/
  │    └─ conversation_created.py   # Router for Zapier integration

  ├─ omi-shopify-app/
  │    ├─ models.py                 # Pydantic models for Shopify

  │    └─ main.py                   # Core plugin logic

  └─ models.py                       # Shared types: PluginResult, Conversation

The shared plugins/models.py file defines the data structures that every plugin uses to communicate with the core system. According to the source code, this includes PluginResult which holds a plugin’s output and is imported by all plugin implementations to ensure type consistency across the ecosystem. PluginResult and Conversation types are defined here to standardize how plugins read conversation data and return processing results.

FastAPI Router Registration

The central integration point for all plugins is plugins/main.py, which imports individual plugin routers and registers them with the main FastAPI application instance.

As implemented in plugins/main.py (lines 70-88), the system wires each plugin's router into the central app:

# plugins/main.py

app.include_router(basic_conversation_created_router.router)      # Basic plugin

app.include_router(oauth_conversation_created_router.router)      # OAuth plugin

app.include_router(zapier_conversation_created_router.router)     # Zapier plugin

app.include_router(multion_router.router)                         # External integration

# … (other routers)

This file serves as the entry point for the Modal-hosted service and acts as the single source of truth for which plugins are currently active. When adding a new plugin, developers must import their router module and call app.include_router() to expose the plugin's endpoints to the network.

Containerized Deployment with Modal

The plugin service runs as a Modal application, utilizing serverless containers to handle HTTP requests without managing infrastructure. The deployment configuration in plugins/main.py (lines 54-66) uses Modal-specific decorators to build the runtime environment.

The @modal_app.function decorator constructs a container image that pip-installs dependencies from requirements.txt and mounts static template directories. The function then returns the FastAPI application via the @asgi_app() decorator, allowing Modal to serve the HTTP traffic:

@modal_app.function(
    image=image,
    mounts=[modal.Mount.from_local_dir("./templates", remote_path="/templates")],
)
@modal.asgi_app()
def fastapi_app():
    return app

This architecture ensures that plugins run in isolated, reproducible environments while sharing the same network surface through the unified FastAPI app.

Frontend Discovery Mechanisms

The web frontend discovers available plugins through two distinct channels: an official approved catalog and a community-maintained JSON file.

Approved Apps Endpoint

The backend exposes /v1/approved-apps to serve curated plugins. The frontend consumes this through web/frontend/src/lib/api/apps.ts:

const response = await fetch(`${envConfig.API_URL}/v1/approved-apps?...`);

TypeScript definitions in web/frontend/src/types/plugins/plugins.types.ts (lines 2-14) mirror the backend schema, defining the Plugin interface with properties like id, name, capabilities, and prompts to ensure type safety across the API boundary.

Community Plugin Catalog

For user-contributed plugins that don't require backend approval, the system reads from community-plugins.json at the repository root. The frontend imports this file via getCommunityPlugins() to display community apps alongside official ones, allowing any developer to publish a plugin by submitting a pull request to update the JSON file.

LLM Integration and Conversation Context

When a user selects a plugin, the backend conversation engine stores the chosen App object in the state under plugin_selected. In backend/utils/llm/chat.py (lines 426-440), the system injects plugin-specific instructions into the LLM prompt generator.

The LLM receives a <plugin_instructions> block containing the plugin’s personality, description, and custom prompts. This allows plugins to dynamically alter the AI's behavior during conversations. The retrieval engine in backend/utils/retrieval/graph.py manages the state transitions to ensure the correct plugin context is active throughout the conversation lifecycle.

Creating a Custom Omi Plugin

To build a new plugin, create a Python module with a FastAPI router and register it in the central application.

First, implement your plugin logic:

# plugins/my-awesome-plugin/main.py

from fastapi import APIRouter, HTTPException
from pydantic import BaseModel

router = APIRouter(prefix="/my-awesome-plugin")

class Input(BaseModel):
    uid: str
    text: str

@router.post("/process")
async def process(req: Input):
    # Your plugin logic here

    return {"result": f"Handled {req.text} for {req.uid}"}

Then register the router in the plugin entry point:

# plugins/main.py (excerpt)

from my_awesome_plugin import main as my_plugin_router

app.include_router(my_plugin_router.router)  # Makes endpoints live

To consume plugins in the frontend:

import { getApprovedApps } from '@/src/lib/api/apps';

export async function loadPlugins() {
  const plugins = await getApprovedApps();   // returns Plugin[]
  return plugins.filter(p => p.capabilities.includes('chat'));
}

Summary

• Modular Structure: Plugins live as independent packages under plugins/, each with their own routers and optional data models, while sharing common types through plugins/models.py.
• Router Aggregation: plugins/main.py serves as the integration point, importing and registering all plugin routers with the central FastAPI application.
• Serverless Deployment: The entire plugin surface deploys via Modal using @modal_app.function and @asgi_app() decorators, containerizing dependencies automatically.
• Dual Discovery: Frontend applications fetch official plugins from /v1/approved-apps and community plugins from community-plugins.json, with TypeScript types ensuring API consistency.
• LLM Context Injection: Selected plugins modify conversation behavior through <plugin_instructions> blocks injected by backend/utils/llm/chat.py, allowing dynamic personality and capability changes.

Frequently Asked Questions

How do I register a new plugin in the Omi system?

To register a new plugin, create a subdirectory under plugins/ containing a FastAPI router that defines your endpoints. Import this router in plugins/main.py and add it using app.include_router(). The plugin automatically becomes available when the Modal service redeploys. You must also add your plugin to community-plugins.json or submit it for inclusion in the approved apps endpoint to make it discoverable by frontend clients.

What is the difference between approved apps and community plugins?

Approved apps are vetted by the Omi team and served via the /v1/approved-apps backend endpoint, offering higher trust and potential backend integration privileges. Community plugins live in the community-plugins.json file at the repository root and are fetched directly by the frontend without backend moderation, allowing faster experimentation but with fewer guarantees about security or reliability.

How do plugins influence the LLM's responses during conversations?

When a user activates a plugin, the conversation engine stores the selection in state under plugin_selected. According to the source code in backend/utils/llm/chat.py, the system injects a <plugin_instructions> XML block into the LLM prompt containing the plugin's description, personality settings, and custom prompts. This contextual injection modifies how the AI interprets and responds to user messages while the plugin remains active.

What infrastructure hosts the Omi plugin system?

The plugin system runs on Modal, a serverless compute platform. The plugins/main.py file defines a Modal app using the @modal_app.function decorator to build a container image with all Python dependencies, then exposes the FastAPI application through @modal.asgi_app(). This setup handles automatic scaling, container management, and HTTP request routing without requiring manual server configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}