How Action Items Extraction and Tracking Works in Omi: A Technical Deep Dive

Omi converts spoken conversations into structured, trackable tasks using a three-stage pipeline that leverages LLM-based extraction, Firestore persistence, and LangChain-powered retrieval tools.

The action items extraction and tracking system in the Omi open-source repository transforms raw conversation transcripts into persistent, manageable tasks. By combining specialized LLM prompts with a robust Firestore backend, Omi ensures that every "remind me to..." or "we need to..." moment gets captured, stored, and surfaced through both HTTP APIs and agent tools.

The Three-Stage Action Items Pipeline

Omi's workflow for handling tasks is divided into three distinct stages, each implemented in specific modules of the codebase.

Stage 1: Extraction via LLM

When a conversation finishes processing, the system calls extract_action_items in backend/utils/llm/conversation_processing.py. This function builds a rich context from the transcript, optional photo captions, calendar meeting data, and recent existing action items (for deduplication). It then sends this to an LLM using a cached prompt key omi-extract-actions, ensuring efficient token usage across conversations.

Stage 2: Persistence to Firestore

Once extracted, the action items are persisted via _save_action_items in backend/utils/conversations/process_conversation.py. This function first deletes any existing action items for the current conversation (preventing duplicates on re-processing), then batch-creates new documents in the action_items Firestore collection. Each document includes timestamps, completion status, due dates, and the originating conversation ID.

Stage 3: Retrieval and Updates

Users and agents interact with stored action items through LangChain tools defined in backend/utils/retrieval/tools/action_item_tools.py. The get_action_items_tool, create_action_item_tool, and update_action_item_tool provide a structured interface for listing, creating, and modifying tasks. These tools are exposed via the FastAPI router in backend/routers/action_items.py, enabling the Flutter frontend to sync data through HTTP endpoints.

Deep Dive: Extracting Action Items from Conversations

The extraction process begins with _build_conversation_context, which concatenates the transcript with optional visual and calendar context. To prevent duplicate tasks, the system fetches action items from the last two days and appends them to the prompt as deduplication hints.

The LLM receives a static instruction prefix (ACTION_ITEMS_INSTRUCTIONS) that mandates:

• Explicit request patterns (e.g., "Remind me to...", "I need to...") must always be extracted
• Real participant names must be used when calendar data is present
• Aggressive duplicate filtering using >95% similarity thresholds
• Separate due date extraction from the task description
• Workflow prioritization: read the whole conversation, prioritize explicit requests, discard low-importance implicit tasks

The prompt is bound to the LLM using llm_medium_experiment.bind(prompt_cache_key="omi-extract-actions"), enabling cross-conversation caching. The response is parsed by action_items_parser into structured ActionItem objects containing title, description, due date, timestamps, and speaker information.

How Action Items Are Stored and Tracked

Persistence occurs in backend/utils/conversations/process_conversation.py through the _save_action_items function:

def _save_action_items(uid: str, conversation: Conversation):
    if not conversation.structured or not conversation.structured.action_items:
        return

    is_locked = conversation.is_locked
    now = datetime.now(timezone.utc)
    action_items_data = []

    for ai in conversation.structured.action_items:
        action_items_data.append({
            'description': ai.description,
            'completed': ai.completed,
            'created_at': ai.created_at or now,
            'updated_at': ai.updated_at or now,
            'due_at': ai.due_at,
            'completed_at': ai.completed_at,
            'conversation_id': conversation.id,
            'is_locked': is_locked,
        })

    # Remove stale items from the same conversation (re‑process safety)

    action_items_db.delete_action_items_for_conversation(uid, conversation.id)
    # Batch write → returns the new Firestore document IDs

    action_item_ids = action_items_db.create_action_items_batch(uid, action_items_data)

The Firestore collection action_items is scoped per user. Each document stores UTC timestamps, completion status, due dates, the originating conversation_id, and an is_locked flag for conflict resolution during later edits.

After persistence, the system triggers Firebase Cloud Messaging (FCM) data messages for items with due dates via send_action_item_data_message in backend/utils/notifications.py. It also initiates async synchronization to external task providers through auto_sync_action_items_batch.

Accessing Action Items: APIs and Tools

Omi exposes action items through both LangChain tools for AI agents and REST endpoints for client applications.

LangChain Tools

Located in backend/utils/retrieval/tools/action_item_tools.py, these tools provide structured access:

• get_action_items_tool: Retrieves items filtered by date range, completion status, or conversation ID
• create_action_item_tool: Creates new tasks with optional due dates
• update_action_item_tool: Modifies existing items, including marking completion (which sets completed_at)

These tools automatically resolve the user_id from the agent configuration and format responses with status icons and human-readable dates.

REST API and Frontend Integration

The FastAPI router in backend/routers/action_items.py exposes HTTP endpoints consumed by the Flutter application:

curl -X GET "https://api.omi.app/v1/action_items?completed=false&limit=20" \
     -H "Authorization: Bearer <user‑jwt>"

The Flutter provider in app/lib/providers/action_items_provider.dart wraps these endpoints, enabling the UI to create tasks:

await ActionItemsProvider.instance.createActionItem(
  description: "Buy groceries",
  dueAt: DateTime.now().add(Duration(days: 1)),
);

Summary

• Extraction occurs in backend/utils/llm/conversation_processing.py using cached LLM prompts with aggressive deduplication rules and calendar context awareness.
• Persistence happens via _save_action_items in backend/utils/conversations/process_conversation.py, which batch-writes to the Firestore action_items collection after removing stale entries.
• Tracking is enabled through LangChain tools in backend/utils/retrieval/tools/action_item_tools.py and REST endpoints in backend/routers/action_items.py, supporting filtering, creation, completion, and push notifications.
• Integration extends to external task providers via async sync and FCM data messages for due date reminders.

Frequently Asked Questions

How does Omi prevent duplicate action items from being created?

Omi implements deduplication at two levels. During extraction, the LLM prompt includes action items from the last two days as context, instructing the model to filter out tasks with greater than 95% similarity. During persistence, _save_action_items first deletes all existing action items for the specific conversation before writing new ones, ensuring re-processing never creates duplicates.

Can action items be created manually, or only extracted from conversations?

Both methods are supported. While the primary flow extracts tasks automatically via extract_action_items in backend/utils/llm/conversation_processing.py, users can manually create action items through the create_action_item_tool (LangChain) or the Flutter provider in app/lib/providers/action_items_provider.dart, which posts to the REST API endpoint.

What happens when an action item reaches its due date?

When an action item with a due_at timestamp is created or updated, the system triggers send_action_item_data_message from backend/utils/notifications.py, sending an FCM data message to the user's device. Additionally, the auto_sync_action_items_batch function initiates async synchronization to external task providers (like Google Tasks or Todoist) if the user has configured integrations.

How does the system handle action items when a conversation is edited?

The is_locked flag in the action item document plays a critical role. When _save_action_items persists tasks, it stores the conversation's is_locked status. If a user later edits a conversation that was previously locked, the system can detect conflicts between newly extracted items and existing locked items, preventing accidental overwrites of manually modified tasks while allowing updates to unlocked entries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}