# How Action Items Extraction and Tracking Works in Omi: A Technical Deep Dive

> Learn how Omi extracts and tracks action items from conversations. Discover its LLM, Firestore, and LangChain powered three-stage technical pipeline for organized task management.

- Repository: [omi/omi](https://github.com/basedhardware/omi)
- Tags: deep-dive
- Published: 2026-02-26

---

**Omi converts spoken conversations into structured, trackable tasks using a three-stage pipeline that leverages LLM-based extraction, Firestore persistence, and LangChain-powered retrieval tools.**

The **action items extraction and tracking** system in the Omi open-source repository transforms raw conversation transcripts into persistent, manageable tasks. By combining specialized LLM prompts with a robust Firestore backend, Omi ensures that every "remind me to..." or "we need to..." moment gets captured, stored, and surfaced through both HTTP APIs and agent tools.

## The Three-Stage Action Items Pipeline

Omi's workflow for handling tasks is divided into three distinct stages, each implemented in specific modules of the codebase.

### Stage 1: Extraction via LLM

When a conversation finishes processing, the system calls `extract_action_items` in [`backend/utils/llm/conversation_processing.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/llm/conversation_processing.py). This function builds a rich context from the transcript, optional photo captions, calendar meeting data, and recent existing action items (for deduplication). It then sends this to an LLM using a cached prompt key `omi-extract-actions`, ensuring efficient token usage across conversations.

### Stage 2: Persistence to Firestore

Once extracted, the action items are persisted via `_save_action_items` in [`backend/utils/conversations/process_conversation.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/conversations/process_conversation.py). This function first deletes any existing action items for the current conversation (preventing duplicates on re-processing), then batch-creates new documents in the `action_items` Firestore collection. Each document includes timestamps, completion status, due dates, and the originating conversation ID.

### Stage 3: Retrieval and Updates

Users and agents interact with stored action items through LangChain tools defined in [`backend/utils/retrieval/tools/action_item_tools.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/retrieval/tools/action_item_tools.py). The `get_action_items_tool`, `create_action_item_tool`, and `update_action_item_tool` provide a structured interface for listing, creating, and modifying tasks. These tools are exposed via the FastAPI router in [`backend/routers/action_items.py`](https://github.com/basedhardware/omi/blob/main/backend/routers/action_items.py), enabling the Flutter frontend to sync data through HTTP endpoints.

## Deep Dive: Extracting Action Items from Conversations

The extraction process begins with `_build_conversation_context`, which concatenates the transcript with optional visual and calendar context. To prevent duplicate tasks, the system fetches action items from the last two days and appends them to the prompt as deduplication hints.

The LLM receives a static instruction prefix (`ACTION_ITEMS_INSTRUCTIONS`) that mandates:

- **Explicit request patterns** (e.g., "Remind me to...", "I need to...") must always be extracted
- **Real participant names** must be used when calendar data is present
- **Aggressive duplicate filtering** using >95% similarity thresholds
- **Separate due date extraction** from the task description
- **Workflow prioritization**: read the whole conversation, prioritize explicit requests, discard low-importance implicit tasks

The prompt is bound to the LLM using `llm_medium_experiment.bind(prompt_cache_key="omi-extract-actions")`, enabling cross-conversation caching. The response is parsed by `action_items_parser` into structured `ActionItem` objects containing title, description, due date, timestamps, and speaker information.

## How Action Items Are Stored and Tracked

Persistence occurs in [`backend/utils/conversations/process_conversation.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/conversations/process_conversation.py) through the `_save_action_items` function:

```python
def _save_action_items(uid: str, conversation: Conversation):
    if not conversation.structured or not conversation.structured.action_items:
        return

    is_locked = conversation.is_locked
    now = datetime.now(timezone.utc)
    action_items_data = []

    for ai in conversation.structured.action_items:
        action_items_data.append({
            'description': ai.description,
            'completed': ai.completed,
            'created_at': ai.created_at or now,
            'updated_at': ai.updated_at or now,
            'due_at': ai.due_at,
            'completed_at': ai.completed_at,
            'conversation_id': conversation.id,
            'is_locked': is_locked,
        })

    # Remove stale items from the same conversation (re‑process safety)

    action_items_db.delete_action_items_for_conversation(uid, conversation.id)
    # Batch write → returns the new Firestore document IDs

    action_item_ids = action_items_db.create_action_items_batch(uid, action_items_data)

```

The Firestore collection `action_items` is scoped per user. Each document stores UTC timestamps, completion status, due dates, the originating `conversation_id`, and an `is_locked` flag for conflict resolution during later edits.

After persistence, the system triggers **Firebase Cloud Messaging (FCM)** data messages for items with due dates via `send_action_item_data_message` in [`backend/utils/notifications.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/notifications.py). It also initiates async synchronization to external task providers through `auto_sync_action_items_batch`.

## Accessing Action Items: APIs and Tools

Omi exposes action items through both LangChain tools for AI agents and REST endpoints for client applications.

### LangChain Tools

Located in [`backend/utils/retrieval/tools/action_item_tools.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/retrieval/tools/action_item_tools.py), these tools provide structured access:

- **`get_action_items_tool`**: Retrieves items filtered by date range, completion status, or conversation ID
- **`create_action_item_tool`**: Creates new tasks with optional due dates
- **`update_action_item_tool`**: Modifies existing items, including marking completion (which sets `completed_at`)

These tools automatically resolve the `user_id` from the agent configuration and format responses with status icons and human-readable dates.

### REST API and Frontend Integration

The FastAPI router in [`backend/routers/action_items.py`](https://github.com/basedhardware/omi/blob/main/backend/routers/action_items.py) exposes HTTP endpoints consumed by the Flutter application:

```bash
curl -X GET "https://api.omi.app/v1/action_items?completed=false&limit=20" \
     -H "Authorization: Bearer <user‑jwt>"

```

The Flutter provider in `app/lib/providers/action_items_provider.dart` wraps these endpoints, enabling the UI to create tasks:

```dart
await ActionItemsProvider.instance.createActionItem(
  description: "Buy groceries",
  dueAt: DateTime.now().add(Duration(days: 1)),
);

```

## Summary

- **Extraction** occurs in [`backend/utils/llm/conversation_processing.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/llm/conversation_processing.py) using cached LLM prompts with aggressive deduplication rules and calendar context awareness.
- **Persistence** happens via `_save_action_items` in [`backend/utils/conversations/process_conversation.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/conversations/process_conversation.py), which batch-writes to the Firestore `action_items` collection after removing stale entries.
- **Tracking** is enabled through LangChain tools in [`backend/utils/retrieval/tools/action_item_tools.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/retrieval/tools/action_item_tools.py) and REST endpoints in [`backend/routers/action_items.py`](https://github.com/basedhardware/omi/blob/main/backend/routers/action_items.py), supporting filtering, creation, completion, and push notifications.
- **Integration** extends to external task providers via async sync and FCM data messages for due date reminders.

## Frequently Asked Questions

### How does Omi prevent duplicate action items from being created?

Omi implements deduplication at two levels. During extraction, the LLM prompt includes action items from the last two days as context, instructing the model to filter out tasks with greater than 95% similarity. During persistence, `_save_action_items` first deletes all existing action items for the specific conversation before writing new ones, ensuring re-processing never creates duplicates.

### Can action items be created manually, or only extracted from conversations?

Both methods are supported. While the primary flow extracts tasks automatically via `extract_action_items` in [`backend/utils/llm/conversation_processing.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/llm/conversation_processing.py), users can manually create action items through the `create_action_item_tool` (LangChain) or the Flutter provider in `app/lib/providers/action_items_provider.dart`, which posts to the REST API endpoint.

### What happens when an action item reaches its due date?

When an action item with a `due_at` timestamp is created or updated, the system triggers `send_action_item_data_message` from [`backend/utils/notifications.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/notifications.py), sending an FCM data message to the user's device. Additionally, the `auto_sync_action_items_batch` function initiates async synchronization to external task providers (like Google Tasks or Todoist) if the user has configured integrations.

### How does the system handle action items when a conversation is edited?

The `is_locked` flag in the action item document plays a critical role. When `_save_action_items` persists tasks, it stores the conversation's `is_locked` status. If a user later edits a conversation that was previously locked, the system can detect conflicts between newly extracted items and existing locked items, preventing accidental overwrites of manually modified tasks while allowing updates to unlocked entries.