# How Omi Stores and Retrieves Memories Using the RAG System

> Discover how Omi uses Firestore and Pinecone with a RAG system to store and retrieve memories. Learn about direct lookup, vector search, and RAG pipelines.

- Repository: [omi/omi](https://github.com/basedhardware/omi)
- Tags: deep-dive
- Published: 2026-02-26

---

**Omi stores memories in both Firestore for structured querying and Pinecone for semantic search, then retrieves them via direct lookup, vector similarity search, or conversation-aware RAG pipelines.**

The open-source Omi project ([basedhardware/omi](https://github.com/basedhardware/omi)) implements a dual-layer storage architecture for user memories that powers its Retrieval-Augmented Generation (RAG) capabilities. Each memory is persisted as structured data while simultaneously indexed as a dense vector, enabling both precise filtering and semantic similarity search when the AI agent needs context.

## Memory Data Model Definition

In [`backend/models/memories.py`](https://github.com/basedhardware/omi/blob/main/backend/models/memories.py), memories are defined as Pydantic models with strict typing. The `Memory` class contains fields for `content`, `category`, `visibility`, and `tags`, while `MemoryDB` extends this with database-specific metadata including `uid`, timestamps, and `memory_id`. Legacy category strings are automatically mapped to standardized enum values during instantiation to ensure backward compatibility.

## Dual-Layer Storage Architecture

### Structured Storage in Firestore

When a memory is created, `MemoryDB.from_memory()` converts the Pydantic model into a database-ready format. This structured record is written to Firestore via the `database.memories` module, preserving exact fields for filtering by date, category, or visibility. The Firestore layer serves direct lookups when the AI agent needs specific, non-semantic retrieval.

### Vector Storage in Pinecone

Simultaneously, the memory text is embedded using the LLM embedding client (`utils.llm.clients.embeddings`). The `upsert_memory_vector` function in [`backend/database/vector_db.py`](https://github.com/basedhardware/omi/blob/main/backend/database/vector_db.py) (lines 56-77) stores this vector in Pinecone under the **memories namespace (`ns2`)**, using a composite ID format `{uid}-{memory_id}`. Metadata attached to the vector includes `uid`, `memory_id`, `category`, and creation timestamp, enabling filtered semantic searches without loading full Firestore records.

## Retrieval Mechanisms

### Direct Lookup with get_memories_tool

For exhaustive retrieval of user facts, the LangChain tool `get_memories_tool` in [`backend/utils/retrieval/tools/memory_tools.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/retrieval/tools/memory_tools.py) (lines 27-95) queries Firestore directly. It accepts `limit` and `offset` parameters for pagination and can filter by date ranges, returning formatted strings of `MemoryDB` objects. This tool is exposed to the AI agent when the query requires a complete inventory of memories rather than semantic relevance.

### Semantic Search with search_memories_tool

When the user asks targeted questions like "What are my favorite cooking habits?", the `search_memories_tool` (lines 98-130) performs vector similarity search. It calls `vector_db.find_similar_memories` to query Pinecone namespace `ns2`, retrieves the top-k matching memory IDs with similarity scores, then hydrates full records from Firestore. This RAG pipeline ensures the AI agent receives only the most contextually relevant memories rather than the entire corpus.

### Conversation-Aware RAG with retrieve_rag_conversation_context

For multi-turn dialogue requiring deep context, `retrieve_rag_conversation_context` in [`backend/utils/retrieval/rag.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/retrieval/rag.py) (lines 68-110) orchestrates a sophisticated retrieval flow. First, `retrieve_memory_context_params` (from [`backend/utils/llm/chat.py`](https://github.com/basedhardware/omi/blob/main/backend/utils/llm/chat.py)) extracts topical keywords from the current conversation. Then `retrieve_memories_for_topics` performs topic-based vector search using `query_vectors`. Finally, `get_better_conversation_chunk` extracts concise chunks from the retrieved conversations, which are stitched together as the final RAG context string passed to the LLM.

## Complete Implementation Examples

```python

# ----------------------------------------------------------------------

# 1️⃣  Store a new memory (structured + vector)

# ----------------------------------------------------------------------

from backend.models.memories import Memory, MemoryDB
from backend.database.vector_db import upsert_memory_vector

def persist_memory(uid: str, conversation_id: str, text: str, category: str = "system"):
    # 1️⃣ Build a Memory object

    mem = Memory(content=text, category=category)

    # 2️⃣ Convert to DB object (adds timestamps, ids, etc.)

    mem_db = MemoryDB.from_memory(mem, uid=uid,
                                 conversation_id=conversation_id,
                                 manually_added=False)

    # 3️⃣ Write to Firestore (the `database.memories` module handles this)

    #    (pseudo‑code – actual call lives in the API layer)

    # database.memories.save_memory(uid, mem_db.dict())

    # 4️⃣ Index the vector for semantic search

    upsert_memory_vector(uid, mem_db.id, mem_db.content, mem_db.category.value)

```

```python

# ----------------------------------------------------------------------

# 2️⃣  Retrieve all user facts (direct lookup)

# ----------------------------------------------------------------------

from backend.utils.retrieval.tools.memory_tools import get_memories_tool

# Simulate the agent passing a RunnableConfig that contains the user_id

config = {"configurable": {"user_id": "user_123"}}
print(get_memories_tool(limit=200, config=config))

```

```python

# ----------------------------------------------------------------------

# 3️⃣  Semantic memory search (RAG)

# ----------------------------------------------------------------------

from backend.utils.retrieval.tools.memory_tools import search_memories_tool

config = {"configurable": {"user_id": "user_123"}}
query = "What are my favorite cooking habits?"
print(search_memories_tool(query=query, limit=5, config=config))

```

```python

# ----------------------------------------------------------------------

# 4️⃣  Conversation‑level RAG context (used internally by the chat router)

# ----------------------------------------------------------------------

from backend.utils.retrieval.rag import retrieve_rag_conversation_context

uid = "user_123"

# `memory` is a Conversation object representing the current chat turn

context_str, related_convs = retrieve_rag_conversation_context(uid, memory)
print("RAG context:\n", context_str)

```

## Summary

- Omi uses a **dual-storage architecture**: Firestore for structured data and Pinecone for vector embeddings.
- Memories are stored in Pinecone under namespace **`ns2`** with metadata enabling filtered retrieval.
- **Three retrieval modes** exist: direct Firestore lookup (`get_memories_tool`), semantic search (`search_memories_tool`), and conversation-aware RAG (`retrieve_rag_conversation_context`).
- The RAG pipeline extracts topics, queries vectors, and chunks conversations to provide contextual memory augmentation.

## Frequently Asked Questions

### What is the difference between Firestore and Pinecone storage in Omi?

Firestore stores the complete structured memory records with exact fields for filtering and pagination, while Pinecone stores dense vector embeddings of memory content for semantic similarity search. The system uses both to enable both precise and fuzzy retrieval.

### How does Omi handle legacy memory categories?

The `Memory` model in [`backend/models/memories.py`](https://github.com/basedhardware/omi/blob/main/backend/models/memories.py) automatically maps legacy category strings to standardized enum values during instantiation, ensuring backward compatibility while maintaining strict typing for new records.

### What namespace does Omi use for memory vectors in Pinecone?

Omi stores all memory vectors in Pinecone under the namespace **`ns2`**, using vector IDs formatted as `{uid}-{memory_id}` to ensure unique identification across users.

### How does the conversation-aware RAG system retrieve relevant context?

The system first extracts topical keywords from the current conversation using `retrieve_memory_context_params`, then performs topic-based vector search via `retrieve_memories_for_topics`, and finally chunks the retrieved conversations using `get_better_conversation_chunk` to build the final RAG context string.