How Omi Stores and Retrieves Memories Using the RAG System

Omi stores memories in both Firestore for structured querying and Pinecone for semantic search, then retrieves them via direct lookup, vector similarity search, or conversation-aware RAG pipelines.

The open-source Omi project (basedhardware/omi) implements a dual-layer storage architecture for user memories that powers its Retrieval-Augmented Generation (RAG) capabilities. Each memory is persisted as structured data while simultaneously indexed as a dense vector, enabling both precise filtering and semantic similarity search when the AI agent needs context.

Memory Data Model Definition

In backend/models/memories.py, memories are defined as Pydantic models with strict typing. The Memory class contains fields for content, category, visibility, and tags, while MemoryDB extends this with database-specific metadata including uid, timestamps, and memory_id. Legacy category strings are automatically mapped to standardized enum values during instantiation to ensure backward compatibility.

Dual-Layer Storage Architecture

Structured Storage in Firestore

When a memory is created, MemoryDB.from_memory() converts the Pydantic model into a database-ready format. This structured record is written to Firestore via the database.memories module, preserving exact fields for filtering by date, category, or visibility. The Firestore layer serves direct lookups when the AI agent needs specific, non-semantic retrieval.

Vector Storage in Pinecone

Simultaneously, the memory text is embedded using the LLM embedding client (utils.llm.clients.embeddings). The upsert_memory_vector function in backend/database/vector_db.py (lines 56-77) stores this vector in Pinecone under the memories namespace (ns2), using a composite ID format {uid}-{memory_id}. Metadata attached to the vector includes uid, memory_id, category, and creation timestamp, enabling filtered semantic searches without loading full Firestore records.

Retrieval Mechanisms

Direct Lookup with get_memories_tool

For exhaustive retrieval of user facts, the LangChain tool get_memories_tool in backend/utils/retrieval/tools/memory_tools.py (lines 27-95) queries Firestore directly. It accepts limit and offset parameters for pagination and can filter by date ranges, returning formatted strings of MemoryDB objects. This tool is exposed to the AI agent when the query requires a complete inventory of memories rather than semantic relevance.

Semantic Search with search_memories_tool

When the user asks targeted questions like "What are my favorite cooking habits?", the search_memories_tool (lines 98-130) performs vector similarity search. It calls vector_db.find_similar_memories to query Pinecone namespace ns2, retrieves the top-k matching memory IDs with similarity scores, then hydrates full records from Firestore. This RAG pipeline ensures the AI agent receives only the most contextually relevant memories rather than the entire corpus.

Conversation-Aware RAG with retrieve_rag_conversation_context

For multi-turn dialogue requiring deep context, retrieve_rag_conversation_context in backend/utils/retrieval/rag.py (lines 68-110) orchestrates a sophisticated retrieval flow. First, retrieve_memory_context_params (from backend/utils/llm/chat.py) extracts topical keywords from the current conversation. Then retrieve_memories_for_topics performs topic-based vector search using query_vectors. Finally, get_better_conversation_chunk extracts concise chunks from the retrieved conversations, which are stitched together as the final RAG context string passed to the LLM.

Complete Implementation Examples

# ----------------------------------------------------------------------

# 1️⃣  Store a new memory (structured + vector)

# ----------------------------------------------------------------------

from backend.models.memories import Memory, MemoryDB
from backend.database.vector_db import upsert_memory_vector

def persist_memory(uid: str, conversation_id: str, text: str, category: str = "system"):
    # 1️⃣ Build a Memory object

    mem = Memory(content=text, category=category)

    # 2️⃣ Convert to DB object (adds timestamps, ids, etc.)

    mem_db = MemoryDB.from_memory(mem, uid=uid,
                                 conversation_id=conversation_id,
                                 manually_added=False)

    # 3️⃣ Write to Firestore (the `database.memories` module handles this)

    #    (pseudo‑code – actual call lives in the API layer)

    # database.memories.save_memory(uid, mem_db.dict())

    # 4️⃣ Index the vector for semantic search

    upsert_memory_vector(uid, mem_db.id, mem_db.content, mem_db.category.value)

# ----------------------------------------------------------------------

# 2️⃣  Retrieve all user facts (direct lookup)

# ----------------------------------------------------------------------

from backend.utils.retrieval.tools.memory_tools import get_memories_tool

# Simulate the agent passing a RunnableConfig that contains the user_id

config = {"configurable": {"user_id": "user_123"}}
print(get_memories_tool(limit=200, config=config))

# ----------------------------------------------------------------------

# 3️⃣  Semantic memory search (RAG)

# ----------------------------------------------------------------------

from backend.utils.retrieval.tools.memory_tools import search_memories_tool

config = {"configurable": {"user_id": "user_123"}}
query = "What are my favorite cooking habits?"
print(search_memories_tool(query=query, limit=5, config=config))

# ----------------------------------------------------------------------

# 4️⃣  Conversation‑level RAG context (used internally by the chat router)

# ----------------------------------------------------------------------

from backend.utils.retrieval.rag import retrieve_rag_conversation_context

uid = "user_123"

# `memory` is a Conversation object representing the current chat turn

context_str, related_convs = retrieve_rag_conversation_context(uid, memory)
print("RAG context:\n", context_str)

Summary

• Omi uses a dual-storage architecture: Firestore for structured data and Pinecone for vector embeddings.
• Memories are stored in Pinecone under namespace ns2 with metadata enabling filtered retrieval.
• Three retrieval modes exist: direct Firestore lookup (get_memories_tool), semantic search (search_memories_tool), and conversation-aware RAG (retrieve_rag_conversation_context).
• The RAG pipeline extracts topics, queries vectors, and chunks conversations to provide contextual memory augmentation.

Frequently Asked Questions

What is the difference between Firestore and Pinecone storage in Omi?

Firestore stores the complete structured memory records with exact fields for filtering and pagination, while Pinecone stores dense vector embeddings of memory content for semantic similarity search. The system uses both to enable both precise and fuzzy retrieval.

How does Omi handle legacy memory categories?

The Memory model in backend/models/memories.py automatically maps legacy category strings to standardized enum values during instantiation, ensuring backward compatibility while maintaining strict typing for new records.

What namespace does Omi use for memory vectors in Pinecone?

Omi stores all memory vectors in Pinecone under the namespace ns2, using vector IDs formatted as {uid}-{memory_id} to ensure unique identification across users.

How does the conversation-aware RAG system retrieve relevant context?

The system first extracts topical keywords from the current conversation using retrieve_memory_context_params, then performs topic-based vector search via retrieve_memories_for_topics, and finally chunks the retrieved conversations using get_better_conversation_chunk to build the final RAG context string.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}