Hybrid Memory Systems for Agents: A Three-Store Architecture Implementation

Hybrid memory systems combine vector, key-value, and graph stores into a unified façade to handle semantic similarity, exact fact lookup, and relationship reasoning within a single agent turn.

The rohitg00/ai-engineering-from-scratch repository provides a production-ready reference implementation of hybrid memory architectures. According to the Mem0 paper (arXiv:2504.19413) and the corresponding lesson in phases/14-agent-engineering/09-hybrid-memory-mem0/, agents overcome the limitations of single-store memory by wiring three complementary backends behind one public API. This design pattern ensures that no single query type forces a compromise on retrieval accuracy or latency.

The Three-Store Memory Model

A hybrid memory system aggregates three specialized storage backends, each optimized for a distinct query class. As documented in phases/14-agent-engineering/09-hybrid-memory-mem0/docs/en.md, no single store can efficiently answer every type of question:

• Vector Store: Handles semantic similarity searches (e.g., "what did we discuss about agent drift last week?") via nearest-neighbor embeddings. Key-value stores fail here because they only support exact matches, while graph traversal is too heavyweight for fuzzy similarity.
• Key-Value (KV) Store: Provides deterministic fact lookup (e.g., "what is the user’s phone number?") using hash-based indexing. Vector search introduces noise for exact identifiers, and graph traversal adds unnecessary overhead for simple retrieval.
• Graph Store: Enables relationship reasoning (e.g., "which customers share the same billing entity?") through typed edges. Vector stores cannot express relational constraints, while KV stores cannot traverse multi-hop paths.

Because real-world agents require all three query types within a single turn, the Mem0 architecture writes data to all three stores simultaneously during ingestion.

Core Architectural Steps

The implementation in phases/14-agent-engineering/09-hybrid-memory-mem0/code/main.py exposes two primary methods: add() for ingestion and search() for retrieval.

Ingestion via add()

When an agent receives new information, the system extracts facts and writes them to all three backends in lockstep:

1. The vector store embeds the fact text and stores the embedding.
2. The KV store indexes a deterministic key constructed from user_id, fact_type, and entity.
3. The graph store creates typed edges (subject → relation → object) with a validity flag and timestamp.

This triple-write ensures that subsequent queries can leverage the most appropriate retrieval mechanism regardless of query type.

Retrieval via search() and Fusion Scoring

The search() method queries all three stores in parallel and combines their results using a fusion scorer. As defined in phases/14-agent-engineering/09-hybrid-memory-mem0/docs/en.md, the scoring formula is:

score = (w_relevance * relevance(q, rec) +
         w_importance * importance(rec) +
         w_recency * recency(rec))

Parameters include:

• w_relevance: Weight for vector similarity or exact match strength.
• w_importance: Static weight assigned to critical facts.
• w_recency: Temporal decay factor (typically math.exp(-age / 3600) for hourly decay).

Each store returns a ranked list of candidate records. The fusion layer unifies these candidates, computes the weighted score for each, and returns the top-k results.

Scope-Aware Fusion and Temporal Invalidation

Memories are divided into user, session, and agent scopes. The fusion layer can bias queries toward specific scopes by adjusting weights—for example, increasing w_recency for chat agents or boosting w_importance for compliance-oriented workflows.

The graph store (Mem0g) implements temporal invalidation rather than deletion. When a new fact contradicts an existing edge (e.g., a user moving from Berlin to Lisbon), the old edge is marked invalid but retained. This enables historical queries such as "what was the user’s city in March?" without complex versioning schemes.

Production Benefits of Hybrid Memory

The three-store approach offers three operational advantages over monolithic memory:

• Complementarity: Each query class receives the optimal retrieval mechanism—vector for semantics, KV for exactitude, graph for relations.
• Scalable Fusion: The weighted sum is computationally cheap and tunable per product requirement without rewriting storage backends.
• Operational Flexibility: Teams can swap implementations independently—replacing the naive vector similarity with dense embeddings or substituting the graph backend—without modifying the other two stores.

Reference Implementation

The lesson provides a minimal, stdlib-only implementation in phases/14-agent-engineering/09-hybrid-memory-mem0/code/main.py. The code defines three storage classes and a unified Mem0 façade:

import time
import math
from collections import namedtuple
from typing import List, Tuple

Record = namedtuple("Record", "text user_id fact_type entity embed ts")

class VectorStore:
    """Naïve token-overlap similarity stand-in for embeddings."""
    def __init__(self):
        self.records: List[Record] = []

    def add(self, rec: Record):
        self.records.append(rec)

    def search(self, query: str, k: int = 5) -> List[Tuple[Record, float]]:
        def overlap(a, b):
            return len(set(a.split()) & set(b.split()))
        scored = [(r, overlap(query, r.text)) for r in self.records]
        scored.sort(key=lambda x: -x[1])
        return scored[:k]

class KVStore:
    """Exact-match store keyed on (user_id, fact_type, entity)."""
    def __init__(self):
        self.index = {}

    def add(self, rec: Record):
        key = (rec.user_id, rec.fact_type, rec.entity)
        self.index[key] = rec

    def search(self, user_id, fact_type, entity) -> List[Tuple[Record, float]]:
        key = (user_id, fact_type, entity)
        if key in self.index:
            return [(self.index[key], 1.0)]
        return []

class GraphStore:
    """Typed edges with temporal validity."""
    def __init__(self):
        self.edges = []  # (subj, rel, obj, ts, valid)

    def add(self, rec: Record):
        self.edges.append((rec.user_id, rec.fact_type, rec.entity,
                           rec.ts, True))

    def search(self, user_id, relation) -> List[Tuple[Record, float]]:
        hits = [(e, 1.0) for e in self.edges
                if e[0] == user_id and e[1] == relation and e[4]]
        return hits

class Mem0:
    """Top-level façade exposing add() and search()."""
    def __init__(self, w_relevance=0.5, w_importance=0.3, w_recency=0.2):
        self.vec = VectorStore()
        self.kv = KVStore()
        self.graph = GraphStore()
        self.weights = dict(w_relevance=w_relevance,
                            w_importance=w_importance,
                            w_recency=w_recency)

    def add(self, text, user_id, fact_type, entity):
        ts = time.time()
        rec = Record(text, user_id, fact_type, entity, None, ts)
        self.vec.add(rec)
        self.kv.add(rec)
        self.graph.add(rec)

    def _recency(self, ts):
        age = time.time() - ts
        return math.exp(-age / 3600)

    def search(self, query, user_id):
        vec_hits = self.vec.search(query)
        kv_hits = self.kv.search(user_id, "type", "entity")
        graph_hits = self.graph.search(user_id, "relation")

        results = {}
        for rec, base in vec_hits + kv_hits + graph_hits:
            recency = self._recency(rec.ts)
            score = (self.weights["w_relevance"] * base +
                     self.weights["w_importance"] * 1.0 +
                     self.weights["w_recency"] * recency)
            results[rec] = max(results.get(rec, 0), score)

        fused = sorted(results.items(), key=lambda x: -x[1])
        return fused[:5]

if __name__ == "__main__":
    mem = Mem0()
    mem.add("Discussed agent drift last week", "alice", "topic", "drift")
    mem.add("Alice phone is 555-1234", "alice", "contact", "phone")
    
    for rec, score in mem.search("agent drift", "alice"):
        print(f"{rec.text!r} → score {score:.3f}")

Running this script demonstrates how the fusion layer surfaces the appropriate memory regardless of whether the query is semantic ("agent drift"), factual ("phone"), or relational.

Related Hybrid Patterns

The repository contains additional lessons that apply the same hybridization principles to other agent components:

• Hybrid Retrieval (BM25 + Dense) – Lesson 65 (phases/19-capstone-projects/65-hybrid-retrieval-bm25-dense/docs/en.md): Combines sparse keyword BM25 with dense vector similarity, merging results via Reciprocal Rank Fusion.
• Hybrid Planner – Lesson 11 (phases/14-agent-engineering/11-planning-htn-and-evolutionary/docs/en.md): Integrates symbolic Hierarchical Task Network planning with LLM-based evolutionary search.
• Hybrid Transformer/SSM – Lesson 21 (phases/10-llms-from-scratch/21-jamba-hybrid-ssm-transformer/docs/en.md): Discusses model-level hybridization combining Transformers with Structured State Space models.

Summary

• Three-store architecture: Vector, KV, and graph stores cover semantic similarity, exact lookup, and relational reasoning respectively.
• Fusion scoring: The weighted combination of relevance, importance, and recency provides tunable ranking without heavy computation.
• Temporal invalidation: Marking outdated graph edges as invalid preserves historical context for temporal queries.
• Production ready: The rohitg00/ai-engineering-from-scratch implementation uses only standard library modules, making it immediately runnable and extensible.

Frequently Asked Questions

Why can't I use just a vector store for agent memory?

Vector stores excel at semantic similarity but fail on exact identifiers and cannot traverse relationships. When an agent needs to verify a phone number or determine billing hierarchies, vector similarity introduces noise and hallucination risks. The hybrid approach delegates each query type to the store optimized for that access pattern.

How does the fusion scorer decide which store's results to prioritize?

The fusion scorer applies the formula score = (w_relevance * relevance) + (w_importance * importance) + (w_recency * recency) to all candidates regardless of source. By adjusting the three weights at initialization (e.g., higher w_recency for chatbots, higher w_importance for compliance tools), developers can bias the system toward specific behavioral characteristics without modifying storage backends.

What is temporal invalidation and why does it matter?

Temporal invalidation is a graph store technique where outdated facts are marked invalid rather than deleted. This preserves the ability to answer questions about past states (e.g., "where did the user live in March?") while ensuring current queries only see valid edges. Without this mechanism, agents would lose historical context or require complex versioning schemas.

How does hybrid memory relate to hybrid retrieval methods like BM25 + Dense?

Both patterns address the same fundamental problem: no single algorithm satisfies all query requirements. Hybrid retrieval (BM25 + Dense) merges sparse and dense text representations for document search, while hybrid memory (Vector + KV + Graph) merges storage paradigms for agent context. Both use fusion techniques—weighted scoring or Reciprocal Rank Fusion—to unify results into a single ranked list.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}