How to Implement User Memory and Knowledge Bases for Persistent Agent Context

User memory and knowledge bases provide persistent agent context by storing typed interaction histories in a SQLite-backed MemoryStore and external documents in retrievable vector, sparse, or structured indexes, bridged through a contextual retrieval pipeline.

In the bojieli/ai-agent-book repository, a complete architecture for persistent LLM context is implemented through two complementary subsystems. The user memory layer tracks personal interaction history across sessions, while the knowledge base (KB) layer manages externally acquired information through dense, sparse, and structured retrieval methods.

The Two-Layer Architecture for Persistent Context

Persistent agent systems require separation between personal memory (who the user is) and general knowledge (what the agent knows). The implementation in chapter3/ provides distinct storage mechanisms for each.

User Memory: Typed Personal Storage

User memory captures facts derived from conversations and maintains them across sessions. According to the source in chapter3/memobase/agent.py, the MemoryStore class persists data to a local SQLite database (default ./memory.db) and supports four typed categories:

• Episodic: Raw conversation turns and specific events.
• Semantic: Distilled facts (e.g., "user prefers Python over R").
• Procedural: Patterns describing how the user solves problems.
• Working: Temporary context for the current active task.

Each Memory object carries an importance_score managed by a decay algorithm. The decay() method reduces scores over time, while frequent access boosts retention, ensuring high-relevance memories surface in retrieval.

Knowledge Base: External Document Retrieval

The KB layer stores externally acquired documents and enables RAG (Retrieval-Augmented Generation). As implemented in chapter3/dense-embedding/indexing.py and related modules, three indexing strategies are supported:

• Dense: FAISS vector indexes for semantic similarity search.
• Sparse: BM25 inverted indexes for keyword-based retrieval (chapter3/sparse-embedding/bm25_engine.py).
• Structured: Hierarchical RAPTOR or GraphRAG indexes for complex document relationships (chapter3/structured-index/raptor_indexer.py and graphrag_indexer.py).

Documents are chunked using fixed-window or semantic splitting before indexing, enabling precise snippet retrieval to fit context windows.

The Retrieval Pipeline Flow

The ContextualAgent class orchestrates the bridge between memory and knowledge. When an incoming user message arrives, the system assembles the prompt through four stages:

1. Context Assembly: The agent loads static instructions, a user-memory snapshot (recent episodic entries weighted by importance), and knowledge-base snippets retrieved via the appropriate search method.
2. Tool-Based Retrieval: The LLM decides whether to invoke search_user_memory, search_dense, search_sparse, or search_structured based on the query intent.
3. Result Integration: Retrieved texts are injected into the prompt, grounding the model in both personal history and external facts.
4. Post-Response Persistence: After generation, the system stores new episodic memories via MemoryStore.add_memory, updates importance scores, and optionally enriches the KB with generated documents.

Implementing Typed User Memory with Memobase

The MemobaseAgent and MemoryStore classes provide the primary interface for persistent user context. The following example demonstrates initializing storage, adding episodic memory, and performing typed retrieval:

from chapter3.memobase.agent import MemoryStore, Memory, MemobaseAgent

# Initialise a persistent store (creates ./memory.db if missing)

store = MemoryStore(db_path="memory.db")

# Add an episodic memory

mem = store.add_memory(
    memory_type="episodic",
    content="User asked how to plot a histogram in Python",
    metadata={"session": "2024-07-25"},
    importance=2.0,
)

print(f"Added memory ID: {mem.id}")

# Retrieve recent memories matching a query

results = store.search(query="histogram python", top_k=3)
for r in results:
    print(f"{r.id}: {r.content}")

Source: chapter3/memobase/agent.py

The search method performs fast lookup against the SQLite backing, returning Memory objects filtered by relevance. For production deployments, the database path is configurable via chapter3/memobase/config.py.

Building Retrievable Knowledge Bases

Dense Vector Indexes

For semantic document retrieval, the DenseEmbeddingIndex class wraps a FAISS index. The implementation in chapter3/dense-embedding/indexing.py handles embedding generation and persistence:

from chapter3.dense_embedding.indexing import DenseEmbeddingIndex
from chapter3.dense_embedding.embedding_service import EmbeddingService

# Initialise the embedding service (e.g., OpenAI or local model)

embedder = EmbeddingService(model_name="text-embedding-ada-002")

# Build the index from a list of documents

documents = [
    "The Eiffel Tower is located in Paris, France.",
    "Python was created by Guido van Rossum in 1991.",
]
index = DenseEmbeddingIndex(embedding_fn=embedder.encode)
index.build(documents)

# Persist the index for later reuse

index.save("dense_index.faiss")

Source: chapter3/dense-embedding/indexing.py

Sparse and Structured Alternatives

For keyword-heavy queries, chapter3/sparse-embedding/bm25_engine.py provides BM25 indexing. For hierarchical or relationship-heavy data, chapter3/structured-index/raptor_indexer.py implements RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval), while graphrag_indexer.py offers GraphRAG capabilities. These can be swapped into the retrieval pipeline without modifying the high-level agent logic.

Orchestrating Hybrid Retrieval

The ContextualAgent enables hybrid retrieval, combining multiple search strategies to balance precision and recall. The example below demonstrates merging user memory, dense, and sparse results before prompt injection:

from chapter3.contextual_retrieval.contextual_agent import ContextualAgent
from chapter3.contextual_retrieval.tools import search_dense, search_sparse, search_user_memory

agent = ContextualAgent()

def retrieve(query: str):
    # 1. Search user memory (fast)

    mem_hits = search_user_memory(query, top_k=2)

    # 2. Search dense vector KB

    dense_hits = search_dense(query, top_k=3)

    # 3. Search sparse BM25 KB

    sparse_hits = search_sparse(query, top_k=3)

    # Merge results (simple concat, could be re-ranked)

    return mem_hits + dense_hits + sparse_hits

# Example usage

answer = agent.run(query="How do I install pandas?", retrieval_fn=retrieve)
print(answer)

Source: chapter3/contextual-retrieval-for-user-memory/contextual_agent.py

This multi-source approach ensures the agent retains personal conversational context while accessing authoritative external documentation.

Summary

• User memory in chapter3/memobase/agent.py provides typed, persistent storage (episodic, semantic, procedural, working) with importance-based decay and SQLite backing.
• Knowledge bases support dense (FAISS), sparse (BM25), and structured (RAPTOR/GraphRAG) retrieval, implemented across chapter3/dense-embedding/, sparse-embedding/, and structured-index/ directories.
• The retrieval pipeline uses ContextualAgent to decide between memory and KB queries, assembling hybrid context for the LLM prompt.
• All storage is disk-persistent, ensuring agents retain context across restarts via SQLite for memory and FAISS/JSON indexes for KB data.

Frequently Asked Questions

What is the difference between user memory and a knowledge base in AI agents?

User memory stores facts about the specific user and interaction history (e.g., "Alice prefers dark mode"), while the knowledge base holds external domain information (e.g., Python documentation). The former is retrieved via search_user_memory from MemoryStore, and the latter via dense, sparse, or structured search tools against document indexes.

How does the memory decay mechanism work in the Memobase implementation?

Each Memory object maintains an importance_score. The decay() method reduces this score over time, simulating forgetting. When a memory is accessed frequently, its score increases, keeping it near the top of retrieval results. This biological-inspired weighting ensures only relevant, recent, or frequently used context persists in the active context window.

Can I use multiple retrieval methods simultaneously?

Yes. The ContextualAgent supports hybrid retrieval by issuing multiple tool calls—such as search_user_memory, search_dense, and search_sparse—and merging their results before prompt assembly. This approach balances the high recall of keyword search with the semantic precision of vector embeddings.

What storage backend is used for persisting user memory across sessions?

The MemoryStore class uses a local SQLite database (default path ./memory.db) to persist memory entries. This lightweight backend guarantees durability across process restarts without requiring external database infrastructure, as configured in chapter3/memobase/config.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}