# How to Implement User Memory and Knowledge Bases for Persistent Agent Context

> Implement user memory and knowledge bases for persistent agent context. Store interaction histories and external documents in indexes for seamless retrieval. Learn how to bridge information with a contextual pipeline.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: how-to-guide
- Published: 2026-07-25

---

**User memory and knowledge bases provide persistent agent context by storing typed interaction histories in a SQLite-backed MemoryStore and external documents in retrievable vector, sparse, or structured indexes, bridged through a contextual retrieval pipeline.**

In the `bojieli/ai-agent-book` repository, a complete architecture for persistent LLM context is implemented through two complementary subsystems. The **user memory** layer tracks personal interaction history across sessions, while the **knowledge base** (KB) layer manages externally acquired information through dense, sparse, and structured retrieval methods.

## The Two-Layer Architecture for Persistent Context

Persistent agent systems require separation between personal memory (who the user is) and general knowledge (what the agent knows). The implementation in `chapter3/` provides distinct storage mechanisms for each.

### User Memory: Typed Personal Storage

User memory captures facts derived from conversations and maintains them across sessions. According to the source in [`chapter3/memobase/agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/memobase/agent.py), the `MemoryStore` class persists data to a local SQLite database (default `./memory.db`) and supports four typed categories:

- **Episodic**: Raw conversation turns and specific events.
- **Semantic**: Distilled facts (e.g., "user prefers Python over R").
- **Procedural**: Patterns describing how the user solves problems.
- **Working**: Temporary context for the current active task.

Each `Memory` object carries an `importance_score` managed by a decay algorithm. The `decay()` method reduces scores over time, while frequent access boosts retention, ensuring high-relevance memories surface in retrieval.

### Knowledge Base: External Document Retrieval

The KB layer stores externally acquired documents and enables RAG (Retrieval-Augmented Generation). As implemented in [`chapter3/dense-embedding/indexing.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/dense-embedding/indexing.py) and related modules, three indexing strategies are supported:

- **Dense**: FAISS vector indexes for semantic similarity search.
- **Sparse**: BM25 inverted indexes for keyword-based retrieval ([`chapter3/sparse-embedding/bm25_engine.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/sparse-embedding/bm25_engine.py)).
- **Structured**: Hierarchical RAPTOR or GraphRAG indexes for complex document relationships ([`chapter3/structured-index/raptor_indexer.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/structured-index/raptor_indexer.py) and [`graphrag_indexer.py`](https://github.com/bojieli/ai-agent-book/blob/main/graphrag_indexer.py)).

Documents are chunked using fixed-window or semantic splitting before indexing, enabling precise snippet retrieval to fit context windows.

## The Retrieval Pipeline Flow

The `ContextualAgent` class orchestrates the bridge between memory and knowledge. When an incoming user message arrives, the system assembles the prompt through four stages:

1. **Context Assembly**: The agent loads static instructions, a **user-memory** snapshot (recent episodic entries weighted by importance), and **knowledge-base** snippets retrieved via the appropriate search method.
2. **Tool-Based Retrieval**: The LLM decides whether to invoke `search_user_memory`, `search_dense`, `search_sparse`, or `search_structured` based on the query intent.
3. **Result Integration**: Retrieved texts are injected into the prompt, grounding the model in both personal history and external facts.
4. **Post-Response Persistence**: After generation, the system stores new episodic memories via `MemoryStore.add_memory`, updates importance scores, and optionally enriches the KB with generated documents.

## Implementing Typed User Memory with Memobase

The `MemobaseAgent` and `MemoryStore` classes provide the primary interface for persistent user context. The following example demonstrates initializing storage, adding episodic memory, and performing typed retrieval:

```python
from chapter3.memobase.agent import MemoryStore, Memory, MemobaseAgent

# Initialise a persistent store (creates ./memory.db if missing)

store = MemoryStore(db_path="memory.db")

# Add an episodic memory

mem = store.add_memory(
    memory_type="episodic",
    content="User asked how to plot a histogram in Python",
    metadata={"session": "2024-07-25"},
    importance=2.0,
)

print(f"Added memory ID: {mem.id}")

# Retrieve recent memories matching a query

results = store.search(query="histogram python", top_k=3)
for r in results:
    print(f"{r.id}: {r.content}")

```

*Source:* [`chapter3/memobase/agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/memobase/agent.py)

The `search` method performs fast lookup against the SQLite backing, returning `Memory` objects filtered by relevance. For production deployments, the database path is configurable via [`chapter3/memobase/config.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/memobase/config.py).

## Building Retrievable Knowledge Bases

### Dense Vector Indexes

For semantic document retrieval, the `DenseEmbeddingIndex` class wraps a FAISS index. The implementation in [`chapter3/dense-embedding/indexing.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/dense-embedding/indexing.py) handles embedding generation and persistence:

```python
from chapter3.dense_embedding.indexing import DenseEmbeddingIndex
from chapter3.dense_embedding.embedding_service import EmbeddingService

# Initialise the embedding service (e.g., OpenAI or local model)

embedder = EmbeddingService(model_name="text-embedding-ada-002")

# Build the index from a list of documents

documents = [
    "The Eiffel Tower is located in Paris, France.",
    "Python was created by Guido van Rossum in 1991.",
]
index = DenseEmbeddingIndex(embedding_fn=embedder.encode)
index.build(documents)

# Persist the index for later reuse

index.save("dense_index.faiss")

```

*Source:* [`chapter3/dense-embedding/indexing.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/dense-embedding/indexing.py)

### Sparse and Structured Alternatives

For keyword-heavy queries, [`chapter3/sparse-embedding/bm25_engine.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/sparse-embedding/bm25_engine.py) provides BM25 indexing. For hierarchical or relationship-heavy data, [`chapter3/structured-index/raptor_indexer.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/structured-index/raptor_indexer.py) implements RAPTOR (Recursive Abstractive Processing for Tree-Organized Retrieval), while [`graphrag_indexer.py`](https://github.com/bojieli/ai-agent-book/blob/main/graphrag_indexer.py) offers GraphRAG capabilities. These can be swapped into the retrieval pipeline without modifying the high-level agent logic.

## Orchestrating Hybrid Retrieval

The `ContextualAgent` enables **hybrid retrieval**, combining multiple search strategies to balance precision and recall. The example below demonstrates merging user memory, dense, and sparse results before prompt injection:

```python
from chapter3.contextual_retrieval.contextual_agent import ContextualAgent
from chapter3.contextual_retrieval.tools import search_dense, search_sparse, search_user_memory

agent = ContextualAgent()

def retrieve(query: str):
    # 1. Search user memory (fast)

    mem_hits = search_user_memory(query, top_k=2)

    # 2. Search dense vector KB

    dense_hits = search_dense(query, top_k=3)

    # 3. Search sparse BM25 KB

    sparse_hits = search_sparse(query, top_k=3)

    # Merge results (simple concat, could be re-ranked)

    return mem_hits + dense_hits + sparse_hits

# Example usage

answer = agent.run(query="How do I install pandas?", retrieval_fn=retrieve)
print(answer)

```

*Source:* [`chapter3/contextual-retrieval-for-user-memory/contextual_agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/contextual-retrieval-for-user-memory/contextual_agent.py)

This multi-source approach ensures the agent retains personal conversational context while accessing authoritative external documentation.

## Summary

- **User memory** in [`chapter3/memobase/agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/memobase/agent.py) provides typed, persistent storage (episodic, semantic, procedural, working) with importance-based decay and SQLite backing.
- **Knowledge bases** support dense (FAISS), sparse (BM25), and structured (RAPTOR/GraphRAG) retrieval, implemented across `chapter3/dense-embedding/`, `sparse-embedding/`, and `structured-index/` directories.
- The **retrieval pipeline** uses `ContextualAgent` to decide between memory and KB queries, assembling hybrid context for the LLM prompt.
- All storage is **disk-persistent**, ensuring agents retain context across restarts via SQLite for memory and FAISS/JSON indexes for KB data.

## Frequently Asked Questions

### What is the difference between user memory and a knowledge base in AI agents?

User memory stores facts about the specific user and interaction history (e.g., "Alice prefers dark mode"), while the knowledge base holds external domain information (e.g., Python documentation). The former is retrieved via `search_user_memory` from `MemoryStore`, and the latter via dense, sparse, or structured search tools against document indexes.

### How does the memory decay mechanism work in the Memobase implementation?

Each `Memory` object maintains an `importance_score`. The `decay()` method reduces this score over time, simulating forgetting. When a memory is accessed frequently, its score increases, keeping it near the top of retrieval results. This biological-inspired weighting ensures only relevant, recent, or frequently used context persists in the active context window.

### Can I use multiple retrieval methods simultaneously?

Yes. The `ContextualAgent` supports hybrid retrieval by issuing multiple tool calls—such as `search_user_memory`, `search_dense`, and `search_sparse`—and merging their results before prompt assembly. This approach balances the high recall of keyword search with the semantic precision of vector embeddings.

### What storage backend is used for persisting user memory across sessions?

The `MemoryStore` class uses a local **SQLite database** (default path `./memory.db`) to persist memory entries. This lightweight backend guarantees durability across process restarts without requiring external database infrastructure, as configured in [`chapter3/memobase/config.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/memobase/config.py).