How to Implement User Memory Across Sessions in AI Agents Using RAG and Knowledge Graphs

Implement persistent cross-session memory by extracting JSON memory cards from conversations, indexing them in a dual dense-sparse vector store, and enriching retrieval with a structured knowledge graph that encodes entity relationships.

The bojieli/ai-agent-book repository provides a production-grade reference architecture for maintaining user context across multiple conversations. By combining Retrieval-Augmented Generation (RAG) with knowledge graph reasoning, you can build AI agents that remember preferences, identifiers, and interaction history beyond a single session.

Architecture Overview

The hierarchical memory system organizes context into three persistent layers. The User-Memory Store maintains JSON-based memory cards extracted after each turn, as configured in [chapter3/user-memory/config.py](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/user-memory/config.py). The RAG Retrieval layer combines FAISS vector search with BM25 sparse indexing to fetch relevant historical facts. Finally, the Knowledge-Graph Layer structures entity relationships using GraphRAG or RAPTOR indexes, enabling relational reasoning that pure vector search cannot capture.

This architecture feeds into the ReAct planner, which queries both the vector store and knowledge graph before generating responses, as demonstrated in the learning documentation.

Extracting and Storing Memory Cards

Memory Extraction

After each assistant turn, the agent extracts structured facts about the user into "memory cards." In [chapter3/user-memory/agent.py](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/user-memory/agent.py), the system prompts the LLM to summarize preferences and identifiers into JSON format:

def extract_memory_card(conversation: List[Dict[str, str]]) -> Dict[str, Any]:
    # Prompt the LLM to summarise user‑relevant facts

    prompt = f"""
    Summarise the user's preferences, identifiers and recent actions from the
    following dialogue history (JSON list of messages):
    {json.dumps(conversation)}
    Output a JSON object with keys: name, preferences, recent_actions.
    """
    response = llm.complete(prompt)               # ← uses the same LLM as the agent

    return json.loads(response)

Persistent Storage

The extracted cards append to a deterministic log file defined in [chapter3/user-memory/config.py](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/user-memory/config.py). The LOG_FILE environment variable controls the storage location:

import os, json

LOG_FILE = os.getenv("LOG_FILE", "logs/user_memory.log")

def append_memory(card: Dict[str, Any]) -> None:
    with open(LOG_FILE, "a") as f:
        f.write(json.dumps(card) + "\n")

A background process periodically aggregates these logs into searchable indexes, preventing unbounded growth while ensuring new facts become retrievable immediately.

Building the Dual-Index RAG System

Dense and Sparse Indexing

The RAG implementation in [chapter3/memory_rag_campaign.py](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/memory_rag_campaign.py) creates both dense (FAISS) and sparse (BM25) indexes over the aggregated memory cards. This dual approach handles semantic similarity and exact keyword matches:

from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import FAISS
from langchain.retrievers import BM25Retriever

def build_indexes(log_path: str):
    # Load all JSON cards

    cards = [json.loads(l) for l in open(log_path)]
    texts = [json.dumps(c) for c in cards]

    # Dense index

    embed = OpenAIEmbeddings()
    dense_index = FAISS.from_texts(texts, embed)

    # Sparse BM25 index

    sparse_index = BM25Retriever.from_texts(texts)

    return dense_index, sparse_index

ReAct Tool Integration

The search_user_memory tool registered in [chapter3/memory_rag_campaign.py](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/memory_rag_campaign.py) merges results from both indexes:

def search_user_memory(query: str, dense, sparse, top_k: int = 5):
    # Dense retrieval

    dense_hits = dense.similarity_search(query, k=top_k)

    # Sparse retrieval (fallback)

    sparse_hits = sparse.get_relevant_documents(query)[:top_k]

    # Merge and dedupe

    results = {hit.metadata["source"]: hit for hit in dense_hits + sparse_hits}
    return list(results.values())[:top_k]

Integrating Knowledge Graphs

Graph-Based Reasoning

To capture relationships between entities (e.g., "user-lives-in-city → city-has-climate"), the system integrates a knowledge graph layer. The [chapter3/agentic-rag-for-user-memory/evaluator.py](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/agentic-rag-for-user-memory/evaluator.py) demonstrates GraphRAG evaluation:

def query_knowledge_graph(subject: str, predicate: str = None):
    # Simple triple store lookup; replace with RAPTOR / GraphRAG client

    triples = KG_STORE.get_triples(subject)
    if predicate:
        return [t for t in triples if t.predicate == predicate]
    return triples

Entity Linking

When processing memory cards, the system extracts entities and links them to graph nodes. This enables traversal from user preferences to related world knowledge, enhancing the agent's reasoning capabilities beyond the stored conversation history.

Orchestrating the Full ReAct Loop

The evaluation harness in [chapter3/user-memory-evaluation/main.py](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/user-memory-evaluation/main.py) demonstrates the complete integration. The planner retrieves relevant memory snippets, queries the knowledge graph for relational facts, and augments the system prompt:

def run_user_query(user_query: str):
    # 1️⃣ Retrieve relevant memory

    mem_hits = search_user_memory(user_query, dense_idx, sparse_idx)

    # 2️⃣ Optionally enrich with graph facts

    graph_facts = []
    for hit in mem_hits:
        ent = extract_entities(hit.page_content)        # e.g., "city", "cuisine"

        graph_facts.extend(query_knowledge_graph(ent))

    # 3️⃣ Build augmented prompt

    augmented_prompt = f"""
    You are a helpful assistant. Use the following user memory snippets:
    {format_hits(mem_hits)}
    And these knowledge‑graph facts:
    {format_facts(graph_facts)}
    Answer the user's question: {user_query}
    """

    # 4️⃣ Generate response

    return llm.complete(augmented_prompt)

Summary

• Extract structured memory cards after each turn using LLM-based summarization, storing them as JSON in a persistent log configured in [config.py](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/user-memory/config.py).
• Index the aggregated cards using both dense (FAISS) and sparse (BM25) retrieval to balance semantic similarity with keyword precision, as implemented in [memory_rag_campaign.py](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/memory_rag_campaign.py).
• Enrich retrieval results by querying a knowledge graph for entity relationships, demonstrated in [evaluator.py](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/agentic-rag-for-user-memory/evaluator.py).
• Orchestrate the complete pipeline through a ReAct planner that concatenates memory snippets and graph facts into the context window before generation.

Frequently Asked Questions

How does the dual-index RAG system improve memory retrieval?

The dual-index approach combines FAISS for semantic vector search and BM25 for sparse keyword matching. Dense retrieval captures conceptual similarity between user queries and memory cards, while BM25 handles exact matches for names, dates, and identifiers that vector embeddings might miss. This hybrid strategy ensures robust recall across different query types, as configured in [chapter3/memory_rag_campaign.py](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/memory_rag_campaign.py).

What is the role of the knowledge graph when vector search already retrieves memories?

While vector search retrieves semantically similar memory cards, it cannot infer indirect relationships between entities. The knowledge graph stores structured triples (subject-predicate-object) that enable traversal from user preferences to related facts—for example, determining climate-appropriate recommendations based on the user's city. This relational reasoning complements the similarity-based retrieval in [chapter3/agentic-rag-for-user-memory/evaluator.py](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/agentic-rag-for-user-memory/evaluator.py).

How do you prevent the memory log from growing indefinitely?

The system uses a background memory processor that periodically rewrites the accumulated JSON logs into consolidated indexes. You should configure log rotation via the LOG_FILE environment variable and implement cleanup policies in the background processor to archive or compress old entries while maintaining the searchable vector indexes in FAISS.

Can this architecture handle multiple users simultaneously?

Yes, by prefixing memory card identifiers with unique user IDs and maintaining separate FAISS indexes or metadata filters per user. The [chapter3/user-memory/config.py](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/user-memory/config.py) file allows dynamic configuration of storage paths, enabling multi-tenant deployments where each user's memory isolates to distinct log files and vector namespaces.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}