# How Data Is Managed and Stored in the AI Agent Book: File-Based Persistence Architecture

> Discover how AI Agent Book manages data with a file-based persistence architecture. Learn about JSON file storage, atomic writes, and memory modes for robust data integrity. No external databases needed.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: architecture
- Published: 2026-07-29

---

**The AI Agent Book project implements a lightweight, file-based persistence layer that stores user memories and conversation history as JSON files, utilizing atomic write operations and configurable memory modes to ensure data integrity without requiring external databases.**

According to the `bojieli/ai-agent-book` source code, this repository demonstrates practical data management patterns for AI agents through a modular Python architecture located in `chapter3/user-memory/`. The system handles user-specific data storage entirely through local JSON files, making the project portable and easy to deploy while maintaining enterprise-grade persistence guarantees.

## Core Components of the Data Storage Architecture

The data management system centers on three primary modules that govern how information is structured, accessed, and persisted.

### Memory Manager (memory_manager.py)

The **[`memory_manager.py`](https://github.com/bojieli/ai-agent-book/blob/main/memory_manager.py)** file defines the `BaseMemoryManager` abstract class and three concrete implementations: `NotesMemoryManager`, `JSONMemoryManager`, and `AdvancedJSONMemoryManager`. Each manager handles user "memories"—facts about the user that persist across sessions—using a factory function `create_memory_manager()` that instantiates the appropriate class based on `Config.MEMORY_MODE`.

Memories are stored in `Config.MEMORY_STORAGE_DIR` (defaulting to `data/memories/<user_id>_memory.json`). All manager classes implement a common interface including `load_memory()`, `save_memory()`, `add_memory()`, `update_memory()`, `delete_memory()`, and `search_memories()`.

### Conversation History (conversation_history.py)

The **`ConversationHistory`** class in [`chapter3/user-memory/conversation_history.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/user-memory/conversation_history.py) manages turn-by-turn dialogue logs. It captures user messages, assistant responses, timestamps, and session IDs, persisting them to `Config.CONVERSATION_HISTORY_DIR` (default `data/conversations/<user_id>_history.json`).

Key methods include `add_turn()` for logging new interactions, `get_recent_turns()` for retrieving context windows, `get_session_turns()` for filtering by session, and `search_history()` for full-text lookup. The class also supports optional integration with Dify's vector-search API through `DifySearchClient` when semantic retrieval is required.

### Configuration Layer (config.py)

The **`Config`** class in [`chapter3/user-memory/config.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/user-memory/config.py) serves as an in-memory singleton that defines storage paths, limits, and runtime flags. It provides constants such as `MEMORY_STORAGE_DIR`, `CONVERSATION_HISTORY_DIR`, `MAX_MEMORY_ITEMS`, and `MEMORY_MODE`. The `create_directories()` method ensures required folder structures exist before first write operations.

## Storage Formats and Memory Modes

The architecture supports three interchangeable memory formats controlled by the `Config.MEMORY_MODE` setting.

**Notes Mode** stores each entry as a `MemoryNote` object containing simple text content and optional tags. This format is ideal for unstructured, free-form user information.

**JSON Cards Mode** organizes facts into a three-level hierarchy: `category → subcategory → key → value`. This structured approach enables efficient categorical retrieval when the agent needs specific types of information.

**Advanced JSON Cards Mode** stores full card objects with rich metadata including `_metadata` and `backstory` fields, supporting complex memory schemas required for sophisticated agent behaviors.

Regardless of format, all data serializes to JSON and writes atomically using a temporary file followed by `os.replace()` to prevent corruption during system crashes.

## Data Persistence and Integrity Mechanisms

The storage layer implements several safeguards to ensure reliability across write operations.

**Atomic Write Operations** ensure that both memory and conversation data use atomic file operations. The system writes to a temporary file first, then uses `os.replace()` to move the completed file into place, eliminating the risk of half-written JSON files during power failures or process interruptions.

**Automatic Cleanup** prevents unbounded growth. The `NotesMemoryManager` enforces storage limits through `Config.MAX_MEMORY_ITEMS`. When the cap is reached, older notes are automatically pruned.

**Search Capabilities** provide retrieval across all storage types. All memory managers implement `search_memories()` for substring matching. The conversation history supports both plain text search and, when configured, vector search through Dify integration for semantic retrieval of past interactions.

## Practical Implementation Examples

Initializing the storage system requires only a user ID and the global configuration:

```python
from memory_manager import create_memory_manager

# Initialize memory manager using Config.MEMORY_MODE

mem = create_memory_manager(user_id="alice")

```

Adding memories differs slightly by mode. For **Notes Mode**:

```python

# Add a simple note with tags

note_id = mem.add_memory(
    content="Alice likes hiking in the Alps.",
    session_id="sess-01",
    tags=["hobby", "travel"]
)

# Update existing memory

mem.update_memory(
    memory_id=note_id,
    content="Alice loves hiking in the Swiss Alps.",
    session_id="sess-02"
)

```

For **JSON Cards Mode**, the same methods accept hierarchical data structures automatically formatted according to the active mode.

Retrieving context for LLM prompts:

```python

# Get formatted memory string for injection into prompts

context = mem.get_context_string()
print(context)  # Outputs formatted user memories

```

Working with conversation history:

```python
from conversation_history import ConversationHistory

hist = ConversationHistory(user_id="alice")
hist.add_turn(
    session_id="sess-01",
    user_message="What should I pack for a trek?",
    assistant_message="Bring a lightweight jacket, waterproof boots, and sun protection."
)

# Retrieve recent context window

recent = hist.get_recent_turns(limit=5)
for turn in recent:
    print(f"User: {turn.user_message}")
    print(f"Assistant: {turn.assistant_message}")

```

The [`chapter3/user-memory/conversational_agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/user-memory/conversational_agent.py) file demonstrates how these storage classes combine to create agents that recall previous interactions and user preferences across sessions.

## Summary

- **File-based JSON storage** eliminates external database dependencies by using local files in `data/memories/` and `data/conversations/`, configured through `Config.MEMORY_STORAGE_DIR` and `Config.CONVERSATION_HISTORY_DIR`.
- **Atomic write safety** guarantees data integrity through temporary file creation followed by `os.replace()` operations in both [`memory_manager.py`](https://github.com/bojieli/ai-agent-book/blob/main/memory_manager.py) and [`conversation_history.py`](https://github.com/bojieli/ai-agent-book/blob/main/conversation_history.py).
- **Mode-driven flexibility** enables three storage schemas (Notes, JSON Cards, Advanced JSON Cards) selectable via `Config.MEMORY_MODE` through the `create_memory_manager()` factory pattern.
- **Automatic maintenance** includes storage caps via `Config.MAX_MEMORY_ITEMS` and built-in search across memories and conversation history.
- **Extensible architecture** allows new storage backends by subclassing `BaseMemoryManager` and registering implementations in the factory function.

## Frequently Asked Questions

### What file format does the AI Agent Book use for data storage?

The project uses **JSON** as its exclusive serialization format. User memories store in `data/memories/<user_id>_memory.json` while conversation histories write to `data/conversations/<user_id>_history.json`, both located relative to the project root and managed through the `Config` class constants.

### How does the AI Agent Book prevent data corruption during writes?

The system implements **atomic file replacement** using Python's `os.replace()` method. When saving data, classes in [`memory_manager.py`](https://github.com/bojieli/ai-agent-book/blob/main/memory_manager.py) and [`conversation_history.py`](https://github.com/bojieli/ai-agent-book/blob/main/conversation_history.py) first write to a temporary file, then atomically move that file to the target location. This ensures that readers never encounter partially written JSON, even if the process crashes mid-operation.

### Can I switch between different memory modes without losing existing data?

Switching `Config.MEMORY_MODE` changes which manager class `create_memory_manager()` instantiates, but the file paths remain determined by `Config.MEMORY_STORAGE_DIR`. While existing JSON files persist on disk, switching modes may cause compatibility issues since **Notes Mode**, **JSON Cards Mode**, and **Advanced JSON Cards Mode** each expect specific schema structures in [`memory_manager.py`](https://github.com/bojieli/ai-agent-book/blob/main/memory_manager.py).

### How is conversation history retrieved for LLM context windows?

The `ConversationHistory` class provides `get_recent_turns(limit=N)` to fetch the latest N dialogue turns as objects containing `user_message` and `assistant_message` attributes. The example implementation in [`chapter3/user-memory/conversational_agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/user-memory/conversational_agent.py) demonstrates combining this with `MemoryManager.get_context_string()` to provide the model with both historical facts and recent dialogue context.