# MemPalace Agent Diary System and Cross-Wing Navigation: Complete Technical Guide

> Explore the MemPalace agent diary system and cross-wing navigation. Learn how this local-first architecture uses hierarchical drawers and graph tunnels for semantic concept connections across projects.

- Repository: [MemPalace/mempalace](https://github.com/MemPalace/mempalace)
- Tags: technical-guide
- Published: 2026-06-07

---

**The MemPalace agent diary system is a local-first verbatim-memory architecture that stores every user interaction as hierarchical drawers (Wings → Rooms → Drawers) and enables semantic cross-wing navigation through a graph-based tunnel system connecting related concepts across different project boundaries.**

MemPalace (`mempalace`) implements a privacy-preserving memory palace pattern to organize code and text into navigable structures. The **agent diary system** records every word a user shares into versioned drawers, while **cross-wing navigation** provides both implicit discovery (via shared entities) and explicit linking (via user-created tunnels) to traverse the knowledge graph. This article examines the source code implementation in [`mempalace/palace.py`](https://github.com/MemPalace/mempalace/blob/main/mempalace/palace.py) and [`mempalace/palace_graph.py`](https://github.com/MemPalace/mempalace/blob/main/mempalace/palace_graph.py) to demonstrate how agents store memories and navigate between disconnected wings.

## How the Agent Diary System Stores Verbatim Memory

The ingestion layer in [`mempalace/palace.py`](https://github.com/MemPalace/mempalace/blob/main/mempalace/palace.py) handles **atomic mining operations** that transform raw files into searchable memory structures. When an agent or user initiates a mining session, the system splits content into drawers and generates compact index pointers called **closet lines**.

### Drawer Versioning and Locking

To prevent concurrent mines from corrupting the HNSW graph in vector backends like ChromaDB, MemPalace implements filesystem-level locking:

```python
from mempalace.palace import mine_palace_lock, get_collection

palace_path = "/home/alice/.mempalace"
with mine_palace_lock(palace_path):
    col = get_collection(palace_path)          # resolves backend automatically

    # … mining operations proceed atomically …

```

The `mine_palace_lock` function creates a palace-wide lockfile, while `mine_lock` provides granular drawer-level protection. The system checks `file_already_mined` to avoid reprocessing unchanged content and uses `NORMALIZE_VERSION` constants to handle drawer format migrations.

### Closet Index Generation

The `build_closet_lines` function (lines 82-112 in [`palace.py`](https://github.com/MemPalace/mempalace/blob/main/palace.py)) extracts entity candidates, filters stop-words, and emits pointer lines in the format `topic|entities|→drawer_ids` or the Tier-6a variant including date and line ranges (`topic|entities|YYYY-MM-DD:Lstart-Lend|→drawer_ids`). These pointers enable O(1) lookup without scanning the entire vector corpus:

```python
lines = build_closet_lines(source_file, drawer_ids, content, wing, room)
upsert_closet_lines(closets_col, closet_id_base, lines, metadata)

```

## Cross-Wing Navigation Architecture

MemPalace organizes memory hierarchically: **Wings** represent top-level projects or domains, **Rooms** contain related drawers, and **Drawers** store the actual verbatim text chunks. The navigation layer treats these as a graph where edges represent navigable pathways between rooms.

### Graph Construction and Caching

The `build_graph` function in [`mempalace/palace_graph.py`](https://github.com/MemPalace/mempalace/blob/main/mempalace/palace_graph.py) scans the `mempalace_closets` collection to aggregate per-room metadata and construct nodes and edges. To ensure fast repeated access, the module maintains `_graph_cache_*` variables providing O(1) graph lookups; cache invalidation occurs atomically via `invalidate_graph_cache` on write operations.

```python
nodes, edges = build_graph(col, config)

```

### Implicit Tunnels and Graph Traversal

**Implicit tunnels** emerge naturally when rooms appear in multiple wings. The `traverse` function performs breadth-first search across these shared relationships:

```python
results = traverse("my_idea", max_hops=3)

```

This returns paths with hop distances, allowing agents to discover connections between seemingly unrelated project wings. The `find_tunnels` function (lines 151-176) specifically identifies rooms that act as hallways between wings:

```python
tunnels = find_tunnels(wing_a="project_api", wing_b="project_db")

```

## Creating Explicit Tunnels with `create_tunnel`

Agents and users can forge **explicit, symmetric tunnels** linking any two `(wing, room)` pairs. These diary entries represent deliberate associations between concepts and persist in [`tunnels.json`](https://github.com/MemPalace/mempalace/blob/main/tunnels.json) within the palace directory.

### Tunnel Structure and Dynamics

Each tunnel receives a **canonical ID** generated as a deterministic hash of the ordered endpoint strings. The `create_tunnel` function (lines 488-557) attaches dynamic metadata via `initialize_dynamics_fields`, tracking:

- **Strength**: Link utilization frequency
- **Stability**: Persistence confidence
- **Activation timestamps**: Last traversal time

```python
from mempalace.palace_graph import create_tunnel

t = create_tunnel(
    source_wing="project_api",
    source_room="utils",
    target_wing="project_db",
    target_room="models",
    label="Shared utility code",
    kind="explicit",
)
print("Tunnel created:", t["id"])

```

### Atomic Operations and Race Condition Prevention

All tunnel mutations use **atomic read-modify-write** cycles protected by `mine_lock`. This ensures that parallel agents creating tunnels simultaneously do not corrupt the [`tunnels.json`](https://github.com/MemPalace/mempalace/blob/main/tunnels.json) structure.

## Automatic Topic and Entity Tunnels

Beyond manual links, MemPalace generates **automatic tunnels** based on shared semantic content. These function identically to explicit tunnels in storage format but are computed algorithmically.

### Topic Tunnel Generation

The `compute_topic_tunnels` function (lines 643-724) analyzes topic distributions across wings and creates synthetic rooms prefixed with `topic:`. When two wings share a topic like "REST", the system automatically links them via a `topic:REST` node:

```python
from mempalace.palace_graph import compute_topic_tunnels

topics_by_wing = {
    "project_api": ["REST", "OAuth"],
    "project_db": ["REST", "SQL"],
}
new_tunnels = compute_topic_tunnels(topics_by_wing, min_count=1)

```

### Entity-Based Navigation

Similarly, `entity_tunnels_for_wing` (lines 785-845) detects person and project entities using [`entity_detector.py`](https://github.com/MemPalace/mempalace/blob/main/entity_detector.py), creating `entity:<name>` tunnels when the same entity appears in multiple wings. Both automatic functions call `create_tunnel` internally, ensuring a unified storage schema.

## Working with the Navigation API

Agents interact with the tunnel network through the `follow_tunnels` API, which retrieves both outgoing and incoming explicit links with optional preview snippets:

```python
from mempalace.palace_graph import follow_tunnels, _get_collection

col = _get_collection()          # get the underlying collection

connections = follow_tunnels("project_api", "utils", col=col)
for c in connections:
    print(f"{c['direction']} → {c['connected_wing']}/{c['connected_room']}")

```

### CLI and MCP Server Integration

The navigation system exposes identical functionality through both command-line and RPC interfaces. The `mempalace graph stats` command (implemented in [`cli.py`](https://github.com/MemPalace/mempalace/blob/main/cli.py)) returns graph metrics:

```bash
$ mempalace graph stats
{
  "total_rooms": 1824,
  "tunnel_rooms": 127,
  "total_edges": 342,
  "rooms_per_wing": {"project_api": 312, "project_db": 298, ...}
}

```

For external agents like Claude Code, [`mcp_server.py`](https://github.com/MemPalace/mempalace/blob/main/mcp_server.py) provides local HTTP-tool endpoints mirroring the Python API, while [`hooks/mempal_save_hook.sh`](https://github.com/MemPalace/mempalace/blob/main/hooks/mempal_save_hook.sh) demonstrates triggering diary saves on agent termination.

## Concurrency and Data Integrity

MemPalace implements a sophisticated backend resolution system (`resolve_backend_name` in [`palace.py`](https://github.com/MemPalace/mempalace/blob/main/palace.py)) that selects between ChromaDB, Qdrant, PGVector, or SQLite-Exact based on configuration flags, environment variables, or detected artifacts. Regardless of backend, all write operations respect the locking hierarchy:

1. **Palace-level lock** (`mine_palace_lock`): Prevents concurrent mining sessions
2. **Collection-level lock** (`mine_lock`): Protects individual drawer operations
3. **Graph cache invalidation**: Ensures navigation queries see consistent state

## Summary

- **Verbatim storage**: Every user input becomes a versioned drawer in a hierarchical Wing → Room → Drawer structure managed by [`mempalace/palace.py`](https://github.com/MemPalace/mempalace/blob/main/mempalace/palace.py).
- **Cross-wing navigation**: The graph system in [`mempalace/palace_graph.py`](https://github.com/MemPalace/mempalace/blob/main/mempalace/palace_graph.py) provides BFS traversal (`traverse`) and tunnel discovery (`find_tunnels`) across project boundaries.
- **Explicit tunnels**: Agents create durable links between concepts using `create_tunnel`, storing deterministic IDs and dynamic metadata (strength, stability) in [`tunnels.json`](https://github.com/MemPalace/mempalace/blob/main/tunnels.json).
- **Automatic discovery**: Shared topics and entities generate implicit tunnels via `compute_topic_tunnels` and `entity_tunnels_for_wing`, surfacing hidden relationships without manual tagging.
- **Local-first privacy**: All vector storage (default ChromaDB), graph caching, and tunnel persistence occur on-device with no external API dependencies.

## Frequently Asked Questions

### How does the MemPalace agent diary system prevent data corruption during concurrent mining?

The system implements hierarchical filesystem locks through `mine_palace_lock` (palace-wide) and `mine_lock` (operation-level) in [`mempalace/palace.py`](https://github.com/MemPalace/mempalace/blob/main/mempalace/palace.py). These locks prevent race conditions when multiple agents simultaneously write drawers or create tunnels, ensuring the HNSW graph and [`tunnels.json`](https://github.com/MemPalace/mempalace/blob/main/tunnels.json) remain consistent. Atomic read-modify-write cycles protect the tunnel registry during explicit link creation.

### What is the difference between implicit and explicit tunnels in cross-wing navigation?

**Implicit tunnels** emerge automatically when rooms appear in multiple wings or share topics/entities; they are discovered via `find_tunnels` and `traverse` without persistent storage. **Explicit tunnels** are user or agent-created via `create_tunnel`, stored permanently in [`tunnels.json`](https://github.com/MemPalace/mempalace/blob/main/tunnels.json) with canonical IDs and dynamic fields (strength, stability), and retrieved through `follow_tunnels` for deliberate navigation pathways.

### Can MemPalace work with vector backends other than ChromaDB?

Yes. While ChromaDB is the default, [`mempalace/palace.py`](https://github.com/MemPalace/mempalace/blob/main/mempalace/palace.py) includes a `resolve_backend_name` function that detects and configures alternative backends including Qdrant, PGVector, and SQLite-Exact. The backend abstraction in `mempalace/backends/` ensures the closet indexing and tunnel graph systems operate identically regardless of the underlying vector store.

### How do automatic topic tunnels improve navigation efficiency?

The `compute_topic_tunnels` function analyzes topic distributions across wings and creates synthetic `topic:<name>` rooms (e.g., `topic:REST`) linking all wings mentioning that topic. This prevents agents from performing expensive full-text scans when cross-referencing concepts, instead providing O(1) graph traversals through pre-computed tunnel edges.