# How OpenHuman Stores and Retrieves Hierarchical Knowledge in Its Memory Tree Using SQLite

> Learn how OpenHuman stores and retrieves hierarchical knowledge in its Memory Tree using SQLite. Explore parent-child structures and Rust APIs for efficient data management.

- Repository: [Tiny Humans/openhuman](https://github.com/tinyhumansai/openhuman)
- Tags: internals
- Published: 2026-05-12

---

**OpenHuman persists hierarchical knowledge in a single SQLite database file using a parent-child tree structure where atomic knowledge units called "chunks" reference ancestors via a nullable `parent_id` column, enabling bidirectional traversal through Rust APIs that drill down to roots or fetch leaf descendants.**

The **Memory Tree** in the OpenHuman project (available at `tinyhumansai/openhuman`) provides a persistent, embedded knowledge store implemented in Rust. Unlike flat vector databases, this architecture treats knowledge as a true hierarchy where internal nodes aggregate summaries and embeddings while leaf nodes store raw content, all persisted within a local SQLite file named `chunks.db`.

## SQLite Schema Design for Hierarchical Storage

The Memory Tree schema implements a classical adjacency list model optimized for local-first AI applications, supporting both graph traversal and vector similarity search within transactional boundaries.

### Physical Database Layout

All hierarchical data resides in `<workspace>/memory_tree/chunks.db`. The database connection is managed through `openhuman::memory::tree::store::with_connection` in [`src/openhuman/memory/tree/store.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/memory/tree/store.rs), which configures **Write-Ahead Logging (WAL) mode** for concurrent read/write performance and implements automatic retry logic on SQLite "busy" errors.

### Core Tables and Relationships

The `store::initialize_schema` function creates five interlinked tables that separate content from metadata and search indices:

- **`chunks`**: The primary tree structure storing `id` (primary key), `source_kind`, `source_id`, `owner`, `status`, `content_path`, and the nullable `parent_id` that establishes hierarchy.
- **`raw_refs`**: Byte-range pointers (`path`, `start`, `end`) linking chunks to original source files.
- **`embeddings`**: Vector storage for similarity search, containing `chunk_id` and float arrays.
- **`summaries`**: LLM-generated condensed representations stored at `summary_path`.
- **`metadata`**: Implicit status tracking through column values like `CHUNK_STATUS_PENDING_EXTRACTION`, `CHUNK_STATUS_ADMITTED`, and `CHUNK_STATUS_SEALED`.

### The parent_id Tree Mechanism

**Hierarchical relationships** are encoded through the `parent_id` foreign key in the `chunks` table. When `parent_id` is `NULL`, the chunk represents a root node; when populated, it references another row in the same table. This design enables:

- **Leaf chunks**: Terminal nodes containing raw text or file fragments.
- **Internal chunks**: Aggregated nodes holding summaries and embeddings of their children.

The schema applies lazily on first open with migration logic logged at `src/openhuman/memory/tree/store.rs:660`.

## Storing Knowledge: The Ingestion Pipeline

Knowledge ingestion follows a strict lifecycle from raw data to indexed, queryable nodes, all wrapped in ACID transactions via `with_connection`.

### Inserting Chunks with upsert_chunks

New knowledge enters the system through `store::upsert_chunks`, defined in `src/openhuman/memory/tree/store.rs:242`. This function accepts a slice of `Chunk` structs and performs parameterized inserts using `rusqlite::params!` to prevent SQL injection.

```rust
use openhuman::memory::tree::{store, Chunk, Config};

let cfg = Config::load()?;
let new_chunk = Chunk::new(
    "file://notes/meeting.md",
    "Meeting notes on AI-driven knowledge graphs".into(),
);
store::upsert_chunks(&cfg, &[new_chunk])?;

```

### Lifecycle Status Management

Chunks progress through states managed by `store::set_chunk_lifecycle_status`:

1. **`pending_extraction`**: Initial state after insertion.
2. **`admitted`**: Intermediate state after validation.
3. **`sealed`**: Final immutable state ready for querying.

This state machine prevents incomplete data from appearing in search results while embeddings are being computed.

### Embedding and Vector Indexing

After chunk creation, the system computes embeddings via external LLM services and persists them using `store::set_chunk_embedding` (located at `store.rs:912`):

```rust
let embedding = my_embedder.compute(&new_chunk.content)?;
store::set_chunk_embedding(&cfg, &new_chunk.id, &embedding)?;

```

The embedding is stored in the `embeddings` table while the chunk's `status` column transitions toward `sealed`.

## Retrieving Hierarchical Data: Query Patterns

OpenHuman exposes retrieval through the `memory_tree` skill, dispatching to specialized modules in `src/openhuman/memory/tree/retrieval/` that issue compiled SQL queries.

### Drill-Down: Traversing to Root Ancestors

The `memory_tree_drill_down` tool invokes `retrieval::drill_down::drill_down` in [`src/openhuman/memory/tree/retrieval/drill_down.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/memory/tree/retrieval/drill_down.rs). This function recursively walks **up** the tree from a leaf node to the root via `parent_id` joins, returning the complete ancestry path for context assembly:

```rust
use openhuman::memory::tree::retrieval::drill_down::drill_down;

let leaf_id = "chunk-23";
let path = drill_down(&cfg, leaf_id)?; // Returns Vec<Chunk> from leaf → root

```

### Fetch Leaves: Collecting Descendants

Conversely, `memory_tree_fetch_leaves` triggers `retrieval::fetch::fetch_leaves` ([`fetch.rs`](https://github.com/tinyhumansai/openhuman/blob/main/fetch.rs)), which walks **down** from a root ID to collect all terminal nodes:

```rust
use openhuman::memory::tree::retrieval::fetch::fetch_leaves;

let root_id = "chunk-root-1";
let leaves = fetch_leaves(&cfg, &[root_id])?; // Returns Vec<Chunk>

```

This uses parameterized queries like `SELECT … FROM chunks WHERE parent_id = ?` to traverse subtrees without loading the entire graph into memory.

### Vector and Full-Text Search Integration

OpenHuman combines hierarchical traversal with semantic search:

- **`retrieval::search::search_entities`**: Implements FTS5 full-text search over `chunks.content_path` for the `memory_tree_search_entities` tool.
- **`retrieval::global::query_topic`**: Performs embedding similarity queries via the `memory_tree_query_topic` tool to locate the most relevant entry point in the tree for a given topic.

These retrieval methods work hierarchically—first finding a relevant node via vector similarity, then using `drill_down` or `fetch_leaves` to gather complete context.

## Summary

- **Storage**: Hierarchical knowledge is stored in `<workspace>/memory_tree/chunks.db` using an adjacency list pattern via the `parent_id` column in the `chunks` table.
- **Schema**: Five core tables (`chunks`, `raw_refs`, `embeddings`, `summaries`, `metadata`) separate content, vectors, and provenance.
- **Lifecycle**: Chunks progress from `pending_extraction` to `sealed` via `store::set_chunk_lifecycle_status`, ensuring data consistency.
- **Traversal**: `drill_down` walks upward to roots; `fetch_leaves` collects terminal descendants; both use parameterized SQL with WAL-mode transactions.
- **Search**: Hybrid retrieval combining FTS5 text search, vector similarity, and tree traversal.

## Frequently Asked Questions

### How does OpenHuman prevent SQLite contention during concurrent writes?

The `store::with_connection` wrapper configures SQLite with **WAL (Write-Ahead Logging) mode** and implements automatic retry logic when the database returns "busy" errors. All chunk operations—including `upsert_chunks` and `set_chunk_embedding`—execute within transactions that retry on contention, ensuring atomic updates to both the `chunks` table and its associated `embeddings` rows without blocking readers.

### What is the difference between "fetch_leaves" and "drill_down" in the Memory Tree?

**`fetch_leaves`** traverses downward from a parent chunk to collect all descendant terminal nodes (children with no further children), useful for gathering all raw source material under a topic summary. **`drill_down`** traverses upward from a specific chunk to collect all ancestors back to the root, used to assemble hierarchical context for LLM prompts. Both operate via recursive SQL queries on the `parent_id` column without loading the entire tree into RAM.

### How are embeddings stored while maintaining hierarchical relationships?

Embeddings are stored in a separate **`embeddings`** table linked by `chunk_id`, keeping the `chunks` table row size manageable for tree traversal queries. When `store::set_chunk_embedding` is called, it updates this external table while the `chunks` table maintains the `parent_id` hierarchy. This separation allows vector similarity searches to return chunk IDs that are then passed to `drill_down` or `fetch_leaves` for hierarchical context retrieval.

### Can the Memory Tree handle file renames or moves while preserving hierarchy?

Yes. The **`raw_refs`** table stores byte-range pointers (`path`, `start`, `end`) independently of the hierarchical structure in `chunks`. If a file moves, updating the `content_path` in the `chunks` table and the `path` values in `raw_refs` preserves the entire `parent_id` tree structure and all computed embeddings, as the chunk identities remain stable UUIDs referenced by foreign keys.