How OpenHuman Stores and Retrieves Hierarchical Knowledge in Its Memory Tree Using SQLite

OpenHuman persists hierarchical knowledge in a single SQLite database file using a parent-child tree structure where atomic knowledge units called "chunks" reference ancestors via a nullable parent_id column, enabling bidirectional traversal through Rust APIs that drill down to roots or fetch leaf descendants.

The Memory Tree in the OpenHuman project (available at tinyhumansai/openhuman) provides a persistent, embedded knowledge store implemented in Rust. Unlike flat vector databases, this architecture treats knowledge as a true hierarchy where internal nodes aggregate summaries and embeddings while leaf nodes store raw content, all persisted within a local SQLite file named chunks.db.

SQLite Schema Design for Hierarchical Storage

The Memory Tree schema implements a classical adjacency list model optimized for local-first AI applications, supporting both graph traversal and vector similarity search within transactional boundaries.

Physical Database Layout

All hierarchical data resides in <workspace>/memory_tree/chunks.db. The database connection is managed through openhuman::memory::tree::store::with_connection in src/openhuman/memory/tree/store.rs, which configures Write-Ahead Logging (WAL) mode for concurrent read/write performance and implements automatic retry logic on SQLite "busy" errors.

Core Tables and Relationships

The store::initialize_schema function creates five interlinked tables that separate content from metadata and search indices:

• chunks: The primary tree structure storing id (primary key), source_kind, source_id, owner, status, content_path, and the nullable parent_id that establishes hierarchy.
• raw_refs: Byte-range pointers (path, start, end) linking chunks to original source files.
• embeddings: Vector storage for similarity search, containing chunk_id and float arrays.
• summaries: LLM-generated condensed representations stored at summary_path.
• metadata: Implicit status tracking through column values like CHUNK_STATUS_PENDING_EXTRACTION, CHUNK_STATUS_ADMITTED, and CHUNK_STATUS_SEALED.

The parent_id Tree Mechanism

Hierarchical relationships are encoded through the parent_id foreign key in the chunks table. When parent_id is NULL, the chunk represents a root node; when populated, it references another row in the same table. This design enables:

• Leaf chunks: Terminal nodes containing raw text or file fragments.
• Internal chunks: Aggregated nodes holding summaries and embeddings of their children.

The schema applies lazily on first open with migration logic logged at src/openhuman/memory/tree/store.rs:660.

Storing Knowledge: The Ingestion Pipeline

Knowledge ingestion follows a strict lifecycle from raw data to indexed, queryable nodes, all wrapped in ACID transactions via with_connection.

Inserting Chunks with upsert_chunks

New knowledge enters the system through store::upsert_chunks, defined in src/openhuman/memory/tree/store.rs:242. This function accepts a slice of Chunk structs and performs parameterized inserts using rusqlite::params! to prevent SQL injection.

use openhuman::memory::tree::{store, Chunk, Config};

let cfg = Config::load()?;
let new_chunk = Chunk::new(
    "file://notes/meeting.md",
    "Meeting notes on AI-driven knowledge graphs".into(),
);
store::upsert_chunks(&cfg, &[new_chunk])?;

Lifecycle Status Management

Chunks progress through states managed by store::set_chunk_lifecycle_status:

1. pending_extraction: Initial state after insertion.
2. admitted: Intermediate state after validation.
3. sealed: Final immutable state ready for querying.

This state machine prevents incomplete data from appearing in search results while embeddings are being computed.

Embedding and Vector Indexing

After chunk creation, the system computes embeddings via external LLM services and persists them using store::set_chunk_embedding (located at store.rs:912):

let embedding = my_embedder.compute(&new_chunk.content)?;
store::set_chunk_embedding(&cfg, &new_chunk.id, &embedding)?;

The embedding is stored in the embeddings table while the chunk's status column transitions toward sealed.

Retrieving Hierarchical Data: Query Patterns

OpenHuman exposes retrieval through the memory_tree skill, dispatching to specialized modules in src/openhuman/memory/tree/retrieval/ that issue compiled SQL queries.

Drill-Down: Traversing to Root Ancestors

The memory_tree_drill_down tool invokes retrieval::drill_down::drill_down in src/openhuman/memory/tree/retrieval/drill_down.rs. This function recursively walks up the tree from a leaf node to the root via parent_id joins, returning the complete ancestry path for context assembly:

use openhuman::memory::tree::retrieval::drill_down::drill_down;

let leaf_id = "chunk-23";
let path = drill_down(&cfg, leaf_id)?; // Returns Vec<Chunk> from leaf → root

Fetch Leaves: Collecting Descendants

Conversely, memory_tree_fetch_leaves triggers retrieval::fetch::fetch_leaves (fetch.rs), which walks down from a root ID to collect all terminal nodes:

use openhuman::memory::tree::retrieval::fetch::fetch_leaves;

let root_id = "chunk-root-1";
let leaves = fetch_leaves(&cfg, &[root_id])?; // Returns Vec<Chunk>

This uses parameterized queries like SELECT … FROM chunks WHERE parent_id = ? to traverse subtrees without loading the entire graph into memory.

Vector and Full-Text Search Integration

OpenHuman combines hierarchical traversal with semantic search:

• retrieval::search::search_entities: Implements FTS5 full-text search over chunks.content_path for the memory_tree_search_entities tool.
• retrieval::global::query_topic: Performs embedding similarity queries via the memory_tree_query_topic tool to locate the most relevant entry point in the tree for a given topic.

These retrieval methods work hierarchically—first finding a relevant node via vector similarity, then using drill_down or fetch_leaves to gather complete context.

Summary

• Storage: Hierarchical knowledge is stored in <workspace>/memory_tree/chunks.db using an adjacency list pattern via the parent_id column in the chunks table.
• Schema: Five core tables (chunks, raw_refs, embeddings, summaries, metadata) separate content, vectors, and provenance.
• Lifecycle: Chunks progress from pending_extraction to sealed via store::set_chunk_lifecycle_status, ensuring data consistency.
• Traversal: drill_down walks upward to roots; fetch_leaves collects terminal descendants; both use parameterized SQL with WAL-mode transactions.
• Search: Hybrid retrieval combining FTS5 text search, vector similarity, and tree traversal.

Frequently Asked Questions

How does OpenHuman prevent SQLite contention during concurrent writes?

The store::with_connection wrapper configures SQLite with WAL (Write-Ahead Logging) mode and implements automatic retry logic when the database returns "busy" errors. All chunk operations—including upsert_chunks and set_chunk_embedding—execute within transactions that retry on contention, ensuring atomic updates to both the chunks table and its associated embeddings rows without blocking readers.

What is the difference between "fetch_leaves" and "drill_down" in the Memory Tree?

fetch_leaves traverses downward from a parent chunk to collect all descendant terminal nodes (children with no further children), useful for gathering all raw source material under a topic summary. drill_down traverses upward from a specific chunk to collect all ancestors back to the root, used to assemble hierarchical context for LLM prompts. Both operate via recursive SQL queries on the parent_id column without loading the entire tree into RAM.

How are embeddings stored while maintaining hierarchical relationships?

Embeddings are stored in a separate embeddings table linked by chunk_id, keeping the chunks table row size manageable for tree traversal queries. When store::set_chunk_embedding is called, it updates this external table while the chunks table maintains the parent_id hierarchy. This separation allows vector similarity searches to return chunk IDs that are then passed to drill_down or fetch_leaves for hierarchical context retrieval.

Can the Memory Tree handle file renames or moves while preserving hierarchy?

Yes. The raw_refs table stores byte-range pointers (path, start, end) independently of the hierarchical structure in chunks. If a file moves, updating the content_path in the chunks table and the path values in raw_refs preserves the entire parent_id tree structure and all computed embeddings, as the chunk identities remain stable UUIDs referenced by foreign keys.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}