TinyMemory Core Crate Architecture: Modules and Components Explained

The TinyMemory Core crate provides a modular Rust memory subsystem for OpenHuman, exposing singleton initialization, hierarchical chunk storage, background job queues, and comprehensive audit logging through a set of well-defined modules.

The TinyMemory Core crate (tinymemory_core) serves as the central library powering OpenHuman’s persistent memory layer. It organizes functionality into discrete modules with clear separation of concerns, from low-level chunk storage in src/openhuman/memory/store/chunks/ to high-level content summarization and external synchronization. Below is a complete architectural breakdown of the crate’s components, including actual source paths and implementation details from the tinyhumansai/openhuman repository.

Global Initialization and Client Management

The global module in src/openhuman/memory/global.rs provides the entry point for the entire memory system. It exposes a singleton pattern through the global::init function, which bootstraps the MemoryClient with a workspace directory, and global::client to retrieve the handle for subsequent operations.

use tinymemory_core::global;

// Initialise the global client with a workspace directory.
global::init("/path/to/workspace").expect("failed to initialise memory");

// Retrieve a handle to the client for subsequent calls.
let client = global::client();

This pattern ensures that all modules share a single, initialized client instance throughout the process lifecycle.

The Unified Storage Interface

At the heart of the crate lies the store module (src/openhuman/memory/store/mod.rs), which exposes a high-level façade through types like UnifiedMemory, NamespaceStore, and the MemoryClient. This layer abstracts the underlying storage backends, providing a consistent API for writing and retrieving memory objects regardless of their specific format.

Chunk Management (store::chunks)

The store::chunks submodule handles fine-grained text fragments and their metadata. Defined in src/openhuman/memory/store/chunks/types.rs, the core Chunk struct carries text content, optional embeddings, and metadata including SourceKind (e.g., Chat, Document) and SourceRef. The companion storage logic in src/openhuman/memory/store/chunks/store.rs provides upsert_chunks for atomic insert-or-update operations and with_connection for transaction-scoped access.

use tinymemory_core::store::chunks::store::upsert_chunks;
use tinymemory_core::store::chunks::types::{Chunk, Metadata, SourceKind, SourceRef};

let chunk = Chunk {
    id: "c1".into(),
    text: "Hello, world!".into(),
    embedding: None,
    metadata: Metadata {
        source_ref: SourceRef::new("source-123".into()),
        source_kind: SourceKind::Chat,
        ..Default::default()
    },
    ..Default::default()
};

upsert_chunks(&client, vec![chunk]).expect("failed to upsert chunk");

Hierarchical Tree Structures (store::trees)

While chunks store raw data, the store::trees module organizes them into hierarchical structures. Located in src/openhuman/memory/store/trees/types.rs, it defines the Tree struct with variants like TreeKind::Topic and TreeKind::Session, along with SummaryNode for aggregated representations. Persistence functions in src/openhuman/memory/store/trees/store.rs expose upsert_tree for creating these organizational structures.

use tinymemory_core::store::trees::store::upsert_tree;
use tinymemory_core::store::trees::types::{Tree, TreeKind, SummaryNode};

let tree = Tree {
    id: "t1".into(),
    kind: TreeKind::Topic,
    summary: SummaryNode::default(),
    ..Default::default()
};

upsert_tree(&client, vec![tree]).expect("failed to create tree");

Content Operations (store::content)

Higher-level content processing resides in src/openhuman/memory/store/content/mod.rs. This module handles summarization workflows through types like SummaryComposeInput and SummaryTreeKind, and manages raw content variants via raw::RawKind. It bridges the gap between raw chunks and semantic understanding.

Identity Mapping (store::identity)

The store::identity module (src/openhuman/memory/store/identity/mod.rs) manages user and tool identity within the memory graph. It provides the IdentityKind enum and helper functions like is_self_identity_any_toolkit to distinguish between human users and autonomous tools when attributing memory entries.

Background Job Processing

The queue module (src/openhuman/memory/queue/mod.rs) implements asynchronous background processing for computationally expensive tasks. It defines payloads such as ReembedBackfillPayload for recalculating embeddings and exposes enqueue_backfill to schedule work without blocking the main execution flow.

use tinymemory_core::queue::types::ReembedBackfillPayload;
use tinymemory_core::queue::enqueue_backfill;

let payload = ReembedBackfillPayload {
    chunk_id: "c1".into(),
    model: "openai_text-embedding-ada-002".into(),
};

enqueue_backfill(&client, payload).expect("failed to enqueue backfill");

Audit Logging with TinyCortex

All mutations within the TinyMemory Core crate are recorded by the tinycortex subsystem (src/openhuman/memory/tinycortex/mod.rs). This audit-log module provides read_audit_log for retrieving historical operations and run_github_sync for external synchronization, enabling reproducible debugging and event replay.

use tinymemory_core::tinycortex::read_audit_log;

let entries = read_audit_log(&client, 100).expect("audit read failed");
for entry in entries {
    println!("{:?}", entry);
}

External Service Synchronization

The sync module (src/openhuman/memory/sync/mod.rs) bridges the core memory system to external providers. Current implementations include Composio integration, accessed through paths like sync::composio::providers::user_scopes, allowing automated ingestion of external tool interactions into the memory graph.

Source Registry and Tree Policies

Three specialized modules handle metadata management and lifecycle policies:

• sources (src/openhuman/memory/sources/registry.rs): Maintains a registry of SourceKind descriptors and provides replace_sources_in for batch source updates.
• tree_source (src/openhuman/memory/tree_source.rs): Offers get_or_create_source_tree to locate or instantiate source-trees that group related chunks under common origins.
• tree_policy (src/openhuman/memory/tree_policy.rs): Implements TreePolicy structs that drive pruning, retention budgeting, and hierarchical cleanup strategies.

How the Components Integrate

When an application interacts with the TinyMemory Core crate, the data flows through a predictable pipeline:

1. Bootstrapping: global::init establishes the singleton MemoryClient.
2. Data Ingestion: Callers use upsert_chunks or upsert_tree to write low-level objects, associating IdentityKind and SourceKind metadata.
3. Organization: The tree_source and tree_policy modules organize chunks into trees and enforce retention rules.
4. Async Processing: Expensive operations route through the queue module for background execution.
5. Auditing: The tinycortex module records every mutation for later inspection.
6. External Sync: The sync module pulls external data (e.g., from Composio) and injects it through the standard store APIs.

Summary

• The TinyMemory Core crate organizes OpenHuman's memory layer into twelve distinct modules with clear separation of concerns.
• Global initialization via global::init creates a singleton MemoryClient shared across all operations.
• Storage splits between atomic chunks (text fragments) and hierarchical trees (organizational structures), managed by the unified store façade.
• Identity metadata distinguishes between users and tools, while the sources registry tracks data provenance.
• Background jobs and audit logging run asynchronously through the queue and tinycortex modules.
• External synchronization integrates third-party services like Composio into the memory graph.

Frequently Asked Questions

What is the difference between chunks and trees in the TinyMemory Core crate?

Chunks represent atomic text fragments with embeddings and metadata, handled by the store::chunks module. Trees provide hierarchical organization for these chunks, managed by store::trees, allowing grouping by topic, session, or custom taxonomies through the Tree and TreeKind types.

How do I initialize the memory client in a Rust application?

Call global::init("/path/to/workspace") once at startup, then retrieve the handle with global::client(). This singleton pattern, defined in src/openhuman/memory/global.rs, ensures all modules share a single initialized MemoryClient instance.

What is the purpose of the TinyCortex module?

The TinyCortex module (src/openhuman/memory/tinycortex/mod.rs) serves as the immutable audit log for the memory system. It records all mutations via read_audit_log and supports event replay through functions like run_github_sync, enabling debugging and historical analysis.

How does the queue module handle background tasks?

The queue module (src/openhuman/memory/queue/mod.rs) offloads expensive operations—such as recomputing vector embeddings via ReembedBackfillPayload—to background workers using enqueue_backfill. This prevents blocking the main thread during heavy computational tasks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}