OpenHuman Memory Subsystem Engine Migration to TinyBus Native Module

The OpenHuman Memory subsystem engine migration to TinyBus native module relocates the storage and retrieval logic from a built-in library to a dynamically-loaded cdylib, eliminating heavy compile-time dependencies while maintaining zero-cost abstractions through shared-memory interfaces.

The tinyhumansai/openhuman repository restructured its persistence architecture to reduce binary size and improve deployment flexibility. This technical guide examines the Memory subsystem engine migration to TinyBus native module, detailing how the core binary delegates storage operations to an external module while preserving type safety through strict compile-time contracts.

Why Migrate to a TinyBus Native Module?

The original in-process memory engine required heavy crates such as git2 and libz-sys to be linked directly into the core binary. By migrating the engine to a TinyBus native module, these dependencies become load-time requirements only, allowing slim builds to exclude them entirely and significantly reducing artifact size.

Process-shared address space ensures the module operates within the same address space as the core, granting direct access to shared state like the SQLite store without inter-process communication overhead. The module's lifetime matches the core process—TinyBus never unloads modules—creating a simple failure model where errors are permanent until restart.

Architecture of the Native Module Migration

The Contract Crate (tinymemory-api)

The migration relies on a shared contract crate that defines the wire-level types crossing the TinyBus boundary. This crate exposes MemoryError, Chunk, and MemoryEvent structures that both the core and module must implement identically, as defined in src/openhuman/memory/api.rs.

Compile-time verification ensures type safety. The test file src/openhuman/memory/api_identity_tests.rs asserts that the contract types remain bitwise identical between the core and module sides, preventing serialization mismatches before runtime.

Core Driver and Proxy Layer

The core side implements the MemoryProvider trait in src/openhuman/memory/driver/mod.rs, but instead of executing storage logic directly, it forwards calls through TinyBus proxies. When a component invokes memory::ops::retrieve, the driver constructs a proxy for the appropriate host object—such as ai.tinyhumans.tinymemory.ChatHost—and marshals the request across the bus.

Error translation occurs in src/openhuman/memory/module.rs via the from_bus function, which maps tinybus::Error::MethodFailed to MemoryError::Unavailable and tinybus::Error::InvalidArgument to MemoryError::Invalid.

Module Registry and Verification

The registry in src/openhuman/modules/registry.rs pins specific module releases using SHA-256 digests. Before loading, the core verifies the module binary's digest against the compile-time expectation, guaranteeing that the loaded library matches the contract defined in src/openhuman/memory/api.rs.

Runtime Loading and Lifecycle

During startup, modules::ops::load checks the OPENHUMAN_MODULE_PATH environment variable for the memory module binary. If present and the digest verification succeeds, the system calls dlopen and registers the module with TinyBus.

Failure handling follows a strict permanent-failure model. If the module fails to load due to missing files, digest mismatches, or runtime errors, the core records the failure and marks the module as permanently unavailable for the process lifetime. Subsequent calls return MemoryError::Unavailable consistently.

// Core side retrieval operation from src/openhuman/memory/ops
use openhuman::memory::ops::retrieve;
use openhuman::memory::rpc_models::ChunkId;

async fn get_chunk(id: ChunkId) -> Result<Vec<u8>, MemoryError> {
    // Builds TinyBus proxy behind the scenes
    retrieve(id).await
}

Error classification bridges the TinyBus and core domains:

// Error translation in src/openhuman/memory/module.rs
fn from_bus(err: &tinybus::Error) -> MemoryError {
    match err {
        tinybus::Error::MethodFailed { .. } => MemoryError::Unavailable,
        tinybus::Error::InvalidArgument { .. } => MemoryError::Invalid,
        _ => MemoryError::Internal,
    }
}

Implementation Steps in the Codebase

The migration followed a structured refactoring path across the codebase:

1. Contract extraction – The tinymemory-api crate was separated from the core to create a shared dependency, forcing all cross-boundary types into src/openhuman/memory/api.rs.

2. Facade consolidation – Legacy in-process implementations in memory::binding were reduced to fallback stubs that forward to the TinyBus module, eliminating duplicate logic.

3. Registry pinning – src/openhuman/modules/registry.rs was updated to enforce SHA-256 digest verification, ensuring loaded modules match the compile-time contract.

4. Error mapping – Module-side errors were classified into core MemoryError variants in src/openhuman/memory/module.rs.

5. Integration testing – src/openhuman/memory/direct_engine_refs_tests.rs and src/openhuman/memory/api_identity_tests.rs verify that contract types are identical and that calls correctly traverse the TinyBus boundary.

Summary

• The Memory subsystem engine migration to TinyBus native module moves storage logic from the core binary into a dynamically-loaded cdylib.
• Heavy dependencies like git2 and libz-sys are no longer required at compile time for the core, reducing binary size.
• Type safety is enforced through the tinymemory-api contract crate and compile-time identity tests in src/openhuman/memory/api_identity_tests.rs.
• The module loader in src/openhuman/modules/ops.rs verifies SHA-256 digests before loading, ensuring binary integrity.
• Errors from the module are mapped to core MemoryError variants via the from_bus function in src/openhuman/memory/module.rs.
• TinyBus loads the module into the shared address space with no unload capability, creating a fail-fast model where module failures are permanent until process restart.

Frequently Asked Questions

How does the core binary communicate with the memory module without IPC overhead?

TinyBus loads the native module as a cdylib into the same process address space using dlopen. This shared-memory approach allows the module to access the SQLite store and other shared state directly while still maintaining a strict boundary defined by the tinymemory-api contract. The communication uses zero-copy proxies rather than sockets or pipes.

What happens if the memory module fails to load or crashes at runtime?

If loading fails due to missing files, digest mismatches, or initialization errors, the modules::ops::load function records the failure and marks the module as permanently unavailable. Because TinyBus never unloads libraries, this state persists until the core process restarts. All subsequent memory operations return MemoryError::Unavailable via the from_bus error translator in src/openhuman/memory/module.rs.

How does the system ensure type compatibility between the core and the external module?

The system uses compile-time contract verification through src/openhuman/memory/api_identity_tests.rs. This test suite asserts that types defined in src/openhuman/memory/api.rs—such as MemoryError, Chunk, and MemoryEvent—are identical on both sides of the TinyBus boundary. Additionally, the registry in src/openhuman/modules/registry.rs pins a specific module release by SHA-256 digest, ensuring the loaded binary matches the contract expected by the compiled core.

Can the memory module be updated independently of the core binary?

Yes, the architecture supports independent updates through the registry system. The core references a specific SHA-256 digest in src/openhuman/modules/registry.rs, allowing the module artifact to be replaced on disk. When the core restarts, it loads the new module version after verifying its digest. This decoupling allows memory engine improvements without recompiling or redeploying the entire OpenHuman core.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}