What Kind of Data Does OpenHuman Process? A Complete Technical Guide

OpenHuman processes nine distinct data categories including user-generated text, embedding vectors, audio streams, video captures, memory graph nodes, tool execution results, external service payloads, configuration credentials, and analytics telemetry.

If you're exploring the tinyhumansai/openhuman repository, understanding what kind of data the system ingests is essential for integration and security audits. The Rust-powered core handles diverse modalities through a modular architecture, routing all data flows via JSON-RPC and persisting them in the workspace directory at ~/.openhuman/.

Core Data Categories Processed by OpenHuman

The architecture isolates each data type into dedicated domain modules, ensuring type-safe handling across the Rust core and React frontend.

User-Generated Text and Chat Messages

User-generated text—including chat messages, prompts, and commands—enters the system through the UI or JSON-RPC API. In src/openhuman/agent_memory/types.rs, the codebase defines structures used to benchmark and record retrieval of these textual conversations. When you submit a message, the chat_send RPC routes to src/openhuman/chat/ops.rs, which stores the data in the chat domain and triggers relevant agent tools.

// Frontend submission via Tauri RPC
await coreRpcClient.call('openhuman.chat_send', {
  thread_id: 'default',
  content: 'Hey OpenHuman, summarize my recent Slack messages.'
});

Embedding Vectors for Semantic Search

The system generates and caches embedding vectors—dense numeric representations of text—for fast similarity search. These vectors are processed through src/openhuman/inference/types.rs, which contains request and response structs for model inference. When calling openhuman.inference_generate, the response payload includes generated text alongside an optional embedding vector for downstream retrieval.

Audio Data and Voice Processing

Audio streams encompass voice recordings and synthesized speech handled by the voice domain. The src/openhuman/voice/types.rs file specifies payloads for audio streams and synthesis results, supporting operations like transcription and diarization. The backend forwards audio blobs to configured speech-to-text backends via src/openhuman/voice/ops.rs.

await coreRpcClient.call('openhuman.voice_transcribe', {
  audio_blob: myBlob,  // Uint8Array of PCM data
  language: 'en',
  diarization: true
});

Video and Screen Intelligence

OpenHuman ingests video and screen captures through two specialized domains. The meet domain handles webcam frames and streaming video via src/openhuman/meet/types.rs, while screen intelligence processes screenshots and visual data via src/openhuman/screen_intelligence/types.rs. These vision data types enable virtual meeting capabilities and screen-based agent interactions.

Memory Graph Structures

The memory graph stores a persistent, version-controlled graph of memory nodes containing documents, snippets, and conversations. Defined in src/openhuman/memory_graph/types.rs, these node and edge structures are queried during agent walks. The retrieval benchmarks are recorded in src/openhuman/agent_memory/types.rs, creating a queryable knowledge base that persists across sessions.

const result = await coreRpcClient.call('openhuman.memory_query', {
  query: 'What did the user say about project X last week?',
  top_k: 5
});

Tool Execution Results

When agents invoke built-in tools—such as web search, file access, or Composio integrations—the tool execution results flow through src/openhuman/tool_registry/types.rs. Policy enforcement and metadata governance occur in src/openhuman/agent_tool_policy/types.rs and src/openhuman/agent_tool_policy/ops.rs, ensuring controlled access to tool outputs.

await coreRpcClient.call('openhuman.tool_execute', {
  tool_id: 'web_search',
  args: { query: 'latest Rust 1.75 release notes' }
});

External Service Integrations

External service payloads from Slack, Gmail, Discord, WhatsApp, and other integrations reside in domain-specific modules like src/openhuman/integrations/types.rs and src/openhuman/webview_notifications/types.rs. Each integration maintains isolated type definitions, preventing data leakage between external services.

Configuration and Credentials

Sensitive configuration data—including environment-driven settings, masked API keys, and user preferences—are structured in src/openhuman/config/schema/types.rs. Runtime values load without exposing secrets, and the frontend mirrors these settings through app/src/utils/config.ts.

Analytics and Telemetry

Internal metrics and telemetry support observability through src/openhuman/health/types.rs and src/openhuman/diagnostics/types.rs. These capture health checks, logs, and performance data essential for monitoring the AI assistant's operational status.

Data Flow Architecture and Key Files

All data flows mediate through the JSON-RPC surface in src/rpc/dispatch.rs, which routes requests to appropriate domain handlers. The modular architecture ensures each data class remains isolated yet composable.

File	Data Role
src/openhuman/agent_memory/types.rs	Benchmark structs for memory retrieval
src/openhuman/inference/types.rs	Model inference request/response payloads
src/openhuman/voice/types.rs	Audio processing definitions
src/openhuman/memory_graph/types.rs	Graph node and edge structures
src/openhuman/tool_registry/types.rs	Tool metadata descriptors
src/openhuman/config/schema/types.rs	Configuration and credential schemas
src/rpc/dispatch.rs	Central JSON-RPC router

Summary

OpenHuman processes a comprehensive range of data types through its Rust core and React frontend:

• Textual data via chat messages and prompts stored in agent memory structures
• Vector embeddings generated by the inference engine for semantic search
• Audio streams supporting transcription and speech synthesis
• Video and screen data for meeting and vision capabilities
• Graph-based memory persisting documents and conversations
• Tool outputs from web search, file access, and third-party integrations
• External payloads from services like Slack, Gmail, and Discord
• Configuration data managing credentials and user preferences
• Telemetry for system health and diagnostics

Each category flows through src/rpc/dispatch.rs and persists in ~/.openhuman/, maintaining strict domain isolation.

Frequently Asked Questions

What types of media files can OpenHuman process?

OpenHuman handles audio streams (PCM data for transcription), video frames (webcam and screen captures), and image data for screen intelligence. The src/openhuman/voice/types.rs module manages audio payloads, while src/openhuman/meet/types.rs and src/openhuman/screen_intelligence/types.rs handle visual media. All media processing occurs through the JSON-RPC interface with type-safe Rust structs.

How does OpenHuman handle sensitive data like API keys?

Sensitive configuration data is defined in src/openhuman/config/schema/types.rs and loaded at runtime without exposing secrets in logs or frontend code. The system masks API keys and credentials, storing them securely in the workspace directory while allowing the frontend to access non-sensitive settings through app/src/utils/config.ts.

Is OpenHuman's data processing modular or monolithic?

The architecture is deliberately modular. Each data type owns its dedicated directory under src/openhuman/ containing types.rs, ops.rs, and schema files. This isolation prevents cross-domain contamination while allowing composability through the central JSON-RPC dispatcher in src/rpc/dispatch.rs.

What database does OpenHuman use for its memory graph?

According to the source code, the memory graph persists in the workspace directory at ~/.openhuman/ using the structures defined in src/openhuman/memory_graph/types.rs. The system implements a version-controlled graph of memory nodes that agents query during walks, with retrieval benchmarks recorded in src/openhuman/agent_memory/types.rs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}