# What Kind of Data Does OpenHuman Process? A Complete Technical Guide

> OpenHuman processes nine data types including text, audio, video, and more for AI applications. Discover the full technical scope of OpenHuman data processing.

- Repository: [Tiny Humans/openhuman](https://github.com/tinyhumansai/openhuman)
- Tags: deep-dive
- Published: 2026-06-11

---

**OpenHuman processes nine distinct data categories including user-generated text, embedding vectors, audio streams, video captures, memory graph nodes, tool execution results, external service payloads, configuration credentials, and analytics telemetry.**

If you're exploring the `tinyhumansai/openhuman` repository, understanding what kind of data the system ingests is essential for integration and security audits. The Rust-powered core handles diverse modalities through a modular architecture, routing all data flows via JSON-RPC and persisting them in the workspace directory at `~/.openhuman/`.

## Core Data Categories Processed by OpenHuman

The architecture isolates each data type into dedicated domain modules, ensuring type-safe handling across the Rust core and React frontend.

### User-Generated Text and Chat Messages

**User-generated text**—including chat messages, prompts, and commands—enters the system through the UI or JSON-RPC API. In [`src/openhuman/agent_memory/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/agent_memory/types.rs), the codebase defines structures used to benchmark and record retrieval of these textual conversations. When you submit a message, the `chat_send` RPC routes to [`src/openhuman/chat/ops.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/chat/ops.rs), which stores the data in the chat domain and triggers relevant agent tools.

```typescript
// Frontend submission via Tauri RPC
await coreRpcClient.call('openhuman.chat_send', {
  thread_id: 'default',
  content: 'Hey OpenHuman, summarize my recent Slack messages.'
});

```

### Embedding Vectors for Semantic Search

The system generates and caches **embedding vectors**—dense numeric representations of text—for fast similarity search. These vectors are processed through [`src/openhuman/inference/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/inference/types.rs), which contains request and response structs for model inference. When calling `openhuman.inference_generate`, the response payload includes generated text alongside an optional `embedding` vector for downstream retrieval.

### Audio Data and Voice Processing

**Audio streams** encompass voice recordings and synthesized speech handled by the voice domain. The [`src/openhuman/voice/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/voice/types.rs) file specifies payloads for audio streams and synthesis results, supporting operations like transcription and diarization. The backend forwards audio blobs to configured speech-to-text backends via [`src/openhuman/voice/ops.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/voice/ops.rs).

```typescript
await coreRpcClient.call('openhuman.voice_transcribe', {
  audio_blob: myBlob,  // Uint8Array of PCM data
  language: 'en',
  diarization: true
});

```

### Video and Screen Intelligence

OpenHuman ingests **video and screen captures** through two specialized domains. The meet domain handles webcam frames and streaming video via [`src/openhuman/meet/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/meet/types.rs), while screen intelligence processes screenshots and visual data via [`src/openhuman/screen_intelligence/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/screen_intelligence/types.rs). These vision data types enable virtual meeting capabilities and screen-based agent interactions.

### Memory Graph Structures

The **memory graph** stores a persistent, version-controlled graph of memory nodes containing documents, snippets, and conversations. Defined in [`src/openhuman/memory_graph/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/memory_graph/types.rs), these node and edge structures are queried during agent walks. The retrieval benchmarks are recorded in [`src/openhuman/agent_memory/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/agent_memory/types.rs), creating a queryable knowledge base that persists across sessions.

```typescript
const result = await coreRpcClient.call('openhuman.memory_query', {
  query: 'What did the user say about project X last week?',
  top_k: 5
});

```

### Tool Execution Results

When agents invoke built-in tools—such as web search, file access, or Composio integrations—the **tool execution results** flow through [`src/openhuman/tool_registry/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/tool_registry/types.rs). Policy enforcement and metadata governance occur in [`src/openhuman/agent_tool_policy/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/agent_tool_policy/types.rs) and [`src/openhuman/agent_tool_policy/ops.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/agent_tool_policy/ops.rs), ensuring controlled access to tool outputs.

```typescript
await coreRpcClient.call('openhuman.tool_execute', {
  tool_id: 'web_search',
  args: { query: 'latest Rust 1.75 release notes' }
});

```

### External Service Integrations

**External service payloads** from Slack, Gmail, Discord, WhatsApp, and other integrations reside in domain-specific modules like [`src/openhuman/integrations/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/integrations/types.rs) and [`src/openhuman/webview_notifications/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/webview_notifications/types.rs). Each integration maintains isolated type definitions, preventing data leakage between external services.

### Configuration and Credentials

Sensitive **configuration data**—including environment-driven settings, masked API keys, and user preferences—are structured in [`src/openhuman/config/schema/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/config/schema/types.rs). Runtime values load without exposing secrets, and the frontend mirrors these settings through [`app/src/utils/config.ts`](https://github.com/tinyhumansai/openhuman/blob/main/app/src/utils/config.ts).

### Analytics and Telemetry

Internal **metrics and telemetry** support observability through [`src/openhuman/health/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/health/types.rs) and [`src/openhuman/diagnostics/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/diagnostics/types.rs). These capture health checks, logs, and performance data essential for monitoring the AI assistant's operational status.

## Data Flow Architecture and Key Files

All data flows mediate through the JSON-RPC surface in [`src/rpc/dispatch.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/rpc/dispatch.rs), which routes requests to appropriate domain handlers. The modular architecture ensures each data class remains isolated yet composable.

| File | Data Role |
|------|-----------|
| [`src/openhuman/agent_memory/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/agent_memory/types.rs) | Benchmark structs for memory retrieval |
| [`src/openhuman/inference/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/inference/types.rs) | Model inference request/response payloads |
| [`src/openhuman/voice/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/voice/types.rs) | Audio processing definitions |
| [`src/openhuman/memory_graph/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/memory_graph/types.rs) | Graph node and edge structures |
| [`src/openhuman/tool_registry/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/tool_registry/types.rs) | Tool metadata descriptors |
| [`src/openhuman/config/schema/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/config/schema/types.rs) | Configuration and credential schemas |
| [`src/rpc/dispatch.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/rpc/dispatch.rs) | Central JSON-RPC router |

## Summary

OpenHuman processes a comprehensive range of data types through its Rust core and React frontend:

- **Textual data** via chat messages and prompts stored in agent memory structures
- **Vector embeddings** generated by the inference engine for semantic search
- **Audio streams** supporting transcription and speech synthesis
- **Video and screen data** for meeting and vision capabilities
- **Graph-based memory** persisting documents and conversations
- **Tool outputs** from web search, file access, and third-party integrations
- **External payloads** from services like Slack, Gmail, and Discord
- **Configuration data** managing credentials and user preferences
- **Telemetry** for system health and diagnostics

Each category flows through [`src/rpc/dispatch.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/rpc/dispatch.rs) and persists in `~/.openhuman/`, maintaining strict domain isolation.

## Frequently Asked Questions

### What types of media files can OpenHuman process?

OpenHuman handles audio streams (PCM data for transcription), video frames (webcam and screen captures), and image data for screen intelligence. The [`src/openhuman/voice/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/voice/types.rs) module manages audio payloads, while [`src/openhuman/meet/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/meet/types.rs) and [`src/openhuman/screen_intelligence/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/screen_intelligence/types.rs) handle visual media. All media processing occurs through the JSON-RPC interface with type-safe Rust structs.

### How does OpenHuman handle sensitive data like API keys?

Sensitive configuration data is defined in [`src/openhuman/config/schema/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/config/schema/types.rs) and loaded at runtime without exposing secrets in logs or frontend code. The system masks API keys and credentials, storing them securely in the workspace directory while allowing the frontend to access non-sensitive settings through [`app/src/utils/config.ts`](https://github.com/tinyhumansai/openhuman/blob/main/app/src/utils/config.ts).

### Is OpenHuman's data processing modular or monolithic?

The architecture is deliberately modular. Each data type owns its dedicated directory under `src/openhuman/` containing [`types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/types.rs), [`ops.rs`](https://github.com/tinyhumansai/openhuman/blob/main/ops.rs), and schema files. This isolation prevents cross-domain contamination while allowing composability through the central JSON-RPC dispatcher in [`src/rpc/dispatch.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/rpc/dispatch.rs).

### What database does OpenHuman use for its memory graph?

According to the source code, the memory graph persists in the workspace directory at `~/.openhuman/` using the structures defined in [`src/openhuman/memory_graph/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/memory_graph/types.rs). The system implements a version-controlled graph of memory nodes that agents query during walks, with retrieval benchmarks recorded in [`src/openhuman/agent_memory/types.rs`](https://github.com/tinyhumansai/openhuman/blob/main/src/openhuman/agent_memory/types.rs).