Architecture of the OpenHuman Web Scraper Tool: CDP-Based Web Content Fetching

OpenHuman does not rely on traditional HTTP clients for scraping; instead, it drives an embedded Chromium (CEF) instance through the Chrome DevTools Protocol (CDP) via WebSocket, using a three-layer architecture of a JSON-RPC connector, DOM snapshot parser, and provider-specific scanners that alternate between rapid DOM-only checks and deep IndexedDB crawls.

The open-source OpenHuman repository (tinyhumansai/openhuman) fetches web content from dynamic single-page applications like WhatsApp Web, Telegram, and Discord by instrumenting the browser directly rather than parsing static HTML. Understanding the architecture of the web scraper tool reveals how the system achieves zero JavaScript injection while capturing cross-frame DOM trees and persistent storage data.

Core Components of the Scraping Architecture

The scraper is implemented in Rust within the Tauri sidecar and consists of three reusable primitives that abstract the low-level CDP into a high-level data extraction pipeline.

CdpConn: The WebSocket JSON-RPC Client

The CdpConn struct in app/src-tauri/src/cdp/snapshot.rs (lines 31–45) establishes the transport layer for CDP commands. It opens a WebSocket connection to the Chromium instance’s debug port and manages request ID auto-incrementing, providing a generic call method for any CDP domain method.

// Conceptual usage from snapshot.rs lines 31-45
impl CdpConn {
    pub async fn open(ws_url: &str) -> Result<Self, Error> {
        // Establishes WebSocket stream to 127.0.0.1:19222
    }
    
    pub async fn call<T>(&mut self, method: &str, params: Value) -> Result<T, Error> {
        // Sends JSON-RPC request with auto-incremented ID
        // Returns typed response matching the request ID
    }
}

Snapshot: Parsing DOMSnapshot Results

Also defined in app/src-tauri/src/cdp/snapshot.rs (lines 81–112), the Snapshot struct parses the flat-array response from DOMSnapshot.captureSnapshot. It reconstructs the document tree from parallel arrays (parentIndex, nodeType, nodeName, etc.), builds child-lookup tables, and optionally stores layout rectangles when includeDOMRects is enabled.

The Snapshot::capture_inner method (lines 91–115) sends the CDP command and populates the struct:

impl Snapshot {
    async fn capture_inner(conn: &mut CdpConn, with_rects: bool) -> Result<Self, Error> {
        let params = json!({
            "includeDOMRects": with_rects,
            // ... other capture options
        });
        let result: CaptureSnapshotResult = conn
            .call("DOMSnapshot.captureSnapshot", params)
            .await?;
        // Parses result.documents into internal node tree
    }
}

Provider Scanners: Orchestration and Domain Logic

Provider-specific modules—such as app/src-tauri/src/whatsapp_scanner/mod.rs—implement the high-level scanning logic. These modules define CSS selectors for chat rows and coordinate the two-phase extraction process (fast DOM-only versus full IndexedDB walk).

How the Scraping Process Works

A complete scraping cycle follows a rigorous sequence from endpoint discovery to data emission.

Step 1: Discovering the CDP Endpoint

The scanner first queries the local debug HTTP server to obtain the WebSocket URL. In app/src-tauri/src/whatsapp_scanner/mod.rs (lines 92–100), the browser_ws_url() function requests http://127.0.0.1:19222/json/version and extracts the webSocketDebuggerUrl.

Step 2: Attaching to the Target

Using CdpConn::open, the scanner establishes the WebSocket stream, then sends Target.attachToTarget to bind to the specific page session, enabling DOM and runtime domain access.

Step 3: Capturing the DOM Snapshot

The scanner invokes Snapshot::capture (which calls capture_inner), retrieving a flattened representation of the entire document tree including cross-origin iframes. This occurs without executing JavaScript in the page context, as the snapshot is generated by the browser’s C++ layer.

Step 4: Fast Tick vs. Full Scan

The architecture implements a dual-mode scanning strategy:

• Fast DOM-only scan (≈2 seconds): Calls dom_snapshot::capture_messages to extract visible message rows from the DOM and computes a hash of the results. Emits updates only when the hash changes, minimizing overhead for idle windows.
• Full scan (≈30 seconds): Combines the DOM capture with idb::walk to traverse IndexedDB stores (e.g., WhatsApp’s model-storage and signal-storage), retrieving message metadata, sender JIDs, and timestamps unavailable in the DOM.

Step 5: Merging DOM and IndexedDB Data

In app/src-tauri/src/whatsapp_scanner/mod.rs (lines 84–100), the merge_dom_into_snapshot function reconciles results. It patches empty message bodies from the DOM with full text retrieved from IndexedDB and appends any unmatched rows, ensuring data completeness without duplication.

Step 6: Emitting to the Core and UI

Finally, the merged transcript is dual-written: emitted as a webview:event for the React frontend and posted via openhuman.memory_doc_ingest to the Rust core sidecar for persistence (lines 88–110 and 119–140).

Why CDP-Based Scraping?

The architecture of the web scraper tool prioritizes CDP over HTTP clients for several technical advantages:

• Zero JavaScript injection: Snapshots are captured at the C++ layer, avoiding security risks and side effects of injecting scraping scripts into the page context.
• Cross-frame consistency: DOMSnapshot.captureSnapshot merges the main frame and all iframe documents into a unified node tree, ensuring content inside nested frames (e.g., Gmail messages) is captured.
• Incremental updates: The fast tick mechanism re-captures only the DOM and checks a content hash, producing zero traffic when the window is idle.
• Rich metadata: The full scan supplements DOM text with IndexedDB-derived fields like precise timestamps and normalized contact identifiers.

Summary

• Transport Layer: CdpConn in app/src-tauri/src/cdp/snapshot.rs handles WebSocket JSON-RPC to the embedded Chromium instance.
• Data Structure: Snapshot parses flat CDP responses into traversable DOM trees with optional layout rectangles.
• Orchestration: Provider scanners in app/src-tauri/src/whatsapp_scanner/mod.rs (and similar modules) implement two-phase extraction: fast DOM hashes (2s) and deep IndexedDB walks (30s).
• Data Flow: Results merge via merge_dom_into_snapshot before dual-emission to the UI and core memory stores via openhuman.memory_doc_ingest.
• Security: The CDP approach executes no JavaScript in the target page, relying on native browser snapshotting for stability.

Frequently Asked Questions

Does OpenHuman use traditional HTTP scraping libraries like requests or Scrapy?

No. According to the source code in the tinyhumansai/openhuman repository, the scraper eschews HTTP clients in favor of driving an embedded Chromium instance via the Chrome DevTools Protocol. This allows it to retrieve dynamically rendered content from SPAs without parsing JavaScript or managing cookies manually.

How does the scraper handle Single Page Applications with heavy client-side rendering?

The Snapshot component captures the complete DOM state—including shadow DOM and iframe contents—using DOMSnapshot.captureSnapshot, which returns the rendered tree directly from the browser’s layout engine. This eliminates the need to execute or wait for application JavaScript, as the snapshot reflects the current visual state.

What is the performance difference between the fast scan and full scan?

The fast scan completes in approximately 2 seconds by querying only the visible DOM and comparing a hash of the message rows, making it suitable for continuous background monitoring. The full scan takes roughly 30 seconds because it additionally walks IndexedDB object stores (via idb::walk) to retrieve metadata not exposed in the DOM, such as full resolution contact details and precise timestamps.

Where are the scanner implementations located for different chat providers?

Each provider has a dedicated module under app/src-tauri/src/: whatsapp_scanner/, telegram_scanner/, discord_scanner/, and slack_scanner/. All follow the same architectural pattern defined in the core CDP modules, differing only in their CSS selectors and IndexedDB store names used to locate conversation threads.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}