How the Network Domain in Lightpanda Captures HTTP Response Data

Lightpanda's CDP Network domain captures HTTP response bodies by appending incremental data chunks to per-request ArrayList(u8) buffers stored in BrowserContext.captured_responses, which are then retrieved on demand via the Network.getResponseBody command.

The Network domain in Lightpanda exposes Chrome DevTools Protocol (CDP) functionality for inspecting HTTP traffic during browser automation. According to the lightpanda-io/browser source code, the implementation uses an event-driven architecture where the browser's internal HTTP client streams response data to registered CDP handlers. This data accumulates in memory until the client explicitly requests the complete body.

Enabling the Network Domain

Capture begins when a CDP client sends the Network.enable command. In src/cdp/cdp.zig, this invokes BrowserContext.networkEnable() (lines 44-50), which registers a set of notification handlers including http_response_data. This registration wires the CDP domain to the browser's low-level HTTP event stream, allowing subsequent response chunks to be intercepted.

Capturing Response Data Chunks

As the underlying HTTP client receives response data, it emits Notification.ResponseData events for each chunk. The CDP implementation routes these events to the onHttpResponseData handler defined in src/cdp/cdp.zig (lines 46-56).

The Notification Handler

The onHttpResponseData function looks up the internal transfer.id (a numeric request identifier) in the browser context's storage map. If no entry exists for that request, the handler initializes an empty ArrayList(u8) buffer. The incoming byte chunk is then appended to this buffer, incrementally building the complete response body.

Storage Architecture

The storage mechanism is declared in the BrowserContext struct at line 88 of src/cdp/cdp.zig (lines 84-90):

captured_responses: std.AutoHashMapUnmanaged(usize, std.ArrayList(u8)),

This hash map uses the numeric transfer.id as the key and maintains a growable byte buffer for each active request. When a new page is created, src/cdp/domains/page.zig resets this map to ensure clean state between navigation sessions and prevent cross-page data leakage.

Retrieving Captured Bodies via CDP

When the client requests a response body using Network.getResponseBody, the implementation in src/cdp/domains/network.zig (lines 98-110) executes a three-step lookup:

1. ID Conversion: Parses the CDP request ID string (e.g., "REQ-42") back to the internal numeric transfer.id using idFromRequestId
2. Buffer Retrieval: Obtains a pointer from bc.captured_responses.getPtr(request_id), returning error.RequestNotFound if the request was not captured
3. Response Formation: Returns the buffer contents as body with base64Encoded set to false

Complete Example: Capturing and Fetching Response Data

The following Zig code demonstrates the full lifecycle of enabling capture and retrieving a specific response:

// Enable network monitoring
try ctx.processMessage(.{
    .id = 1,
    .method = "Network.enable",
    .params = .{},
});

// After HTTP traffic occurs (e.g., a fetch in the page),
// retrieve the body for request "REQ-42"
try ctx.processMessage(.{
    .id = 2,
    .method = "Network.getResponseBody",
    .params = .{ .requestId = "REQ-42" },
});

The internal retrieval logic in src/cdp/domains/network.zig handles the ID translation and buffer extraction:

const request_id = try idFromRequestId(params.requestId);
const bc = cmd.browser_context orelse return error.BrowserContextNotLoaded;
const buf = bc.captured_responses.getPtr(request_id) orelse return error.RequestNotFound;
try cmd.sendResult(.{
    .body = buf.items,
    .base64Encoded = false,
}, .{});

Summary

• Incremental Capture: The onHttpResponseData handler in src/cdp/cdp.zig appends each chunk to a per-request buffer as Notification.ResponseData events arrive
• In-Memory Storage: BrowserContext.captured_responses maintains ArrayList(u8) buffers keyed by numeric transfer.id (declared at line 88 of src/cdp/cdp.zig)
• On-Demand Retrieval: The Network.getResponseBody command in src/cdp/domains/network.zig converts string request IDs, looks up buffers, and returns raw bytes
• Lifecycle Management: src/cdp/domains/page.zig resets the capture map when creating new pages to isolate session data

Frequently Asked Questions

How does Lightpanda store HTTP response chunks internally?

Lightpanda stores response chunks in std.ArrayList(u8) buffers managed by the captured_responses hash map within BrowserContext. Each request uses its internal transfer.id as the key, and the onHttpResponseData handler appends incoming bytes as they arrive from the HTTP client's Notification.ResponseData events.

What CDP command retrieves captured response bodies?

The Network.getResponseBody command retrieves captured bodies from src/cdp/domains/network.zig. It converts the CDP-style request ID (e.g., "REQ-42") to an internal numeric ID via idFromRequestId, looks up the buffer in captured_responses, and returns the data with base64Encoded: false.

When does Lightpanda start capturing network data?

Capture activates immediately when the client sends Network.enable, which triggers BrowserContext.networkEnable() to register the http_response_data notification handler. Only responses received after this enabling event are stored; earlier traffic is not retroactively captured.

How is the captured response data cleared?

The captured_responses map is reset when a new page is created, as implemented in src/cdp/domains/page.zig. This ensures that response data from previous navigation sessions does not persist into new page contexts, preventing memory leaks and data contamination across browser sessions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}