How to Configure Custom HTTP Headers and Proxy Settings in Lightpanda

Lightpanda's HTTP stack is built on libcurl and wrapped by the src/network/http.zig module, accepting a Config instance that controls custom headers, proxy URLs, and authentication tokens either programmatically or via command-line flags.

The lightpanda-io/browser project is a Zig-based headless browser engine that centralizes its network configuration in the Config struct. Understanding how to configure custom HTTP headers and proxy settings in Lightpanda requires interacting with the src/Config.zig parser and the src/network/http.zig connection handler, which together manage the underlying libcurl integration.

Architecture Overview

Lightpanda delegates all HTTP operations to libcurl through an abstraction layer in src/network/http.zig. Every network request originates from a Config instance that stores proxy endpoints, header definitions, and TLS options. This configuration is populated either through Config.parseCommonArg (src/Config.zig, lines 221‑311) when parsing CLI arguments, or manually when using Lightpanda as a library.

Configuring Custom HTTP Headers

User-Agent Customization

The baseline User-Agent string is generated inside Config.HttpHeaders (src/Config.zig, lines 49‑78). You can extend this default agent by providing the --user_agent_suffix flag at startup, which appends your text to the standard Lightpanda/1.0 identifier.

Adding Arbitrary Headers

For request-specific headers, use the Headers.add method defined in src/network/http.zig (lines 83‑91). This method builds a linked list of header strings that Lightpanda eventually passes to libcurl via curl_easy_setopt(..., .http_header, ...) at lines 62‑66.

Proxy Authentication Headers

When a bearer token is configured, Connection.secretHeaders (http.zig, lines 54‑60) automatically injects the Proxy-Authorization: Bearer <token> header into the request. This happens transparently during connection initialization and does not require manual header construction.

Setting Up Proxy Configuration

The proxy URL is retrieved via Config.httpProxy() (Config.zig, lines 78‑83). During Connection.init (http.zig, lines 58‑63), Lightpanda applies this URL to the curl easy handle using curl_easy_setopt(..., .proxy, ...). If TLS host verification is disabled through --insecure_disable_tls_host_verification, the setting is also applied to the proxy tunnel (http.zig, lines 70‑79).

Programmatic Configuration in Zig

The following example demonstrates how to construct a Config instance, set a proxy with authentication, and inject custom headers before executing a request:

const std = @import("std");
const Config = @import("src/Config.zig");
const http = @import("src/network/http.zig");

// Initialize configuration with an allocator
var cfg = try Config.init(std.heap.page_allocator, "lightpanda", .{ .serve = .{} });

// Configure proxy settings
cfg.common.http_proxy = try std.heap.page_allocator.dupeZ(u8, "http://proxy.example:3128");
cfg.common.proxy_bearer_token = try std.heap.page_allocator.dupeZ(u8, "abc123");

// Initialize connection with the configuration
var conn = try http.Connection.init(null, &cfg);
defer conn.deinit();

// Build headers starting with the User-Agent
var hdrs = try http.Headers.init(cfg.http_headers.user_agent_header);
defer hdrs.deinit();

// Add custom headers
try hdrs.add("X-My-Header: hello");
try hdrs.add("X-Trace-Id: 0a1b2c3d");

// Apply headers to the connection
try conn.setHeaders(&hdrs);

// The connection is now ready for requests (e.g., conn.setURL(...), conn.setMethod(...))

Command-Line Interface Options

Lightpanda exposes network configuration through flags parsed by Config.parseCommonArg in src/Config.zig (lines 221‑311):

• --http_proxy <url> – Sets the HTTP proxy URL used for all requests (e.g., http://proxy:3128).
• --proxy_bearer_token <token> – Automatically adds Proxy-Authorization: Bearer <token> via Connection.secretHeaders.
• --user_agent_suffix <text> – Appends the provided text to the default User-Agent string.
• --insecure_disable_tls_host_verification – Disables TLS host verification for both direct and proxy connections.

Example usage:

lightpanda fetch \
    --http_proxy http://proxy.local:8080 \
    --proxy_bearer_token mytoken123 \
    --user_agent_suffix "MyCrawler/0.1" \
    https://example.org

How Configuration Flows Through the System

1. CLI Parsing – Config.parseCommonArg populates Config.Common fields including http_proxy and proxy_bearer_token.
2. Header Generation – HttpHeaders.init (lines 49‑78) constructs the baseline User-Agent and prepares the secret header list.
3. Connection Setup – Connection.init reads proxy and TLS settings from Config, applying them to the libcurl easy handle.
4. Request Execution – Connection.request creates a fresh Headers list, merges secret headers, applies custom additions via Headers.add, and invokes curl_easy_setopt(..., .http_header, ...).

Summary

• Lightpanda delegates HTTP transport to libcurl via the abstraction in src/network/http.zig.
• All network options flow through the Config struct, parsed by Config.parseCommonArg in src/Config.zig (lines 221‑311).
• Add custom headers programmatically using Headers.add (http.zig, lines 83‑91) before calling conn.setHeaders().
• Proxy URLs are set via --http_proxy or programmatically through cfg.common.http_proxy.
• Authentication tokens are handled automatically by Connection.secretHeaders (http.zig, lines 54‑60).

Frequently Asked Questions

How do I add multiple custom headers to a single request?

Call hdrs.add() repeatedly for each header string before invoking conn.setHeaders(). Each call appends to the internal list managed by the Headers struct in src/network/http.zig (lines 83‑91). All headers in the list are passed to libcurl simultaneously during the request phase.

Does Lightpanda support authenticated corporate proxies?

Yes. Provide the --proxy_bearer_token flag or set cfg.common.proxy_bearer_token programmatically. The Connection.secretHeaders function (http.zig, lines 54‑60) automatically transforms this token into a Proxy-Authorization: Bearer <token> header for every request.

Can I disable TLS verification for proxy connections only?

The --insecure_disable_tls_host_verification flag disables verification for both direct and proxy connections (http.zig, lines 70‑79). The current implementation does not provide a separate toggle for proxy-only verification; the setting applies globally to the curl handle.

Where is the default User-Agent string defined?

The default User-Agent is constructed in Config.HttpHeaders within src/Config.zig (lines 49‑78). It combines the base "Lightpanda" identifier with any suffix provided via --user_agent_suffix, storing the final string in cfg.http_headers.user_agent_header for use by Headers.init().

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}