How the closed_book_proxy Integration in Switchyard Uses an Allowlist Rewriter to Restrict Agent Traffic During Benchmarking

The closed_book_proxy integration in NVIDIA-NeMo/Switchyard routes all outbound HTTP traffic through a Mitmproxy addon that enforces a dataset-specific allowlist, blocking any unauthorized external hosts with a 403 response during closed-book benchmarking.

The closed_book_proxy integration provides a deterministic sandbox for evaluating AI agents by controlling network egress in Switchyard’s benchmarking environment. Located in the NVIDIA-NeMo/Switchyard repository, this component ensures that tasks running in Docker side-cars can only communicate with explicitly approved external domains. By implementing an allowlist rewriter, the system prevents data exfiltration while maintaining access to required package repositories and data sources.

Architecture of the closed_book_proxy Integration

The closed_book_proxy operates as a Mitmproxy addon within the Docker side-car, intercepting every outbound HTTP request during closed-book benchmarking runs. The core logic resides in benchmark/closed_book_proxy/proxy/rewriter.py, where the addon initializes from environment variables and applies filtering rules at the network edge.

Environment Configuration Variables

When the side-car container starts, the addon reads two critical environment variables:

• CLOSED_BOOK_MODE (default: enabled) – Toggles the proxy functionality on or off
• SWITCHYARD_PROXY_ALLOWLIST (default: /etc/proxy-public/allowed_domains.txt) – Specifies the path to the hostname allowlist

The _load_allowed_hosts method parses this file into a Python set of permitted hostnames during addon initialization.

# From benchmark/closed_book_proxy/proxy/rewriter.py

def _load_allowed_hosts(self):
    allowlist_path = os.environ.get(
        "SWITCHYARD_PROXY_ALLOWLIST", 
        "/etc/proxy-public/allowed_domains.txt"
    )
    with open(allowlist_path) as f:
        self.allowed_hosts = {line.strip() for line in f if line.strip()}

Allowlist Enforcement and Traffic Filtering

For every intercepted request, the proxy evaluates authorization through the _deny_if_needed method, which handles both requestheaders and http_connect events. The enforcement logic applies three distinct rules to determine whether a connection proceeds.

Host Normalization and Validation Rules

The _normalized_host function standardizes the target hostname before validation. The addon permits traffic matching any of these criteria:

1. Localhost and loopback addresses – Always allowed for internal communication
2. Exact hostname matches – Present in the allowed_hosts set loaded from file
3. Subdomain inheritance – Hosts ending with an allowed domain suffix (e.g., subdomain.pypi.org matches pypi.org)

According to lines 94-103 of benchmark/closed_book_proxy/proxy/rewriter.py, the subdomain check uses Python’s str.endswith method:

def _allowed(self, host: str) -> bool:
    host = _normalized_host(host)
    if not host:
        return True
    if host in {"localhost", "127.0.0.1", "::1", "proxy"}:
        return True
    if host in self.allowed_hosts:
        return True
    return any(host.endswith(f".{allowed}") for allowed in self.allowed_hosts)

Blocking Unauthorized Requests

When a host fails all validation checks, the proxy immediately terminates the connection. The _deny_if_needed method returns an HTTP 403 Forbidden response with a JSON error payload, preventing the agent from establishing unauthorized outbound connections.

As implemented in lines 112-121 of benchmark/closed_book_proxy/proxy/rewriter.py, the response includes diagnostic information while blocking the request:

def _deny_if_needed(self, flow):
    if not self._allowed(flow.request.pretty_host):
        flow.response = http.Response.make(
            403,
            json.dumps({"error": "Host not in allowlist", "host": flow.request.pretty_host}),
            {"Content-Type": "application/json"}
        )

Dataset-Specific Allowlist Generation

Before executing a benchmark, Switchyard dynamically constructs environment-specific allowlists based on the Harbor dataset requirements. The benchmark/prepare_harbor_dataset.py script orchestrates this process, ensuring the proxy only permits hosts necessary for the specific evaluation task.

Harvesting Required Hosts

The _proxy_allowlist_hosts_for_dataset function returns a tuple of required external hosts (such as package mirrors or data sources) for the chosen dataset. For example, terminal-based benchmarks require access to pypi.org and archive.ubuntu.com:

def _proxy_allowlist_hosts_for_dataset(source_dataset: str) -> tuple[str, ...]:
    if source_dataset.startswith("terminal-bench"):
        return ("pypi.org", "archive.ubuntu.com")
    return ()

Merging and Deployment

The _merge_compose function writes the collected hosts into allowlist-base.txt under the environment/proxy directory. During container initialization, this file mounts into the side-car at the path specified by SWITCHYARD_PROXY_ALLOWLIST, creating a tightly restricted network environment tailored to the benchmark’s legitimate dependencies.

Configuring the Benchmark Environment

Operators can override the default allowlist by mounting custom files and setting environment variables:

# Create a dataset-specific allowlist

cat > custom_allowlist.txt <<EOF
pypi.org
archive.ubuntu.com
data.example.com
EOF

# Launch benchmark with custom restrictions

CLOSED_BOOK_MODE=1 \
SWITCHYARD_PROXY_ALLOWLIST=/etc/proxy-public/custom_allowlist.txt \
docker run --rm \
    -e CLOSED_BOOK_MODE \
    -e SWITCHYARD_PROXY_ALLOWLIST \
    -v $(pwd)/custom_allowlist.txt:/etc/proxy-public/custom_allowlist.txt \
    nvidia/switchyard:latest \
    /benchmark/run-baseline.sh <task-id>

Hosted-Tool Stripping for Data Leakage Prevention

Beyond network filtering, the closed_book_proxy integration includes a _strip_hosted_tools method that removes potentially dangerous payloads from outbound JSON requests. This functionality eliminates fields like web_search and code_execution from agent communications, preventing the unintentional transmission of sensitive data to external services. All stripping operations log to strip.jsonl for audit purposes.

Summary

• The closed_book_proxy integration in NVIDIA-NeMo/Switchyard uses a Mitmproxy addon to intercept all outbound HTTP traffic during benchmarking.
• Environment variables CLOSED_BOOK_MODE and SWITCHYARD_PROXY_ALLOWLIST configure the proxy and specify the allowlist file path.
• The _allowed method enforces three rules: localhost is always permitted, exact hostname matches pass, and subdomains of allowed domains are accepted.
• Unauthorized hosts receive an immediate HTTP 403 response with a JSON error payload, blocking the connection attempt.
• Dataset-specific allowlists are generated via benchmark/prepare_harbor_dataset.py, ensuring only required external hosts (like package repositories) are accessible.
• The addon additionally strips hosted-tool payloads from JSON requests to prevent data exfiltration through external service calls.

Frequently Asked Questions

How does the closed_book_proxy integration determine which hostnames to allow?

The integration loads hostnames from a plain text file specified by the SWITCHYARD_PROXY_ALLOWLIST environment variable (defaulting to /etc/proxy-public/allowed_domains.txt). The _load_allowed_hosts function reads this file into a Python set during addon initialization. Additionally, benchmark/prepare_harbor_dataset.py generates dataset-specific allowlists containing only the external hosts required for that particular benchmark task.

What happens when an agent attempts to connect to a non-allowlisted domain?

The proxy immediately blocks the connection by returning an HTTP 403 Forbidden response with a JSON error body indicating the blocked hostname. This occurs in the _deny_if_needed method within benchmark/closed_book_proxy/proxy/rewriter.py, which intercepts both requestheaders and http_connect events before any data transmits to the external host.

Can subdomains of allowed domains be accessed automatically?

Yes. The _allowed method in rewriter.py checks if the target hostname ends with any domain in the allowlist using host.endswith(f".{allowed}"). This means if pypi.org is allowlisted, the agent can also reach files.pythonhosted.org or other subdomains without explicit individual entries.

How does closed_book_proxy prevent agents from leaking data through tool calls?

The addon includes a _strip_hosted_tools function that parses outbound JSON requests and removes fields associated with external tool execution (such as web_search or code_execution). This ensures that even if an agent constructs a payload intended for an external service, the proxy sanitizes the request before transmission, logging the modification to strip.jsonl.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}