# How the closed_book_proxy Integration in Switchyard Uses an Allowlist Rewriter to Restrict Agent Traffic During Benchmarking > Learn how Switchyard's closed_book_proxy uses an allowlist rewriter with Mitmproxy to block unauthorized agent traffic during benchmarking, ensuring data integrity and efficient testing. - Repository: [NVIDIA-NeMo/Switchyard](https://github.com/NVIDIA-NeMo/Switchyard) - Tags: how-to-guide - Published: 2026-09-13 --- **The closed_book_proxy integration in NVIDIA-NeMo/Switchyard routes all outbound HTTP traffic through a Mitmproxy addon that enforces a dataset-specific allowlist, blocking any unauthorized external hosts with a 403 response during closed-book benchmarking.** The closed_book_proxy integration provides a deterministic sandbox for evaluating AI agents by controlling network egress in Switchyard’s benchmarking environment. Located in the NVIDIA-NeMo/Switchyard repository, this component ensures that tasks running in Docker side-cars can only communicate with explicitly approved external domains. By implementing an allowlist rewriter, the system prevents data exfiltration while maintaining access to required package repositories and data sources. ## Architecture of the closed_book_proxy Integration The closed_book_proxy operates as a Mitmproxy addon within the Docker side-car, intercepting every outbound HTTP request during closed-book benchmarking runs. The core logic resides in [`benchmark/closed_book_proxy/proxy/rewriter.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/benchmark/closed_book_proxy/proxy/rewriter.py), where the addon initializes from environment variables and applies filtering rules at the network edge. ### Environment Configuration Variables When the side-car container starts, the addon reads two critical environment variables: - `CLOSED_BOOK_MODE` (default: enabled) – Toggles the proxy functionality on or off - `SWITCHYARD_PROXY_ALLOWLIST` (default: [`/etc/proxy-public/allowed_domains.txt`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main//etc/proxy-public/allowed_domains.txt)) – Specifies the path to the hostname allowlist The `_load_allowed_hosts` method parses this file into a Python `set` of permitted hostnames during addon initialization. ```python # From benchmark/closed_book_proxy/proxy/rewriter.py def _load_allowed_hosts(self): allowlist_path = os.environ.get( "SWITCHYARD_PROXY_ALLOWLIST", "/etc/proxy-public/allowed_domains.txt" ) with open(allowlist_path) as f: self.allowed_hosts = {line.strip() for line in f if line.strip()} ``` ## Allowlist Enforcement and Traffic Filtering For every intercepted request, the proxy evaluates authorization through the `_deny_if_needed` method, which handles both `requestheaders` and `http_connect` events. The enforcement logic applies three distinct rules to determine whether a connection proceeds. ### Host Normalization and Validation Rules The `_normalized_host` function standardizes the target hostname before validation. The addon permits traffic matching any of these criteria: 1. **Localhost and loopback addresses** – Always allowed for internal communication 2. **Exact hostname matches** – Present in the `allowed_hosts` set loaded from file 3. **Subdomain inheritance** – Hosts ending with an allowed domain suffix (e.g., `subdomain.pypi.org` matches `pypi.org`) According to lines 94-103 of [`benchmark/closed_book_proxy/proxy/rewriter.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/benchmark/closed_book_proxy/proxy/rewriter.py), the subdomain check uses Python’s `str.endswith` method: ```python def _allowed(self, host: str) -> bool: host = _normalized_host(host) if not host: return True if host in {"localhost", "127.0.0.1", "::1", "proxy"}: return True if host in self.allowed_hosts: return True return any(host.endswith(f".{allowed}") for allowed in self.allowed_hosts) ``` ### Blocking Unauthorized Requests When a host fails all validation checks, the proxy immediately terminates the connection. The `_deny_if_needed` method returns an HTTP **403 Forbidden** response with a JSON error payload, preventing the agent from establishing unauthorized outbound connections. As implemented in lines 112-121 of [`benchmark/closed_book_proxy/proxy/rewriter.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/benchmark/closed_book_proxy/proxy/rewriter.py), the response includes diagnostic information while blocking the request: ```python def _deny_if_needed(self, flow): if not self._allowed(flow.request.pretty_host): flow.response = http.Response.make( 403, json.dumps({"error": "Host not in allowlist", "host": flow.request.pretty_host}), {"Content-Type": "application/json"} ) ``` ## Dataset-Specific Allowlist Generation Before executing a benchmark, Switchyard dynamically constructs environment-specific allowlists based on the Harbor dataset requirements. The [`benchmark/prepare_harbor_dataset.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/benchmark/prepare_harbor_dataset.py) script orchestrates this process, ensuring the proxy only permits hosts necessary for the specific evaluation task. ### Harvesting Required Hosts The `_proxy_allowlist_hosts_for_dataset` function returns a tuple of required external hosts (such as package mirrors or data sources) for the chosen dataset. For example, terminal-based benchmarks require access to `pypi.org` and `archive.ubuntu.com`: ```python def _proxy_allowlist_hosts_for_dataset(source_dataset: str) -> tuple[str, ...]: if source_dataset.startswith("terminal-bench"): return ("pypi.org", "archive.ubuntu.com") return () ``` ### Merging and Deployment The `_merge_compose` function writes the collected hosts into [`allowlist-base.txt`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/allowlist-base.txt) under the `environment/proxy` directory. During container initialization, this file mounts into the side-car at the path specified by `SWITCHYARD_PROXY_ALLOWLIST`, creating a tightly restricted network environment tailored to the benchmark’s legitimate dependencies. ### Configuring the Benchmark Environment Operators can override the default allowlist by mounting custom files and setting environment variables: ```bash # Create a dataset-specific allowlist cat > custom_allowlist.txt < ``` ## Hosted-Tool Stripping for Data Leakage Prevention Beyond network filtering, the closed_book_proxy integration includes a `_strip_hosted_tools` method that removes potentially dangerous payloads from outbound JSON requests. This functionality eliminates fields like `web_search` and `code_execution` from agent communications, preventing the unintentional transmission of sensitive data to external services. All stripping operations log to `strip.jsonl` for audit purposes. ## Summary - The **closed_book_proxy integration** in NVIDIA-NeMo/Switchyard uses a Mitmproxy addon to intercept all outbound HTTP traffic during benchmarking. - **Environment variables** `CLOSED_BOOK_MODE` and `SWITCHYARD_PROXY_ALLOWLIST` configure the proxy and specify the allowlist file path. - The **`_allowed` method** enforces three rules: localhost is always permitted, exact hostname matches pass, and subdomains of allowed domains are accepted. - Unauthorized hosts receive an immediate **HTTP 403 response** with a JSON error payload, blocking the connection attempt. - **Dataset-specific allowlists** are generated via [`benchmark/prepare_harbor_dataset.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/benchmark/prepare_harbor_dataset.py), ensuring only required external hosts (like package repositories) are accessible. - The addon additionally **strips hosted-tool payloads** from JSON requests to prevent data exfiltration through external service calls. ## Frequently Asked Questions ### How does the closed_book_proxy integration determine which hostnames to allow? The integration loads hostnames from a plain text file specified by the `SWITCHYARD_PROXY_ALLOWLIST` environment variable (defaulting to [`/etc/proxy-public/allowed_domains.txt`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main//etc/proxy-public/allowed_domains.txt)). The `_load_allowed_hosts` function reads this file into a Python set during addon initialization. Additionally, [`benchmark/prepare_harbor_dataset.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/benchmark/prepare_harbor_dataset.py) generates dataset-specific allowlists containing only the external hosts required for that particular benchmark task. ### What happens when an agent attempts to connect to a non-allowlisted domain? The proxy immediately blocks the connection by returning an HTTP 403 Forbidden response with a JSON error body indicating the blocked hostname. This occurs in the `_deny_if_needed` method within [`benchmark/closed_book_proxy/proxy/rewriter.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/benchmark/closed_book_proxy/proxy/rewriter.py), which intercepts both `requestheaders` and `http_connect` events before any data transmits to the external host. ### Can subdomains of allowed domains be accessed automatically? Yes. The `_allowed` method in [`rewriter.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/rewriter.py) checks if the target hostname ends with any domain in the allowlist using `host.endswith(f".{allowed}")`. This means if `pypi.org` is allowlisted, the agent can also reach `files.pythonhosted.org` or other subdomains without explicit individual entries. ### How does closed_book_proxy prevent agents from leaking data through tool calls? The addon includes a `_strip_hosted_tools` function that parses outbound JSON requests and removes fields associated with external tool execution (such as `web_search` or `code_execution`). This ensures that even if an agent constructs a payload intended for an external service, the proxy sanitizes the request before transmission, logging the modification to `strip.jsonl`.