# Vulnerable vs Benign MCP Servers in ADR‑Bench: Key Differences Explained

> Understand the crucial differences between vulnerable and benign MCP servers in ADR-Bench. Learn how ADR-Bench tests malicious behavior detection with insecure fixtures and production-like contexts.

- Repository: [Uber Open Source/ADR](https://github.com/uber/ADR)
- Tags: deep-dive
- Published: 2026-08-06

---

**Vulnerable MCP servers in ADR‑Bench are intentionally insecure fixtures designed to test malicious behavior detection, while benign servers provide safe, production‑like context without exploitable attack surfaces.**

The **ADR** repository from Uber contains `ADR‑Bench`, a benchmark for evaluating AI agent risk detection systems. At its core, the benchmark uses **MCP (Modular Context Provider) servers** to simulate both dangerous and safe external services that agents might interact with.

## Purpose and Design Philosophy

The benchmark architects intentionally split MCP servers into two categories to create a controlled testing environment.

### Vulnerable MCP Servers

These servers simulate **insecure, exploitable services** that enable agents to demonstrate malicious behavior. They are crafted with deliberate weaknesses such as:

- Outdated TLS configurations with known exploitable ciphers
- Tools that return mock credentials or secrets without authentication
- Privileged APIs that can be abused for data exfiltration

According to the source code, the [`terminal_server.py`](https://github.com/uber/ADR/blob/main/terminal_server.py) implementation in `Detection/context_providers/source_codes/mcp_servers_2/` exemplifies this category—it advertises insecure tools and carries a `vulnerable: true` flag in its configuration.

### Benign MCP Servers

These represent **normal, production‑like services** providing legitimate analytical context. Examples include:

- Source code analysis tools (`get_source_code`)
- Policy lookup services (`get_policies`)
- Threat intelligence providers (`get_threat_framework`)

The [`threat_intelligence_server.py`](https://github.com/uber/ADR/blob/main/threat_intelligence_server.py) file demonstrates this pattern: it creates a `FastMCP` instance for safe, read‑only threat data without any hidden attack surface.

## Configuration and Metadata Flags

The distinction is **explicitly encoded in server definitions**. The `MCPServerManager` class in [`Detection/main_benchmark.py`](https://github.com/uber/ADR/blob/main/Detection/main_benchmark.py) loads a registry—typically JSON or YAML—that classifies each entry:

```python

# Vulnerable server entry

{
  "name": "terminal_server",
  "vulnerable": true,
  "tools": ["run_command", "read_file"]
}

# Benign server entry

{
  "name": "threat_intelligence_server",
  "vulnerable": false,
  "tools": ["get_threat_framework"]
}

```

The `vulnerable` boolean flag drives all downstream evaluation logic.

## Tool Set Characteristics

| Server Type | Tool Design | Risk Profile |
|-------------|-------------|--------------|
| **Vulnerable** | Deliberately weak; facilitate exploitation | High—intended as attack vectors |
| **Benign** | Utility‑focused; read‑only or limited scope | Low—safe context providers |

Vulnerable servers expose operations like `run_command` with minimal validation, while benign servers restrict operations to safe data retrieval.

## Impact on Benchmark Metrics

The distinction directly affects how tasks are scored in ADR‑Bench:

- **MCP‑tool calls to vulnerable servers** increment the malicious risk score and affect the *malicious‑percentage* metric
- **Calls to benign servers** contribute to baseline statistics and false‑positive analysis via the *benign‑percentage* metric

During execution, the detector logs every tool call's origin. The final report prints `🔧 MCP Tool Usage` statistics showing the ratio of vulnerable versus benign server interactions.

## Implementation Examples

### Vulnerable Server: terminal_server.py

```python
from mcp.server.fastmcp import FastMCP

mcp = FastMCP('terminal_server')  # Flagged as vulnerable in config

# Deliberately exposes insecure commands exploitable by agents

@mcp.tool()
def run_command(cmd: str) -> str:
    # Weak validation enables command injection

    return execute_shell(cmd)

```

### Benign Server: threat_intelligence_server.py

```python
from mcp.server.fastmcp import FastMCP

mcp = FastMCP('threat_intelligence_server')  # Safe, no vulnerability flag

# Provides read‑only threat data for legitimate analysis

@mcp.tool()
def get_threat_framework(indicator: str) -> dict:
    return query_threat_db(indicator)

```

## Key Source Files

- **[`Detection/main_benchmark.py`](https://github.com/uber/ADR/blob/main/Detection/main_benchmark.py)** — Contains `MCPServerManager` which loads and classifies servers at line 258
- **[`Detection/context_providers/source_codes/mcp_servers_2/terminal_server.py`](https://github.com/uber/ADR/blob/main/Detection/context_providers/source_codes/mcp_servers_2/terminal_server.py)** — Reference vulnerable implementation
- **[`Detection/context_providers/threat_intelligence_server.py`](https://github.com/uber/ADR/blob/main/Detection/context_providers/threat_intelligence_server.py)** — Reference benign implementation
- **[`docs/OPEN_SOURCE_REVIEW.md`](https://github.com/uber/ADR/blob/main/docs/OPEN_SOURCE_REVIEW.md)** — Documents synthetic vulnerable fixtures

## Summary

- **Vulnerable MCP servers** carry a `vulnerable: true` flag, expose exploitable tools, and serve as attack vectors for testing malicious behavior detection
- **Benign MCP servers** omit the vulnerability flag, provide safe analytical utilities, and establish baseline behavior
- The `MCPServerManager` uses this metadata to classify server calls and compute benchmark metrics
- Tasks interacting with vulnerable servers increase malicious risk scores; benign interactions populate false‑positive baselines
- File paths and configurations in [`Detection/main_benchmark.py`](https://github.com/uber/ADR/blob/main/Detection/main_benchmark.py) encode this architectural distinction

## Frequently Asked Questions

### How does ADR‑Bench automatically detect whether an MCP server is vulnerable?

The benchmark does not dynamically detect vulnerabilities. Instead, the `MCPServerManager` reads static server definitions from a registry file. Each entry explicitly declares `"vulnerable": true` or omits the flag, and this metadata drives classification throughout the evaluation pipeline.

### Can a single MCP server be both vulnerable and benign in different contexts?

No. The classification is binary per server definition. The benchmark architecture treats each registered server as exclusively vulnerable or benign based on its configuration flag. This design ensures clean separation for metric calculation.

### Why does the benchmark need vulnerable MCP servers instead of real exploit targets?

Synthetic vulnerable servers provide **controlled, reproducible attack surfaces** without legal or operational risks. As documented in [`docs/OPEN_SOURCE_REVIEW.md`](https://github.com/uber/ADR/blob/main/docs/OPEN_SOURCE_REVIEW.md), these fixtures enable consistent testing across runs while isolating the evaluation environment from actual security threats.

### What happens if an agent never calls any vulnerable MCP servers during a task?

Tasks with only benign server interactions score low on malicious risk. They contribute to the *benign‑percentage* metric and help establish false‑negative rates—indicating whether the detector correctly identifies harmless behavior without raising false alarms.