# How to Create Evaluations for MCP Servers to Test LLM Effectiveness

> Learn to create evaluations for MCP servers using the CLI tool to test LLM effectiveness. Drive Claude models against XML-defined QA pairs and measure tool usage.

- Repository: [Anthropic/skills](https://github.com/anthropics/skills)
- Tags: how-to-guide
- Published: 2026-02-16

---

**The MCP evaluation harness is a CLI tool that drives a Claude model against XML-defined question-answer pairs to measure how effectively an LLM uses MCP server tools.**

When you create evaluations for MCP servers, you validate whether your LLM can correctly orchestrate tool calls to produce accurate results. The `anthropics/skills` repository provides a complete harness in `skills/mcp-builder/scripts/` that handles transport abstraction, agent loops, and automated scoring.

## Understanding the MCP Evaluation Harness

The evaluation system treats your MCP server as a black box and the Claude model as the test subject. The harness lives in [`skills/mcp-builder/scripts/evaluation.py`](https://github.com/anthropics/skills/blob/main/skills/mcp-builder/scripts/evaluation.py) and implements an **agent loop** (`agent_loop()`) that mediates between the model and server.

The architecture follows this flow:

1. Parse the XML evaluation file containing `<qa_pair>` elements
2. Establish transport via `create_connection()` in [`connections.py`](https://github.com/anthropics/skills/blob/main/connections.py)
3. Discover available tools using `MCPConnection.list_tools()`
4. Execute the agent loop, forwarding `tool_use` blocks to `MCPConnection.call_tool()`
5. Extract the final answer from the `<response>` XML tag (enforced by `EVALUATION_PROMPT`)
6. Score against expected answers and render a markdown report

The harness supports three transport mechanisms through concrete subclasses of `MCPConnection`: **stdio** for local subprocesses, **SSE** for Server-Sent Events streams, and **HTTP** for REST endpoints.

## Setting Up Your Evaluation XML File

### Required XML Structure

Create an XML file that wraps each test case in a `<qa_pair>` element. The harness expects exactly two child elements per pair: `<question>` and `<answer>`.

```xml
<!-- eval.xml -->
<evaluation>
   <qa_pair>
      <question>What is the capital of France?</question>
      <answer>Paris</answer>
   </qa_pair>
   <qa_pair>
      <question>Compute 7 × 8.</question>
      <answer>56</answer>
   </qa_pair>
</evaluation>

```

Save this file to disk (e.g., [`my_eval.xml`](https://github.com/anthropics/skills/blob/main/my_eval.xml)). The evaluation harness will parse this file, iterate over each `qa_pair`, and compare the model's output against your expected answers.

## Running Evaluations Against MCP Servers

### Local stdio Transport

For MCP servers implemented as local scripts or compiled binaries, use the **stdio** transport. The harness spawns the server as a subprocess and communicates over stdin/stdout.

```bash
python -m skills.mcp-builder.scripts.evaluation \
   -t stdio \
   -c python \
   -a my_server.py \
   my_eval.xml

```

* `-t stdio` selects the stdio transport
* `-c python` specifies the interpreter command
* `-a my_server.py` passes the server script as an argument

### Remote HTTP Transport

To evaluate against a hosted MCP server exposing an HTTP API, use the **HTTP** transport. This requires the server URL and optional authentication headers.

```bash
python -m skills.mcp-builder.scripts.evaluation \
   -t http \
   -u https://my-mcp.example.com/v1 \
   -H "Authorization: Bearer $TOKEN" \
   my_eval.xml

```

* `-u` supplies the server base URL
* `-H` provides HTTP headers for authentication or custom configuration

### Server-Sent Events (SSE) Transport

For servers streaming responses via SSE, use the **SSE** transport. This maintains a persistent connection suitable for long-running tool executions.

```bash
python -m skills.mcp-builder.scripts.evaluation \
   -t sse \
   -u https://sse-mcp.example.com \
   my_eval.xml

```

## Customizing Model Selection and Parameters

By default, the harness uses a standard Claude model configuration, but you can override this with the `-m` flag to test different model versions or snapshots.

```bash
python -m skills.mcp-builder.scripts.evaluation \
   -t stdio \
   -c python \
   -a my_server.py \
   -m claude-3-5-sonnet-20241022 \
   my_eval.xml

```

This flexibility allows you to benchmark tool-use effectiveness across different Claude model generations or compare performance against specific model snapshots.

## Programmatic Evaluation in Python

For integration into CI/CD pipelines or custom testing frameworks, import the evaluation harness directly in Python. The `run_evaluation()` function in [`skills/mcp-builder/scripts/evaluation.py`](https://github.com/anthropics/skills/blob/main/skills/mcp-builder/scripts/evaluation.py) handles the full lifecycle asynchronously.

```python
import asyncio
from pathlib import Path
from skills.mcp-builder.scripts.evaluation import run_evaluation
from skills.mcp-builder.scripts.connections import create_connection

async def main():
    eval_path = Path("my_eval.xml")
    
    # Create stdio connection to local MCP server

    conn = create_connection(
        transport="stdio",
        command="python",
        args=["my_server.py"]
    )
    
    async with conn:
        report = await run_evaluation(
            eval_path, 
            conn, 
            model="claude-3-7-sonnet-20250219"
        )
        print(report)

asyncio.run(main())

```

This approach gives you full control over connection lifecycle, model parameters, and report handling while maintaining the same scoring logic as the CLI.

## Summary

- **Create evaluations for MCP servers** by authoring XML files with `<qa_pair>` elements containing questions and expected answers.
- The evaluation harness in [`skills/mcp-builder/scripts/evaluation.py`](https://github.com/anthropics/skills/blob/main/skills/mcp-builder/scripts/evaluation.py) orchestrates Claude model interactions through an agent loop that handles `tool_use` blocks and extracts answers from `<response>` tags.
- Support for **stdio**, **HTTP**, and **SSE** transports via `create_connection()` in [`connections.py`](https://github.com/anthropics/skills/blob/main/connections.py) enables testing against local scripts or remote hosted servers.
- Use CLI flags (`-t`, `-u`, `-m`) or the Python `run_evaluation()` API to customize transport, authentication, and model selection.
- The harness automatically scores results and generates markdown reports comparing model outputs against expected answers.

## Frequently Asked Questions

### What is the MCP evaluation harness?

The MCP evaluation harness is a testing framework in the `anthropics/skills` repository that measures how effectively a Claude model uses tools exposed by an MCP server. It drives the model through a series of questions defined in an XML file, captures the model's tool invocations via `MCPConnection.call_tool()`, and scores the final answers against expected values.

### How do I structure an evaluation XML file?

Evaluation XML files must contain a root `<evaluation>` element with one or more `<qa_pair>` children. Each pair requires exactly two child elements: `<question>` containing the prompt to send to the model, and `<answer>` containing the expected correct response. The model must wrap its final output in a `<response>` tag, which the harness extracts for scoring.

### Can I test against remote MCP servers?

Yes, the harness supports remote testing via the **HTTP** and **SSE** transports. Use the `-t http` or `-t sse` flags with the `-u` flag to specify the server URL. For authenticated endpoints, pass headers using the `-H` flag. The `create_connection()` factory in [`connections.py`](https://github.com/anthropics/skills/blob/main/connections.py) instantiates the appropriate `MCPConnection` subclass for your chosen transport.

### How do I change the Claude model used for evaluation?

Override the default model by passing the `-m` flag followed by the model identifier. For example, `-m claude-3-5-sonnet-20241022` selects a specific Sonnet snapshot. When using the Python API, pass the `model` parameter to `run_evaluation()`. This allows you to benchmark tool-use performance across different Claude model versions or compare snapshots.