How to Create Evaluations for MCP Servers to Test LLM Effectiveness

The MCP evaluation harness is a CLI tool that drives a Claude model against XML-defined question-answer pairs to measure how effectively an LLM uses MCP server tools.

When you create evaluations for MCP servers, you validate whether your LLM can correctly orchestrate tool calls to produce accurate results. The anthropics/skills repository provides a complete harness in skills/mcp-builder/scripts/ that handles transport abstraction, agent loops, and automated scoring.

Understanding the MCP Evaluation Harness

The evaluation system treats your MCP server as a black box and the Claude model as the test subject. The harness lives in skills/mcp-builder/scripts/evaluation.py and implements an agent loop (agent_loop()) that mediates between the model and server.

The architecture follows this flow:

1. Parse the XML evaluation file containing <qa_pair> elements
2. Establish transport via create_connection() in connections.py
3. Discover available tools using MCPConnection.list_tools()
4. Execute the agent loop, forwarding tool_use blocks to MCPConnection.call_tool()
5. Extract the final answer from the <response> XML tag (enforced by EVALUATION_PROMPT)
6. Score against expected answers and render a markdown report

The harness supports three transport mechanisms through concrete subclasses of MCPConnection: stdio for local subprocesses, SSE for Server-Sent Events streams, and HTTP for REST endpoints.

Setting Up Your Evaluation XML File

Required XML Structure

Create an XML file that wraps each test case in a <qa_pair> element. The harness expects exactly two child elements per pair: <question> and <answer>.

<!-- eval.xml -->
<evaluation>
   <qa_pair>
      <question>What is the capital of France?</question>
      <answer>Paris</answer>
   </qa_pair>
   <qa_pair>
      <question>Compute 7 × 8.</question>
      <answer>56</answer>
   </qa_pair>
</evaluation>

Save this file to disk (e.g., my_eval.xml). The evaluation harness will parse this file, iterate over each qa_pair, and compare the model's output against your expected answers.

Running Evaluations Against MCP Servers

Local stdio Transport

For MCP servers implemented as local scripts or compiled binaries, use the stdio transport. The harness spawns the server as a subprocess and communicates over stdin/stdout.

python -m skills.mcp-builder.scripts.evaluation \
   -t stdio \
   -c python \
   -a my_server.py \
   my_eval.xml

• -t stdio selects the stdio transport
• -c python specifies the interpreter command
• -a my_server.py passes the server script as an argument

Remote HTTP Transport

To evaluate against a hosted MCP server exposing an HTTP API, use the HTTP transport. This requires the server URL and optional authentication headers.

python -m skills.mcp-builder.scripts.evaluation \
   -t http \
   -u https://my-mcp.example.com/v1 \
   -H "Authorization: Bearer $TOKEN" \
   my_eval.xml

• -u supplies the server base URL
• -H provides HTTP headers for authentication or custom configuration

Server-Sent Events (SSE) Transport

For servers streaming responses via SSE, use the SSE transport. This maintains a persistent connection suitable for long-running tool executions.

python -m skills.mcp-builder.scripts.evaluation \
   -t sse \
   -u https://sse-mcp.example.com \
   my_eval.xml

Customizing Model Selection and Parameters

By default, the harness uses a standard Claude model configuration, but you can override this with the -m flag to test different model versions or snapshots.

python -m skills.mcp-builder.scripts.evaluation \
   -t stdio \
   -c python \
   -a my_server.py \
   -m claude-3-5-sonnet-20241022 \
   my_eval.xml

This flexibility allows you to benchmark tool-use effectiveness across different Claude model generations or compare performance against specific model snapshots.

Programmatic Evaluation in Python

For integration into CI/CD pipelines or custom testing frameworks, import the evaluation harness directly in Python. The run_evaluation() function in skills/mcp-builder/scripts/evaluation.py handles the full lifecycle asynchronously.

import asyncio
from pathlib import Path
from skills.mcp-builder.scripts.evaluation import run_evaluation
from skills.mcp-builder.scripts.connections import create_connection

async def main():
    eval_path = Path("my_eval.xml")
    
    # Create stdio connection to local MCP server

    conn = create_connection(
        transport="stdio",
        command="python",
        args=["my_server.py"]
    )
    
    async with conn:
        report = await run_evaluation(
            eval_path, 
            conn, 
            model="claude-3-7-sonnet-20250219"
        )
        print(report)

asyncio.run(main())

This approach gives you full control over connection lifecycle, model parameters, and report handling while maintaining the same scoring logic as the CLI.

Summary

• Create evaluations for MCP servers by authoring XML files with <qa_pair> elements containing questions and expected answers.
• The evaluation harness in skills/mcp-builder/scripts/evaluation.py orchestrates Claude model interactions through an agent loop that handles tool_use blocks and extracts answers from <response> tags.
• Support for stdio, HTTP, and SSE transports via create_connection() in connections.py enables testing against local scripts or remote hosted servers.
• Use CLI flags (-t, -u, -m) or the Python run_evaluation() API to customize transport, authentication, and model selection.
• The harness automatically scores results and generates markdown reports comparing model outputs against expected answers.

Frequently Asked Questions

What is the MCP evaluation harness?

The MCP evaluation harness is a testing framework in the anthropics/skills repository that measures how effectively a Claude model uses tools exposed by an MCP server. It drives the model through a series of questions defined in an XML file, captures the model's tool invocations via MCPConnection.call_tool(), and scores the final answers against expected values.

How do I structure an evaluation XML file?

Evaluation XML files must contain a root <evaluation> element with one or more <qa_pair> children. Each pair requires exactly two child elements: <question> containing the prompt to send to the model, and <answer> containing the expected correct response. The model must wrap its final output in a <response> tag, which the harness extracts for scoring.

Can I test against remote MCP servers?

Yes, the harness supports remote testing via the HTTP and SSE transports. Use the -t http or -t sse flags with the -u flag to specify the server URL. For authenticated endpoints, pass headers using the -H flag. The create_connection() factory in connections.py instantiates the appropriate MCPConnection subclass for your chosen transport.

How do I change the Claude model used for evaluation?

Override the default model by passing the -m flag followed by the model identifier. For example, -m claude-3-5-sonnet-20241022 selects a specific Sonnet snapshot. When using the Python API, pass the model parameter to run_evaluation(). This allows you to benchmark tool-use performance across different Claude model versions or compare snapshots.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}