How to Create Evaluations for MCP Servers: A Complete Guide

To create evaluations for MCP servers, author an XML test suite defining user prompts and expected tool invocation sequences, then execute scripts/evaluation.py from the ComposioHQ/awesome-claude-skills repository to validate server behavior against LLM interactions.

MCP (Model‑Controlled Programming) servers expose tools that large language models invoke to answer complex, realistic questions. To ensure your server responds correctly to these tool-driven interactions, you must create evaluations for MCP servers that verify both the dispatch logic and final output accuracy. The evaluation framework in ComposioHQ/awesome-claude-skills provides a lightweight XML schema and Python harness for automating this validation.

Understanding the Evaluation XML Structure

Evaluations are XML documents that wrap one or more test cases inside <evaluation> tags. Each <testcase> contains a simulated user prompt and the expected sequence of tool calls the LLM should generate.

Root Elements and Test Cases

The root <evaluation> element contains multiple <testcase> nodes, each identified by a unique id attribute. Inside each test case, you define:

• <prompt> – The natural language request sent to the LLM
• <expected> – A block containing the mandatory tool calls and final answer

A minimal evaluation file looks like this:

<evaluation>
  <testcase id="basic-search">
    <prompt>What is the current temperature in Paris?</prompt>
    <expected>
      <tool name="weather_lookup" args="city=Paris"/>
      <final_answer>Paris is currently 18 °C.</final_answer>
    </expected>
  </testcase>
</evaluation>

Defining Tool Call Expectations

The <tool> elements inside <expected> specify the exact name and arguments the LLM must use. You can chain multiple <tool> entries to represent multi-step workflows. For example, a complex query requiring search, extraction, and visualization would list three sequential tool elements before the <final_answer>.

Running the Evaluation Harness

The Python harness at scripts/evaluation.py orchestrates the test execution. It parses your XML, initiates the specified transport connection to your MCP server, sends prompts to the configured LLM, and compares actual tool invocations against expectations.

Supported Transport Modes

The harness supports three transport mechanisms for connecting to your server:

• stdio – The harness spawns the server process (e.g., python -m my_mcp_server) and communicates via stdin/stdout
• sse – Connects via Server-Sent Events to a running server
• http – Makes HTTP requests to a pre-running server endpoint

Command Line Options

Execute evaluations using the following pattern:

python scripts/evaluation.py \
  -t stdio \
  -m gpt-4o \
  -c "python -m my_mcp_server" \
  simple_eval.xml

Key flags include:

• -t – Transport type (stdio, sse, or http)
• -m – LLM model identifier (e.g., gpt-4o, claude-3-sonnet)
• -c – Command or URL for server connection
• -o – Optional output path for the report (defaults to evaluation_report.md)

For HTTP transport, start your server separately before running:

python scripts/evaluation.py \
  -t http \
  -m claude-3-sonnet \
  -c "http://localhost:8000" \
  --output detailed_report.md \
  complex_eval.xml

Interpreting Evaluation Results

After execution, the harness generates a markdown report—evaluation_report.md by default—that lists each test case as PASS or FAIL. Failed entries include diff details showing which tool calls were missing, which arguments mismatched, or where the final answer deviated from expectations.

Use these reports to identify gaps in your server’s tool definitions, incorrect parameter handling, or prompt engineering issues that prevent the LLM from selecting the correct tools.

Summary

• Create evaluations by writing XML files with <testcase> elements containing prompts and expected tool sequences
• Validate behavior using scripts/evaluation.py with -t stdio, -t sse, or -t http transport modes
• Reference specifications in mcp-builder/reference/evaluation.md for advanced schema details and best practices
• Review results in the generated evaluation_report.md to iterate on server logic and tool definitions

Frequently Asked Questions

What is MCP (Model‑Controlled Programming)?

MCP is a paradigm where servers expose discrete tools that large language models call to answer complex user queries. Each tool performs a specific function—such as data retrieval, calculation, or visualization—and the LLM orchestrates these calls to generate comprehensive responses.

Which transport mode should I use for local development?

Use stdio transport during development. The harness starts your server process automatically, captures stdout/stderr for debugging, and terminates the process when evaluation completes. For integration testing against deployed services, use http or sse to connect to running instances.

How specific must tool arguments be in the expected block?

Arguments in the <tool> element's args attribute must match exactly what the LLM generates, including formatting and key-value ordering. If your server expects city=Paris but the LLM sends {"city": "Paris"}, the evaluation will flag a mismatch until you align the expected format with the actual LLM output pattern.

Where is the complete evaluation schema documented?

The full specification resides in mcp-builder/reference/evaluation.md within the ComposioHQ/awesome-claude-skills repository. This file documents advanced features, multiple test case organization, and troubleshooting tips for complex evaluation suites.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}