# How to Create Evaluations for MCP Servers: A Complete Guide

> Learn how to create evaluations for MCP servers with this guide. Author XML test suites and run scripts to validate server behavior against LLM interactions.

- Repository: [Composio/awesome-claude-skills](https://github.com/composiohq/awesome-claude-skills)
- Tags: how-to-guide
- Published: 2026-07-24

---

**To create evaluations for MCP servers, author an XML test suite defining user prompts and expected tool invocation sequences, then execute [`scripts/evaluation.py`](https://github.com/ComposioHQ/awesome-claude-skills/blob/main/scripts/evaluation.py) from the ComposioHQ/awesome-claude-skills repository to validate server behavior against LLM interactions.**

MCP (Model‑Controlled Programming) servers expose tools that large language models invoke to answer complex, realistic questions. To ensure your server responds correctly to these tool-driven interactions, you must create evaluations for MCP servers that verify both the dispatch logic and final output accuracy. The evaluation framework in ComposioHQ/awesome-claude-skills provides a lightweight XML schema and Python harness for automating this validation.

## Understanding the Evaluation XML Structure

Evaluations are XML documents that wrap one or more test cases inside `<evaluation>` tags. Each `<testcase>` contains a simulated user prompt and the expected sequence of tool calls the LLM should generate.

### Root Elements and Test Cases

The root `<evaluation>` element contains multiple `<testcase>` nodes, each identified by a unique `id` attribute. Inside each test case, you define:

- **`<prompt>`** – The natural language request sent to the LLM
- **`<expected>`** – A block containing the mandatory tool calls and final answer

A minimal evaluation file looks like this:

```xml
<evaluation>
  <testcase id="basic-search">
    <prompt>What is the current temperature in Paris?</prompt>
    <expected>
      <tool name="weather_lookup" args="city=Paris"/>
      <final_answer>Paris is currently 18 °C.</final_answer>
    </expected>
  </testcase>
</evaluation>

```

### Defining Tool Call Expectations

The `<tool>` elements inside `<expected>` specify the exact name and arguments the LLM must use. You can chain multiple `<tool>` entries to represent multi-step workflows. For example, a complex query requiring search, extraction, and visualization would list three sequential tool elements before the `<final_answer>`.

## Running the Evaluation Harness

The Python harness at [`scripts/evaluation.py`](https://github.com/ComposioHQ/awesome-claude-skills/blob/main/scripts/evaluation.py) orchestrates the test execution. It parses your XML, initiates the specified transport connection to your MCP server, sends prompts to the configured LLM, and compares actual tool invocations against expectations.

### Supported Transport Modes

The harness supports three transport mechanisms for connecting to your server:

- **stdio** – The harness spawns the server process (e.g., `python -m my_mcp_server`) and communicates via stdin/stdout
- **sse** – Connects via Server-Sent Events to a running server
- **http** – Makes HTTP requests to a pre-running server endpoint

### Command Line Options

Execute evaluations using the following pattern:

```bash
python scripts/evaluation.py \
  -t stdio \
  -m gpt-4o \
  -c "python -m my_mcp_server" \
  simple_eval.xml

```

Key flags include:

- **`-t`** – Transport type (`stdio`, `sse`, or `http`)
- **`-m`** – LLM model identifier (e.g., `gpt-4o`, `claude-3-sonnet`)
- **`-c`** – Command or URL for server connection
- **`-o`** – Optional output path for the report (defaults to [`evaluation_report.md`](https://github.com/ComposioHQ/awesome-claude-skills/blob/main/evaluation_report.md))

For HTTP transport, start your server separately before running:

```bash
python scripts/evaluation.py \
  -t http \
  -m claude-3-sonnet \
  -c "http://localhost:8000" \
  --output detailed_report.md \
  complex_eval.xml

```

## Interpreting Evaluation Results

After execution, the harness generates a markdown report—[`evaluation_report.md`](https://github.com/ComposioHQ/awesome-claude-skills/blob/main/evaluation_report.md) by default—that lists each test case as **PASS** or **FAIL**. Failed entries include diff details showing which tool calls were missing, which arguments mismatched, or where the final answer deviated from expectations.

Use these reports to identify gaps in your server’s tool definitions, incorrect parameter handling, or prompt engineering issues that prevent the LLM from selecting the correct tools.

## Summary

- **Create evaluations** by writing XML files with `<testcase>` elements containing prompts and expected tool sequences
- **Validate behavior** using [`scripts/evaluation.py`](https://github.com/ComposioHQ/awesome-claude-skills/blob/main/scripts/evaluation.py) with `-t stdio`, `-t sse`, or `-t http` transport modes
- **Reference specifications** in [`mcp-builder/reference/evaluation.md`](https://github.com/ComposioHQ/awesome-claude-skills/blob/main/mcp-builder/reference/evaluation.md) for advanced schema details and best practices
- **Review results** in the generated [`evaluation_report.md`](https://github.com/ComposioHQ/awesome-claude-skills/blob/main/evaluation_report.md) to iterate on server logic and tool definitions

## Frequently Asked Questions

### What is MCP (Model‑Controlled Programming)?

MCP is a paradigm where servers expose discrete tools that large language models call to answer complex user queries. Each tool performs a specific function—such as data retrieval, calculation, or visualization—and the LLM orchestrates these calls to generate comprehensive responses.

### Which transport mode should I use for local development?

Use **stdio** transport during development. The harness starts your server process automatically, captures stdout/stderr for debugging, and terminates the process when evaluation completes. For integration testing against deployed services, use **http** or **sse** to connect to running instances.

### How specific must tool arguments be in the expected block?

Arguments in the `<tool>` element's `args` attribute must match exactly what the LLM generates, including formatting and key-value ordering. If your server expects `city=Paris` but the LLM sends `{"city": "Paris"}`, the evaluation will flag a mismatch until you align the expected format with the actual LLM output pattern.

### Where is the complete evaluation schema documented?

The full specification resides in [`mcp-builder/reference/evaluation.md`](https://github.com/ComposioHQ/awesome-claude-skills/blob/main/mcp-builder/reference/evaluation.md) within the ComposioHQ/awesome-claude-skills repository. This file documents advanced features, multiple test case organization, and troubleshooting tips for complex evaluation suites.