# How to Create Evaluations for MCP Servers: A Complete Testing Guide

> Learn how to create evaluations for MCP servers. Define XML schemas, run the evaluation harness, and refine tool signatures for better API testing.

- Repository: [Composio/awesome-codex-skills](https://github.com/composiohq/awesome-codex-skills)
- Tags: how-to-guide
- Published: 2026-04-26

---

**Creating evaluations for MCP servers requires defining an XML schema with stable question/answer pairs, executing the [`evaluation.py`](https://github.com/ComposioHQ/awesome-codex-skills/blob/main/evaluation.py) harness against your Model Context Protocol implementation, and iterating on the generated feedback to refine tool signatures.**

The ComposioHQ/awesome-codex-skills repository provides a comprehensive evaluation framework that validates MCP server functionality through automated LLM interaction testing. This guide explains how to create robust evaluations that verify your server's tools return accurate, deterministic results across different transport protocols.

## The Three-Phase Evaluation Workflow

Creating reliable evaluations for MCP servers follows a structured three-phase process defined in the [`mcp-builder/reference/evaluation.md`](https://github.com/ComposioHQ/awesome-codex-skills/blob/main/mcp-builder/reference/evaluation.md) documentation. Each phase builds upon the previous to ensure your server meets production quality standards.

### Phase 1: Define the Evaluation Schema

Start by authoring an XML file containing question/answer pairs. Each `<qa_pair>` element must include a human-readable question that tests read-only operations and a single, verifiable answer that remains constant over time. According to the evaluation guide in [`mcp-builder/reference/evaluation.md`](https://github.com/ComposioHQ/awesome-codex-skills/blob/main/mcp-builder/reference/evaluation.md), questions must be independent and stable, while answers must provide exactly one verifiable value that the LLM's `<response>` tag can match against.

### Phase 2: Run the Evaluation Harness

Execute the [`evaluation.py`](https://github.com/ComposioHQ/awesome-codex-skills/blob/main/evaluation.py) script located in [`mcp-builder/scripts/evaluation.py`](https://github.com/ComposioHQ/awesome-codex-skills/blob/main/mcp-builder/scripts/evaluation.py) to load your MCP server and evaluate its performance. The harness discovers available tools, initializes the LLM agent loop via `agent_loop`, and invokes `format_tools_for_openai` to convert MCP tool definitions for the language model. It supports three transport mechanisms—stdio, Server-Sent Events (SSE), and HTTP—managed through the `create_connection` factory function in [`mcp-builder/scripts/connections.py`](https://github.com/ComposioHQ/awesome-codex-skills/blob/main/mcp-builder/scripts/connections.py).

### Phase 3: Interpret the Report

The harness generates a markdown report containing per-task accuracy metrics, total execution duration, tool-call statistics, and structured feedback on tool design. Use this output to refine your server's tool signatures, descriptions, or pagination logic following the guidelines in [`mcp-builder/reference/mcp_best_practices.md`](https://github.com/ComposioHQ/awesome-codex-skills/blob/main/mcp-builder/reference/mcp_best_practices.md).

## Step-by-Step Implementation Guide

Follow these specific steps to implement evaluations for your MCP server using the Composio framework.

### Step 1: Author the XML Evaluation File

Create an evaluation file with at least ten `<qa_pair>` entries. Each pair must follow the schema documented in the evaluation reference:

```xml
<!-- my_evaluation.xml -->
<evaluation>
   <qa_pair>
      <question>Find the repository with the most stars that was created before 2023. What is the repository name?</question>
      <answer>data-pipeline</answer>
   </qa_pair>
   <qa_pair>
      <question>Search for issues labeled "bug" that were closed in March 2024. Which user closed the most issues? Provide their username.</question>
      <answer>sarah_dev</answer>
   </qa_pair>
</evaluation>

```

Questions must test read-only operations only and produce deterministic results. Answers should contain a single value without ambiguity to enable exact string matching against the LLM response.

### Step 2: Configure Your Transport

Select the appropriate transport protocol for your MCP server deployment. The `create_connection` function in [`mcp-builder/scripts/connections.py`](https://github.com/ComposioHQ/awesome-codex-skills/blob/main/mcp-builder/scripts/connections.py) abstracts connection management for stdio, SSE, and HTTP transports. For local development, use stdio with a Python subprocess. For production servers, configure SSE or HTTP endpoints with appropriate authentication headers.

### Step 3: Execute the Evaluation Harness

Run the evaluation using the `main()` function entry point in [`evaluation.py`](https://github.com/ComposioHQ/awesome-codex-skills/blob/main/evaluation.py). The `run_evaluation` function orchestrates the test execution and comparison logic.

For a local stdio server:

```bash
python mcp-builder/scripts/evaluation.py \
   -t stdio \
   -c python \
   -a my_server.py \
   my_evaluation.xml \
   -o evaluation_report.md

```

For a remote SSE endpoint:

```bash
python mcp-builder/scripts/evaluation.py \
   -t sse \
   -u https://example.com/mcp \
   -H "Authorization: Bearer <token>" \
   my_evaluation.xml

```

The `-t` parameter accepts `stdio`, `sse`, or `http`. The `-o` parameter optionally writes the report to a file instead of stdout.

### Step 4: Analyze the Generated Report

Review the markdown output for accuracy percentages, tool utilization patterns, and specific failure modes. The report identifies whether the LLM successfully invoked the correct tools and whether the responses matched expected values exactly.

## Evaluation XML Schema Requirements

The evaluation framework enforces strict requirements for question and answer design to ensure reproducible results.

- **Read-only operations**: Questions must not modify server state
- **Stable answers**: Expected values must remain constant across evaluation runs
- **Single values**: Each `<answer>` tag must contain exactly one verifiable string
- **Independent questions**: Each QA pair should stand alone without dependencies on previous results

## Understanding Transport Configuration

The evaluation harness supports multiple connection types through the abstract `MCPConnection` classes. When you specify `-t stdio`, the harness spawns a subprocess and communicates over standard input/output. For `-t sse` or `-t http`, the harness establishes persistent connections to remote endpoints, passing headers via the `-H` flag for authentication.

## Summary

- Create an XML evaluation file with at least 10 QA pairs following the schema in [`mcp-builder/reference/evaluation.md`](https://github.com/ComposioHQ/awesome-codex-skills/blob/main/mcp-builder/reference/evaluation.md)
- Execute [`evaluation.py`](https://github.com/ComposioHQ/awesome-codex-skills/blob/main/evaluation.py) from `mcp-builder/scripts/` with your chosen transport type (stdio, SSE, or HTTP)
- Use the `create_connection` abstraction in [`connections.py`](https://github.com/ComposioHQ/awesome-codex-skills/blob/main/connections.py) to handle protocol-specific implementation details
- Review the generated markdown report for per-task accuracy and tool-call statistics
- Iterate on tool signatures and descriptions using feedback from the evaluation report and guidelines in [`mcp-builder/reference/mcp_best_practices.md`](https://github.com/ComposioHQ/awesome-codex-skills/blob/main/mcp-builder/reference/mcp_best_practices.md)

## Frequently Asked Questions

### How many QA pairs are required for a valid MCP server evaluation?

The framework requires a minimum of ten `<qa_pair>` entries in your XML evaluation file to generate statistically meaningful accuracy metrics. Each pair must contain one question and one answer tag, with questions testing read-only operations that produce deterministic, stable results.

### Can I evaluate remote MCP servers using this framework?

Yes. The evaluation harness supports three transport types: stdio for local subprocesses, Server-Sent Events (SSE), and HTTP for remote endpoints. When evaluating remote servers, use the `-t sse` or `-t http` flags combined with the `-u` URL parameter and `-H` headers for authentication. The `create_connection` function in [`connections.py`](https://github.com/ComposioHQ/awesome-codex-skills/blob/main/connections.py) handles the underlying protocol implementation.

### What makes a good evaluation question for MCP servers?

Effective questions test read-only operations that return stable, verifiable data. Avoid questions that depend on mutable state or time-sensitive information. Each question should require exactly one specific tool call or sequence to answer, and the corresponding answer must be a single value that can be string-matched against the LLM's `<response>` tag output.

### Where does the evaluation harness output its results?

By default, the harness prints the markdown report to stdout. Use the `-o` flag followed by a filename to write the report directly to disk. The report includes accuracy percentages, execution duration, tool-call statistics, reasoning summaries, and specific feedback on tool design issues that require remediation.