How to Create Evaluations for MCP Servers: A Complete Testing Guide

Creating evaluations for MCP servers requires defining an XML schema with stable question/answer pairs, executing the evaluation.py harness against your Model Context Protocol implementation, and iterating on the generated feedback to refine tool signatures.

The ComposioHQ/awesome-codex-skills repository provides a comprehensive evaluation framework that validates MCP server functionality through automated LLM interaction testing. This guide explains how to create robust evaluations that verify your server's tools return accurate, deterministic results across different transport protocols.

The Three-Phase Evaluation Workflow

Creating reliable evaluations for MCP servers follows a structured three-phase process defined in the mcp-builder/reference/evaluation.md documentation. Each phase builds upon the previous to ensure your server meets production quality standards.

Phase 1: Define the Evaluation Schema

Start by authoring an XML file containing question/answer pairs. Each <qa_pair> element must include a human-readable question that tests read-only operations and a single, verifiable answer that remains constant over time. According to the evaluation guide in mcp-builder/reference/evaluation.md, questions must be independent and stable, while answers must provide exactly one verifiable value that the LLM's <response> tag can match against.

Phase 2: Run the Evaluation Harness

Execute the evaluation.py script located in mcp-builder/scripts/evaluation.py to load your MCP server and evaluate its performance. The harness discovers available tools, initializes the LLM agent loop via agent_loop, and invokes format_tools_for_openai to convert MCP tool definitions for the language model. It supports three transport mechanisms—stdio, Server-Sent Events (SSE), and HTTP—managed through the create_connection factory function in mcp-builder/scripts/connections.py.

Phase 3: Interpret the Report

The harness generates a markdown report containing per-task accuracy metrics, total execution duration, tool-call statistics, and structured feedback on tool design. Use this output to refine your server's tool signatures, descriptions, or pagination logic following the guidelines in mcp-builder/reference/mcp_best_practices.md.

Step-by-Step Implementation Guide

Follow these specific steps to implement evaluations for your MCP server using the Composio framework.

Step 1: Author the XML Evaluation File

Create an evaluation file with at least ten <qa_pair> entries. Each pair must follow the schema documented in the evaluation reference:

<!-- my_evaluation.xml -->
<evaluation>
   <qa_pair>
      <question>Find the repository with the most stars that was created before 2023. What is the repository name?</question>
      <answer>data-pipeline</answer>
   </qa_pair>
   <qa_pair>
      <question>Search for issues labeled "bug" that were closed in March 2024. Which user closed the most issues? Provide their username.</question>
      <answer>sarah_dev</answer>
   </qa_pair>
</evaluation>

Questions must test read-only operations only and produce deterministic results. Answers should contain a single value without ambiguity to enable exact string matching against the LLM response.

Step 2: Configure Your Transport

Select the appropriate transport protocol for your MCP server deployment. The create_connection function in mcp-builder/scripts/connections.py abstracts connection management for stdio, SSE, and HTTP transports. For local development, use stdio with a Python subprocess. For production servers, configure SSE or HTTP endpoints with appropriate authentication headers.

Step 3: Execute the Evaluation Harness

Run the evaluation using the main() function entry point in evaluation.py. The run_evaluation function orchestrates the test execution and comparison logic.

For a local stdio server:

python mcp-builder/scripts/evaluation.py \
   -t stdio \
   -c python \
   -a my_server.py \
   my_evaluation.xml \
   -o evaluation_report.md

For a remote SSE endpoint:

python mcp-builder/scripts/evaluation.py \
   -t sse \
   -u https://example.com/mcp \
   -H "Authorization: Bearer <token>" \
   my_evaluation.xml

The -t parameter accepts stdio, sse, or http. The -o parameter optionally writes the report to a file instead of stdout.

Step 4: Analyze the Generated Report

Review the markdown output for accuracy percentages, tool utilization patterns, and specific failure modes. The report identifies whether the LLM successfully invoked the correct tools and whether the responses matched expected values exactly.

Evaluation XML Schema Requirements

The evaluation framework enforces strict requirements for question and answer design to ensure reproducible results.

• Read-only operations: Questions must not modify server state
• Stable answers: Expected values must remain constant across evaluation runs
• Single values: Each <answer> tag must contain exactly one verifiable string
• Independent questions: Each QA pair should stand alone without dependencies on previous results

Understanding Transport Configuration

The evaluation harness supports multiple connection types through the abstract MCPConnection classes. When you specify -t stdio, the harness spawns a subprocess and communicates over standard input/output. For -t sse or -t http, the harness establishes persistent connections to remote endpoints, passing headers via the -H flag for authentication.

Summary

• Create an XML evaluation file with at least 10 QA pairs following the schema in mcp-builder/reference/evaluation.md
• Execute evaluation.py from mcp-builder/scripts/ with your chosen transport type (stdio, SSE, or HTTP)
• Use the create_connection abstraction in connections.py to handle protocol-specific implementation details
• Review the generated markdown report for per-task accuracy and tool-call statistics
• Iterate on tool signatures and descriptions using feedback from the evaluation report and guidelines in mcp-builder/reference/mcp_best_practices.md

Frequently Asked Questions

How many QA pairs are required for a valid MCP server evaluation?

The framework requires a minimum of ten <qa_pair> entries in your XML evaluation file to generate statistically meaningful accuracy metrics. Each pair must contain one question and one answer tag, with questions testing read-only operations that produce deterministic, stable results.

Can I evaluate remote MCP servers using this framework?

Yes. The evaluation harness supports three transport types: stdio for local subprocesses, Server-Sent Events (SSE), and HTTP for remote endpoints. When evaluating remote servers, use the -t sse or -t http flags combined with the -u URL parameter and -H headers for authentication. The create_connection function in connections.py handles the underlying protocol implementation.

What makes a good evaluation question for MCP servers?

Effective questions test read-only operations that return stable, verifiable data. Avoid questions that depend on mutable state or time-sensitive information. Each question should require exactly one specific tool call or sequence to answer, and the corresponding answer must be a single value that can be string-matched against the LLM's <response> tag output.

Where does the evaluation harness output its results?

By default, the harness prints the markdown report to stdout. Use the -o flag followed by a filename to write the report directly to disk. The report includes accuracy percentages, execution duration, tool-call statistics, reasoning summaries, and specific feedback on tool design issues that require remediation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}