How to Create and Register Custom Evaluation Metrics for agents-cli

You create and register custom evaluation metrics for agents-cli by defining them in the custom_metrics section of an eval_config.yaml file and selecting them via metrics_to_run or the --metrics CLI flag, which the CLI automatically processes through the prepare_eval_metrics function in eval_utils.py.

The google/agents-cli repository provides a declarative framework for evaluating AI agents without requiring additional Python registration code. By leveraging the Agent Platform Evaluation SDK, you can create and register custom evaluation metrics for agents-cli through YAML configuration files that the CLI resolves, compiles, and executes. This guide explains the architecture, implementation patterns, and specific source code mechanisms that enable custom metric registration.

Understanding the Custom Metrics Architecture

The agents-cli supports two distinct metric schemas defined in the Agent Platform Evaluation SDK, both parsed and validated in src/google/agents/cli/eval/eval_utils.py.

CodeExecutionMetric executes Python functions you provide, supporting both local execution within the CLI process and remote execution in a Vertex AI sandbox.

LLMMetric utilizes an LLM judge driven by a prompt template, always executing remotely through Vertex AI.

Three critical functions in eval_utils.py handle the registration lifecycle:

• _resolve_custom_function_file – Resolves a custom_function_file path relative to the configuration file and inlines its contents.
• _compile_custom_function – Compiles inline Python source into a callable named evaluate.
• prepare_eval_metrics – Loads the configuration, merges built-in and custom metrics, and returns metric objects for the Vertex AI evaluation service.

Registration Flow

The registration process is entirely declarative and requires no additional Python code beyond the metric implementation itself:

1. Create an evaluation configuration file (typically tests/eval/eval_config.yaml in scaffolded projects).

2. Define your metric in the custom_metrics list using either the CodeExecutionMetric or LLMMetric schema.

3. Select the metric for execution by adding its name to metrics_to_run or passing it via agents-cli eval run --metrics <name>.

4. Execute the evaluation using agents-cli eval grade or agents-cli eval run, which invokes prepare_eval_metrics to read the file, inline referenced Python files, compile the code, and pass objects to Vertex AI.

Choosing Between Local Code and LLM Judges

Select the appropriate metric type based on your evaluation requirements:

CodeExecutionMetric (custom_function or custom_function_file) – Use for deterministic, code-driven scoring such as counting conversation turns or computing custom statistical scores. Specify execution: remote to run in a Vertex AI sandbox, or omit for local execution (default).

LLMMetric (prompt_template) – Use when you need an LLM to judge qualitative aspects like helpfulness, style, or safety using flexible prompt templates. This type always executes remotely via Vertex AI.

Implementation Examples

Inline Python Function (Local Execution)

Define a metric directly in the YAML file using the custom_function key:

# tests/eval/eval_config.yaml

metrics_to_run:
  - turn_count
custom_metrics:
  - name: turn_count
    custom_function: |
      def evaluate(instance):
          # instance contains {prompt}, {response}, {agent_data}, etc.

          turns = (instance.get("agent_data") or {}).get("turns", [])
          return {"score": len(turns)}

When you run:

agents-cli eval grade --config tests/eval/eval_config.yaml

The CLI inlines the function string and compiles it using _compile_custom_function, then executes it locally for each evaluation case.

External Python File Reference

For cleaner code management, reference a separate Python file:

# tests/eval/eval_config.yaml

metrics_to_run:
  - turn_count
custom_metrics:
  - name: turn_count
    custom_function_file: metrics.py

# tests/eval/metrics.py

def evaluate(instance):
    turns = (instance.get("agent_data") or {}).get("turns", [])
    return {"score": len(turns)}

The function _resolve_custom_function_file resolves the path relative to the configuration file and inlines the source before compilation.

Remote Execution in Vertex AI

To run custom code in a managed sandbox, specify execution: remote:

# tests/eval/eval_config.yaml

metrics_to_run:
  - tool_call_count
custom_metrics:
  - name: tool_call_count
    execution: remote
    custom_function: |
      def evaluate(instance):
          n = 0
          for turn in (instance.get("agent_data") or {}).get("turns", []):
              for event in turn.get("events", []):
                  for part in (event.get("content") or {}).get("parts", []):
                      if "function_call" in part:
                          n += 1
          return {"score": n}

The CLI packages this as a CodeExecutionMetric protobuf and sends it to the Vertex AI evaluation service for execution.

LLM-as-a-Judge Configuration

Configure an LLM judge using a prompt template:

# tests/eval/eval_config.yaml

metrics_to_run:
  - helpfulness
custom_metrics:
  - name: helpfulness
    prompt_template: |
      Rate the agent's response on a 1-5 scale for helpfulness.
      Prompt: {prompt}
      Response: {response}
      Return JSON: {"score": <1|2|3|4|5>, "explanation": "<reason>"}
    judge_model: gemini-1.5-flash-001
    judge_model_sampling_count: 3

The prepare_eval_metrics function constructs an LLMMetric object from this configuration and submits it to Vertex AI.

Key Source Files and Functions

Understanding these implementation files helps debug and extend custom metrics:

• src/google/agents/cli/eval/eval_utils.py – Contains _resolve_custom_function_file, _compile_custom_function, and prepare_eval_metrics, which handle config loading, code inlining, compilation, and metric object construction.

• skills/google-agents-cli-eval/references/metrics-guide.md – Reference documentation describing built-in metrics and custom metric schemas.

• skills/google-agents-cli-eval/SKILL.md – Overview of the Evaluation Configuration Schema.

• tests/eval/eval_config.yaml – Typical location in scaffolded projects for custom metric definitions.

Summary

• Declarative registration: Add custom metrics to custom_metrics in eval_config.yaml and select them via metrics_to_run or --metrics flags.
• Two metric types: Use CodeExecutionMetric for Python-based scoring (local or remote) and LLMMetric for LLM judges (always remote).
• Core implementation: The prepare_eval_metrics function in eval_utils.py orchestrates loading, validation, and execution preparation.
• Flexible coding: Embed Python directly via custom_function or reference external files with custom_function_file, resolved by _resolve_custom_function_file.
• Remote execution: Set execution: remote to run code in Vertex AI sandboxes instead of the local CLI process.

Frequently Asked Questions

What is the difference between custom_function and custom_function_file?

The custom_function key accepts an inline Python string containing an evaluate function, while custom_function_file specifies a relative path to a .py file containing the same function. According to the source code in eval_utils.py, both are ultimately resolved into inline source code by _resolve_custom_function_file before _compile_custom_function transforms them into executable callables.

Can I run custom metrics locally without Vertex AI?

Yes. CodeExecutionMetric types run locally by default within the CLI Python process. Only when you specify execution: remote does the CLI package the code as a CodeExecutionMetric protobuf for Vertex AI sandbox execution. LLMMetric types always require remote execution through Vertex AI to access the LLM judge models.

How does the CLI validate custom metric definitions?

The prepare_eval_metrics function loads the configuration and validates entries against the Agent Platform Evaluation SDK schemas. For code-based metrics, _compile_custom_function attempts to compile the Python source immediately, surfacing syntax errors before evaluation begins. File paths specified in custom_function_file are validated for existence and resolved relative to the configuration file location.

What data is available in the instance parameter?

The instance dictionary passed to your evaluate function contains the evaluation case data, including prompt, response, and agent_data. The agent_data field typically includes conversation metadata such as turns, events, and content parts, allowing you to implement fine-grained scoring logic based on the full interaction history between the user and agent.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}