# How to Create and Register Custom Evaluation Metrics for agents-cli

> Learn how to create and register custom evaluation metrics for agents cli. Define them in eval_config.yaml and run them with flags for seamless integration.

- Repository: [Google/agents-cli](https://github.com/google/agents-cli)
- Tags: how-to-guide
- Published: 2026-07-01

---

**You create and register custom evaluation metrics for agents-cli by defining them in the `custom_metrics` section of an [`eval_config.yaml`](https://github.com/google/agents-cli/blob/main/eval_config.yaml) file and selecting them via `metrics_to_run` or the `--metrics` CLI flag, which the CLI automatically processes through the `prepare_eval_metrics` function in [`eval_utils.py`](https://github.com/google/agents-cli/blob/main/eval_utils.py).**

The `google/agents-cli` repository provides a declarative framework for evaluating AI agents without requiring additional Python registration code. By leveraging the Agent Platform Evaluation SDK, you can **create and register custom evaluation metrics for agents-cli** through YAML configuration files that the CLI resolves, compiles, and executes. This guide explains the architecture, implementation patterns, and specific source code mechanisms that enable custom metric registration.

## Understanding the Custom Metrics Architecture

The agents-cli supports two distinct metric schemas defined in the Agent Platform Evaluation SDK, both parsed and validated in [`src/google/agents/cli/eval/eval_utils.py`](https://github.com/google/agents-cli/blob/main/src/google/agents/cli/eval/eval_utils.py).

**CodeExecutionMetric** executes Python functions you provide, supporting both local execution within the CLI process and remote execution in a Vertex AI sandbox.

**LLMMetric** utilizes an LLM judge driven by a prompt template, always executing remotely through Vertex AI.

Three critical functions in [`eval_utils.py`](https://github.com/google/agents-cli/blob/main/eval_utils.py) handle the registration lifecycle:

- **`_resolve_custom_function_file`** – Resolves a `custom_function_file` path relative to the configuration file and inlines its contents.
- **`_compile_custom_function`** – Compiles inline Python source into a callable named `evaluate`.
- **`prepare_eval_metrics`** – Loads the configuration, merges built-in and custom metrics, and returns metric objects for the Vertex AI evaluation service.

## Registration Flow

The registration process is entirely declarative and requires no additional Python code beyond the metric implementation itself:

1. **Create an evaluation configuration file** (typically [`tests/eval/eval_config.yaml`](https://github.com/google/agents-cli/blob/main/tests/eval/eval_config.yaml) in scaffolded projects).

2. **Define your metric** in the `custom_metrics` list using either the `CodeExecutionMetric` or `LLMMetric` schema.

3. **Select the metric** for execution by adding its name to `metrics_to_run` or passing it via `agents-cli eval run --metrics <name>`.

4. **Execute the evaluation** using `agents-cli eval grade` or `agents-cli eval run`, which invokes `prepare_eval_metrics` to read the file, inline referenced Python files, compile the code, and pass objects to Vertex AI.

## Choosing Between Local Code and LLM Judges

Select the appropriate metric type based on your evaluation requirements:

**CodeExecutionMetric** (`custom_function` or `custom_function_file`) – Use for deterministic, code-driven scoring such as counting conversation turns or computing custom statistical scores. Specify `execution: remote` to run in a Vertex AI sandbox, or omit for local execution (default).

**LLMMetric** (`prompt_template`) – Use when you need an LLM to judge qualitative aspects like helpfulness, style, or safety using flexible prompt templates. This type always executes remotely via Vertex AI.

## Implementation Examples

### Inline Python Function (Local Execution)

Define a metric directly in the YAML file using the `custom_function` key:

```yaml

# tests/eval/eval_config.yaml

metrics_to_run:
  - turn_count
custom_metrics:
  - name: turn_count
    custom_function: |
      def evaluate(instance):
          # instance contains {prompt}, {response}, {agent_data}, etc.

          turns = (instance.get("agent_data") or {}).get("turns", [])
          return {"score": len(turns)}

```

When you run:

```bash
agents-cli eval grade --config tests/eval/eval_config.yaml

```

The CLI inlines the function string and compiles it using `_compile_custom_function`, then executes it locally for each evaluation case.

### External Python File Reference

For cleaner code management, reference a separate Python file:

```yaml

# tests/eval/eval_config.yaml

metrics_to_run:
  - turn_count
custom_metrics:
  - name: turn_count
    custom_function_file: metrics.py

```

```python

# tests/eval/metrics.py

def evaluate(instance):
    turns = (instance.get("agent_data") or {}).get("turns", [])
    return {"score": len(turns)}

```

The function `_resolve_custom_function_file` resolves the path relative to the configuration file and inlines the source before compilation.

### Remote Execution in Vertex AI

To run custom code in a managed sandbox, specify `execution: remote`:

```yaml

# tests/eval/eval_config.yaml

metrics_to_run:
  - tool_call_count
custom_metrics:
  - name: tool_call_count
    execution: remote
    custom_function: |
      def evaluate(instance):
          n = 0
          for turn in (instance.get("agent_data") or {}).get("turns", []):
              for event in turn.get("events", []):
                  for part in (event.get("content") or {}).get("parts", []):
                      if "function_call" in part:
                          n += 1
          return {"score": n}

```

The CLI packages this as a `CodeExecutionMetric` protobuf and sends it to the Vertex AI evaluation service for execution.

### LLM-as-a-Judge Configuration

Configure an LLM judge using a prompt template:

```yaml

# tests/eval/eval_config.yaml

metrics_to_run:
  - helpfulness
custom_metrics:
  - name: helpfulness
    prompt_template: |
      Rate the agent's response on a 1-5 scale for helpfulness.
      Prompt: {prompt}
      Response: {response}
      Return JSON: {"score": <1|2|3|4|5>, "explanation": "<reason>"}
    judge_model: gemini-1.5-flash-001
    judge_model_sampling_count: 3

```

The `prepare_eval_metrics` function constructs an `LLMMetric` object from this configuration and submits it to Vertex AI.

## Key Source Files and Functions

Understanding these implementation files helps debug and extend custom metrics:

- **[`src/google/agents/cli/eval/eval_utils.py`](https://github.com/google/agents-cli/blob/main/src/google/agents/cli/eval/eval_utils.py)** – Contains `_resolve_custom_function_file`, `_compile_custom_function`, and `prepare_eval_metrics`, which handle config loading, code inlining, compilation, and metric object construction.

- **[`skills/google-agents-cli-eval/references/metrics-guide.md`](https://github.com/google/agents-cli/blob/main/skills/google-agents-cli-eval/references/metrics-guide.md)** – Reference documentation describing built-in metrics and custom metric schemas.

- **[`skills/google-agents-cli-eval/SKILL.md`](https://github.com/google/agents-cli/blob/main/skills/google-agents-cli-eval/SKILL.md)** – Overview of the Evaluation Configuration Schema.

- **[`tests/eval/eval_config.yaml`](https://github.com/google/agents-cli/blob/main/tests/eval/eval_config.yaml)** – Typical location in scaffolded projects for custom metric definitions.

## Summary

- **Declarative registration**: Add custom metrics to `custom_metrics` in [`eval_config.yaml`](https://github.com/google/agents-cli/blob/main/eval_config.yaml) and select them via `metrics_to_run` or `--metrics` flags.
- **Two metric types**: Use **CodeExecutionMetric** for Python-based scoring (local or remote) and **LLMMetric** for LLM judges (always remote).
- **Core implementation**: The `prepare_eval_metrics` function in [`eval_utils.py`](https://github.com/google/agents-cli/blob/main/eval_utils.py) orchestrates loading, validation, and execution preparation.
- **Flexible coding**: Embed Python directly via `custom_function` or reference external files with `custom_function_file`, resolved by `_resolve_custom_function_file`.
- **Remote execution**: Set `execution: remote` to run code in Vertex AI sandboxes instead of the local CLI process.

## Frequently Asked Questions

### What is the difference between custom_function and custom_function_file?

The `custom_function` key accepts an inline Python string containing an `evaluate` function, while `custom_function_file` specifies a relative path to a `.py` file containing the same function. According to the source code in [`eval_utils.py`](https://github.com/google/agents-cli/blob/main/eval_utils.py), both are ultimately resolved into inline source code by `_resolve_custom_function_file` before `_compile_custom_function` transforms them into executable callables.

### Can I run custom metrics locally without Vertex AI?

Yes. **CodeExecutionMetric** types run locally by default within the CLI Python process. Only when you specify `execution: remote` does the CLI package the code as a `CodeExecutionMetric` protobuf for Vertex AI sandbox execution. **LLMMetric** types always require remote execution through Vertex AI to access the LLM judge models.

### How does the CLI validate custom metric definitions?

The `prepare_eval_metrics` function loads the configuration and validates entries against the Agent Platform Evaluation SDK schemas. For code-based metrics, `_compile_custom_function` attempts to compile the Python source immediately, surfacing syntax errors before evaluation begins. File paths specified in `custom_function_file` are validated for existence and resolved relative to the configuration file location.

### What data is available in the instance parameter?

The `instance` dictionary passed to your `evaluate` function contains the evaluation case data, including `prompt`, `response`, and `agent_data`. The `agent_data` field typically includes conversation metadata such as `turns`, `events`, and `content` parts, allowing you to implement fine-grained scoring logic based on the full interaction history between the user and agent.