How to Optimize Agent Prompts Using the GEPA Framework with agents-cli eval optimize

The agents-cli eval optimize command runs the Google Agents Development Kit (ADK) GEPA optimizer to iteratively rewrite your agent's system instruction against a target evaluation metric, outputting an optimized prompt you can paste back into your manifest.yaml.

The google/agents-cli repository provides an experimental command-line interface for automating prompt engineering through evolutionary algorithms. Using the GEPA (Guided Evolutionary Prompt-Optimization) framework integrated with the ADK, developers can systematically improve agent prompts without manual A/B testing. This guide explains how to use the optimization pipeline to maximize evaluation metrics while maintaining full control over the evolutionary process.

What is GEPA Prompt Optimization?

GEPA is an evolutionary algorithm that treats prompt optimization as a search problem. When you run agents-cli eval optimize, the tool iteratively generates variations of your agent's system instruction (the top-level prompt in manifest.yaml), tests them against a dataset, and selects high-performing candidates based on your chosen metric.

The command is currently experimental and warns users that behavior may change. It relies on ADK's native optimizer, which is invoked via uv run adk optimize after the CLI prepares the necessary configuration and datasets.

Prerequisites and Default Configuration

Before running optimization, ensure your environment includes the evaluation dependencies. The command automatically syncs these via uv sync --dev --extra eval before execution.

By default, the optimizer looks for configuration at tests/eval/optimization_config.json. This JSON file defines your train_dataset, validation_dataset, and target_metric. You can override these defaults using CLI flags or specify a custom config file path with --config.

The Three-Stage Optimization Pipeline

The optimization process implemented in src/google/agents/cli/eval/cmd_optimize.py follows a strict three-stage pipeline:

Stage 1: Configuration Loading and Validation

The _load_configs_and_datasets function (lines 45-102 in cmd_optimize.py) merges configuration sources in priority order:

1. CLI flags (--dataset, --target-metric)
2. User-supplied JSON config file
3. Default values from tests/eval/optimization_config.json

This merge ensures that explicit command-line arguments always take precedence over file-based configuration.

Stage 2: Dataset Preparation and Conversion

The optimizer requires ADK-compatible EvalSet format, but users typically provide EvaluationDataset JSON files. The utility functions in src/google/agents/cli/eval/optimize_utils.py (lines 26-236) handle this conversion, including:

• Mapping conversations to ADK's expected schema
• Preserving rubrics and reference responses
• Generating unique identifiers for each eval-set

Each converted eval-set is written to a hidden .tmp subdirectory within your agent folder (.tmp/<id>.evalset.json). The relative path to this temporary file is passed to the ADK optimizer.

Stage 3: Optimization Execution

The _execute_optimization_run function (lines 21-68 in cmd_optimize.py) materializes temporary sampler and optimizer configurations, then invokes the ADK optimizer via subprocess:

uv run adk optimize ... --evalset_path .tmp/<id>.evalset.json

During execution, the optimizer rewrites the prompt to maximize your target metric (e.g., relevance, faithfulness). If you set print_detailed_results: true in your config, the CLI surfaces per-iteration performance data in real-time.

Running Your First Optimization

Start with the default configuration to test the pipeline:

agents-cli eval optimize

This command uses tests/eval/optimization_config.json for datasets and metric selection.

Override specific parameters without modifying your config file:

agents-cli eval optimize \
  --dataset path/to/my_dataset.json \
  --target-metric relevance

The --dataset flag overrides any dataset definition in the config, while --target-metric forces the optimizer to focus on the specified evaluation criteria.

Advanced Configuration with JSON

For fine-grained control over the evolutionary process, create a custom configuration file:

{
  "eval_config": { 
    "criteria": { 
      "faithfulness": 1.0,
      "relevance": 0.8 
    } 
  },
  "optimizer_config": {
    "max_iterations": 30,
    "population_size": 10
  },
  "train_dataset": "datasets/train.json",
  "validation_dataset": "datasets/val.json",
  "log_level": "INFO",
  "print_detailed_results": true
}

Execute with your custom config:

agents-cli eval optimize --config my_opt_config.json

The optimizer_config section controls the evolutionary algorithm's behavior, while eval_config defines the weighted criteria for evaluating prompt candidates.

Capturing and Applying Results

The optimized prompt appears in your terminal output upon completion. Capture it programmatically:

OPT=$(agents-cli eval optimize --config my_opt_config.json | awk '/Optimized Prompt:/,0')
echo "$OPT" > optimized_prompt.txt

Important: The optimizer does not automatically update your manifest.yaml. You must manually replace the existing system instruction with the contents of the optimized output.

Summary

• agents-cli eval optimize is an experimental command that runs the ADK GEPA optimizer against specified evaluation metrics.
• The pipeline involves three stages: configuration loading (_load_configs_and_datasets), dataset conversion (optimize_utils.py), and execution (_execute_optimization_run).
• Converted datasets are temporarily stored in .tmp/<id>.evalset.json within your agent directory.
• Use --dataset and --target-metric flags to override defaults, or --config for advanced JSON-based configuration.
• The optimizer modifies the system instruction (top-level prompt) to maximize metrics like relevance or faithfulness, but you must manually copy the result back to manifest.yaml.

Frequently Asked Questions

Is agents-cli eval optimize stable for production use?

No. According to the source code in cmd_optimize.py, this feature is explicitly experimental. The CLI prints a warning at runtime stating that the command behavior may change in future releases. Use it for development and testing, but not for production automation without pinning your CLI version.

Which file does the optimizer actually modify?

The optimizer only reads your agent's manifest.yaml to extract the current system instruction. It does not write back to your source files. Instead, it outputs the optimized prompt to stdout. You must manually update your manifest.yaml with the new prompt after reviewing the results.

How do I control the number of optimization iterations?

Set the max_iterations parameter within the optimizer_config object in your JSON configuration file. For example, "max_iterations": 30 limits the evolutionary process to 30 generations. You can also adjust population_size to control how many prompt variations are tested per iteration.

What evaluation metrics can I target?

You can optimize for any metric defined in your ADK evaluation configuration. Common choices include relevance and faithfulness, specified either via the --target-metric CLI flag or as weighted criteria in the eval_config section of your optimization JSON. The source code in optimize_utils.py handles the conversion of these metrics into ADK-compatible EvalSet rubrics.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}