How to Synthesize Multi-Turn Evaluation Scenarios with agents-cli eval dataset synthesize

Use agents-cli eval dataset synthesize to generate synthetic, multi-turn conversation traces by simulating user interactions with your ADK agent, which can then be graded using agents-cli eval grade.

The agents-cli tool from Google provides a dedicated command for generating evaluation datasets for ADK (Agent Development Kit) agents. By running agents-cli eval dataset synthesize, you can automatically create realistic, multi-turn conversation scenarios that test your agent's behavior across extended interactions without manual scripting.

Architecture and Execution Flow

The synthesis process orchestrates several components across the CLI and a staged runner script to ensure your agent executes within its native environment.

CLI Entry Point and Project Setup

The command begins in src/google/agents/cli/eval/cmd_dataset.py, where the cmd_synthesize function handles argument parsing via Click. When you invoke the command, the CLI first validates your project structure using find_project_root() and read_project_config() to locate agents-cli-manifest.yaml.

The CLI then prepares the execution environment by running uv sync --dev --extra eval, which installs the necessary Vertex AI client and ADK evaluation SDK dependencies required for simulation.

Runner Staging and Subprocess Isolation

To ensure the synthesis runs with your project's specific Python interpreter and virtual environment, the CLI uses _stage_synthesize_runner() (lines 50-66 in cmd_dataset.py). This function copies src/google/agents/cli/eval/_synthesize_runner.py into a hidden directory named .agents-cli-scripts/ within your project root.

This staging approach guarantees that the runner executes inside your project's virtual-env, inheriting environment variables like GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION from your .env file.

Inside the Synthesis Runner

The staged _synthesize_runner.py executes the core synthesis logic:

1. Agent Loading: Uses AgentLoader.load() to instantiate your ADK agent from the specified path.
2. Simulation Setup: Configures a LlmBackedUserSimulator with your chosen model (defaulting to Vertex AI Gemini models) to act as the synthetic user.
3. Trace Generation: The EvaluationGenerator iteratively calls simulator.run_one_turn(agent) for each turn, recording user messages, tool calls, and agent responses.
4. Output Serialization: Transforms each Invocation into JSON-serializable turns using _invocations_to_turns and writes the complete traces to your specified output path.

The runner includes safety mechanisms such as _safe_tool_declarations (lines 90-104), which patches AgentConfig._get_tool_declarations_from_agent to handle non-introspectable tools gracefully.

Command-Line Options and Configuration

The synthesis command accepts several parameters to control the simulation behavior:

• -n, --count: Number of scenarios to generate (default: 3).
• --max-turns: Maximum conversation turns per scenario (default: 5).
• --instruction: Natural language guidance for the user simulator's behavior.
• --environment-context: Contextual information injected into the simulation (e.g., current date, available data).
• --model: Specific Vertex AI model name for the user simulator (e.g., gemini-1.5-flash-preview).
• -o, --output: Destination path for the JSON trace file (default: artifacts/traces/traces-<timestamp>.json).

Path resolution logic in src/google/agents/cli/eval/_paths.py handles the default output directory creation under artifacts/traces/ when no explicit output path is provided.

Practical Examples

Basic Synthesis with Defaults

Run the command with no arguments to generate traces using default settings:

agents-cli eval dataset synthesize

This creates 3 scenarios, each with up to 5 turns, saving results to artifacts/traces/traces-<timestamp>.json in your project directory.

Custom Scenario Parameters

Control the volume and depth of generated scenarios:

agents-cli eval dataset synthesize \
    -n 10 \
    --max-turns 8 \
    --instruction "User changes destination after the first suggestion" \
    --environment-context "Today is Monday; flights to Paris are available."

This configuration produces 10 distinct scenarios with up to 8 turns each, steering the simulated user toward specific interaction patterns.

Selecting Specific Models

Specify which Vertex AI model powers the user simulator:

agents-cli eval dataset synthesize \
    --model gemini-1.5-flash-preview \
    -o my_traces.json

This uses the lighter Flash model for faster generation and writes output directly to my_traces.json rather than the default artifacts directory.

Programmatic Usage

You can also invoke the synthesis logic directly from Python, bypassing the CLI subprocess:

import json
from pathlib import Path
from google.agents.cli.eval._synthesize_runner import (
    EvaluationGenerator,
    LlmBackedUserSimulator,
    LlmBackedUserSimulatorConfig,
    AgentLoader,
)

# Load your ADK agent

agent = AgentLoader.load(Path("my_agent/"))

# Configure simulation parameters

config = {
    "count": 2,
    "generation_instruction": "User wants a cheap flight.",
    "environment_context": "Current date: 2026-07-02",
    "model_name": "gemini-1.5-pro",
}

# Initialize generator with custom config

simulator = LlmBackedUserSimulator(
    config=LlmBackedUserSimulatorConfig(
        model_name=config.get("model_name"),
        max_turns=4
    )
)
generator = EvaluationGenerator(simulator=simulator)

# Generate and save traces

traces = generator.generate(agent, count=config["count"])
output_path = Path("synthetic_traces.json")
output_path.write_text(json.dumps(traces, indent=2))
print(f"Saved {len(traces)} traces to {output_path}")

This approach mirrors the internal flow of _synthesize_runner.py and allows integration into custom data pipelines or debugging workflows.

Summary

• agents-cli eval dataset synthesize generates synthetic multi-turn conversations for ADK agent evaluation by simulating realistic user interactions.
• The command operates through a staged runner pattern, copying _synthesize_runner.py into .agents-cli-scripts/ to execute within your project's virtual environment.
• Key source files include cmd_dataset.py (CLI entry), _synthesize_runner.py (execution logic), and _paths.py (artifact resolution).
• Default behavior produces 3 scenarios with 5 turns each, but you can customize via --count, --max-turns, and --instruction flags.
• Output traces are JSON-formatted conversation records compatible with agents-cli eval grade for downstream automated evaluation.

Frequently Asked Questions

What is the default output location for synthesized traces?

If you do not specify the -o or --output flag, the command writes traces to artifacts/traces/traces-<timestamp>.json relative to your project root. The resolve_output_path helper in _paths.py automatically creates this directory structure and generates timestamped filenames to prevent overwrites.

Why does the CLI copy a runner script into my project directory?

The _stage_synthesize_runner() function copies _synthesize_runner.py into .agents-cli-scripts/ to ensure the synthesis executes using your project's Python interpreter and installed dependencies. This isolation prevents version conflicts between the CLI's own environment and your ADK agent's specific package requirements.

Can I use a different model for the user simulator than the one my agent uses?

Yes. The --model parameter configures the LlmBackedUserSimulator independently from your agent's configuration. You can specify any Vertex AI model (such as gemini-1.5-pro or gemini-1.5-flash-preview) to control the simulation quality and speed without affecting the agent being tested.

How does the tool handle agents with non-standard tool declarations?

The runner includes _safe_tool_declarations, a monkey-patch for AgentConfig._get_tool_declarations_from_agent that safely extracts tool schemas even when agents expose non-introspectable or complex tool configurations. This ensures the EvaluationGenerator can initialize properly regardless of your agent's specific tool setup.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}