# How to Synthesize Multi-Turn Evaluation Scenarios with agents-cli eval dataset synthesize

> Synthesize multi-turn evaluation scenarios for your ADK agent using agents-cli eval dataset synthesize. Generate test conversations and grade them with agents-cli eval grade for comprehensive testing.

- Repository: [Google/agents-cli](https://github.com/google/agents-cli)
- Tags: how-to-guide
- Published: 2026-07-02

---

**Use `agents-cli eval dataset synthesize` to generate synthetic, multi-turn conversation traces by simulating user interactions with your ADK agent, which can then be graded using `agents-cli eval grade`.**

The `agents-cli` tool from Google provides a dedicated command for generating evaluation datasets for ADK (Agent Development Kit) agents. By running `agents-cli eval dataset synthesize`, you can automatically create realistic, multi-turn conversation scenarios that test your agent's behavior across extended interactions without manual scripting.

## Architecture and Execution Flow

The synthesis process orchestrates several components across the CLI and a staged runner script to ensure your agent executes within its native environment.

### CLI Entry Point and Project Setup

The command begins in [`src/google/agents/cli/eval/cmd_dataset.py`](https://github.com/google/agents-cli/blob/main/src/google/agents/cli/eval/cmd_dataset.py), where the `cmd_synthesize` function handles argument parsing via Click. When you invoke the command, the CLI first validates your project structure using `find_project_root()` and `read_project_config()` to locate [`agents-cli-manifest.yaml`](https://github.com/google/agents-cli/blob/main/agents-cli-manifest.yaml).

The CLI then prepares the execution environment by running `uv sync --dev --extra eval`, which installs the necessary Vertex AI client and ADK evaluation SDK dependencies required for simulation.

### Runner Staging and Subprocess Isolation

To ensure the synthesis runs with your project's specific Python interpreter and virtual environment, the CLI uses `_stage_synthesize_runner()` (lines 50-66 in [`cmd_dataset.py`](https://github.com/google/agents-cli/blob/main/cmd_dataset.py)). This function copies [`src/google/agents/cli/eval/_synthesize_runner.py`](https://github.com/google/agents-cli/blob/main/src/google/agents/cli/eval/_synthesize_runner.py) into a hidden directory named `.agents-cli-scripts/` within your project root.

This staging approach guarantees that the runner executes inside your project's virtual-env, inheriting environment variables like `GOOGLE_CLOUD_PROJECT` and `GOOGLE_CLOUD_LOCATION` from your `.env` file.

### Inside the Synthesis Runner

The staged [`_synthesize_runner.py`](https://github.com/google/agents-cli/blob/main/_synthesize_runner.py) executes the core synthesis logic:

1. **Agent Loading**: Uses `AgentLoader.load()` to instantiate your ADK agent from the specified path.
2. **Simulation Setup**: Configures a `LlmBackedUserSimulator` with your chosen model (defaulting to Vertex AI Gemini models) to act as the synthetic user.
3. **Trace Generation**: The `EvaluationGenerator` iteratively calls `simulator.run_one_turn(agent)` for each turn, recording user messages, tool calls, and agent responses.
4. **Output Serialization**: Transforms each `Invocation` into JSON-serializable turns using `_invocations_to_turns` and writes the complete traces to your specified output path.

The runner includes safety mechanisms such as `_safe_tool_declarations` (lines 90-104), which patches `AgentConfig._get_tool_declarations_from_agent` to handle non-introspectable tools gracefully.

## Command-Line Options and Configuration

The synthesis command accepts several parameters to control the simulation behavior:

- **`-n, --count`**: Number of scenarios to generate (default: 3).
- **`--max-turns`**: Maximum conversation turns per scenario (default: 5).
- **`--instruction`**: Natural language guidance for the user simulator's behavior.
- **`--environment-context`**: Contextual information injected into the simulation (e.g., current date, available data).
- **`--model`**: Specific Vertex AI model name for the user simulator (e.g., `gemini-1.5-flash-preview`).
- **`-o, --output`**: Destination path for the JSON trace file (default: `artifacts/traces/traces-<timestamp>.json`).

Path resolution logic in [`src/google/agents/cli/eval/_paths.py`](https://github.com/google/agents-cli/blob/main/src/google/agents/cli/eval/_paths.py) handles the default output directory creation under `artifacts/traces/` when no explicit output path is provided.

## Practical Examples

### Basic Synthesis with Defaults

Run the command with no arguments to generate traces using default settings:

```bash
agents-cli eval dataset synthesize

```

This creates 3 scenarios, each with up to 5 turns, saving results to `artifacts/traces/traces-<timestamp>.json` in your project directory.

### Custom Scenario Parameters

Control the volume and depth of generated scenarios:

```bash
agents-cli eval dataset synthesize \
    -n 10 \
    --max-turns 8 \
    --instruction "User changes destination after the first suggestion" \
    --environment-context "Today is Monday; flights to Paris are available."

```

This configuration produces 10 distinct scenarios with up to 8 turns each, steering the simulated user toward specific interaction patterns.

### Selecting Specific Models

Specify which Vertex AI model powers the user simulator:

```bash
agents-cli eval dataset synthesize \
    --model gemini-1.5-flash-preview \
    -o my_traces.json

```

This uses the lighter Flash model for faster generation and writes output directly to [`my_traces.json`](https://github.com/google/agents-cli/blob/main/my_traces.json) rather than the default artifacts directory.

### Programmatic Usage

You can also invoke the synthesis logic directly from Python, bypassing the CLI subprocess:

```python
import json
from pathlib import Path
from google.agents.cli.eval._synthesize_runner import (
    EvaluationGenerator,
    LlmBackedUserSimulator,
    LlmBackedUserSimulatorConfig,
    AgentLoader,
)

# Load your ADK agent

agent = AgentLoader.load(Path("my_agent/"))

# Configure simulation parameters

config = {
    "count": 2,
    "generation_instruction": "User wants a cheap flight.",
    "environment_context": "Current date: 2026-07-02",
    "model_name": "gemini-1.5-pro",
}

# Initialize generator with custom config

simulator = LlmBackedUserSimulator(
    config=LlmBackedUserSimulatorConfig(
        model_name=config.get("model_name"),
        max_turns=4
    )
)
generator = EvaluationGenerator(simulator=simulator)

# Generate and save traces

traces = generator.generate(agent, count=config["count"])
output_path = Path("synthetic_traces.json")
output_path.write_text(json.dumps(traces, indent=2))
print(f"Saved {len(traces)} traces to {output_path}")

```

This approach mirrors the internal flow of [`_synthesize_runner.py`](https://github.com/google/agents-cli/blob/main/_synthesize_runner.py) and allows integration into custom data pipelines or debugging workflows.

## Summary

- **`agents-cli eval dataset synthesize`** generates synthetic multi-turn conversations for ADK agent evaluation by simulating realistic user interactions.
- The command operates through a **staged runner pattern**, copying [`_synthesize_runner.py`](https://github.com/google/agents-cli/blob/main/_synthesize_runner.py) into `.agents-cli-scripts/` to execute within your project's virtual environment.
- **Key source files** include [`cmd_dataset.py`](https://github.com/google/agents-cli/blob/main/cmd_dataset.py) (CLI entry), [`_synthesize_runner.py`](https://github.com/google/agents-cli/blob/main/_synthesize_runner.py) (execution logic), and [`_paths.py`](https://github.com/google/agents-cli/blob/main/_paths.py) (artifact resolution).
- Default behavior produces **3 scenarios** with **5 turns each**, but you can customize via `--count`, `--max-turns`, and `--instruction` flags.
- Output traces are JSON-formatted conversation records compatible with `agents-cli eval grade` for downstream automated evaluation.

## Frequently Asked Questions

### What is the default output location for synthesized traces?

If you do not specify the `-o` or `--output` flag, the command writes traces to `artifacts/traces/traces-<timestamp>.json` relative to your project root. The `resolve_output_path` helper in [`_paths.py`](https://github.com/google/agents-cli/blob/main/_paths.py) automatically creates this directory structure and generates timestamped filenames to prevent overwrites.

### Why does the CLI copy a runner script into my project directory?

The `_stage_synthesize_runner()` function copies [`_synthesize_runner.py`](https://github.com/google/agents-cli/blob/main/_synthesize_runner.py) into `.agents-cli-scripts/` to ensure the synthesis executes using your project's Python interpreter and installed dependencies. This isolation prevents version conflicts between the CLI's own environment and your ADK agent's specific package requirements.

### Can I use a different model for the user simulator than the one my agent uses?

Yes. The `--model` parameter configures the `LlmBackedUserSimulator` independently from your agent's configuration. You can specify any Vertex AI model (such as `gemini-1.5-pro` or `gemini-1.5-flash-preview`) to control the simulation quality and speed without affecting the agent being tested.

### How does the tool handle agents with non-standard tool declarations?

The runner includes `_safe_tool_declarations`, a monkey-patch for `AgentConfig._get_tool_declarations_from_agent` that safely extracts tool schemas even when agents expose non-introspectable or complex tool configurations. This ensures the `EvaluationGenerator` can initialize properly regardless of your agent's specific tool setup.