# How to Set Up BenchmarkAgent for Automated Dataset Selection in AutoResearchClaw

> Learn how to set up BenchmarkAgent in AutoResearchClaw for automated dataset selection. Automate data loading with this powerful four-stage pipeline and streamline your research workflow.

- Repository: [AIMING Lab/AutoResearchClaw](https://github.com/aiming-lab/AutoResearchClaw)
- Tags: how-to-guide
- Published: 2026-05-28

---

**The BenchmarkAgent in AutoResearchClaw automates dataset and baseline selection through a four-stage pipeline (Survey → Select → Acquire → Validate) that generates ready-to-use data loader code during Stage 9 of the research pipeline.**

AutoResearchClaw transforms single research ideas into full conference-ready papers by orchestrating a 23-stage automated pipeline. A critical component of this system is the **BenchmarkAgent**, which handles intelligent dataset selection and baseline generation during the experiment design phase, eliminating the need for manual dataset crawling and configuration.

## Architectural Overview

The BenchmarkAgent operates as a sub-pipeline within Stage 9 (Experiment Design), coordinating four specialized sub-agents to produce a complete `BenchmarkPlan`. According to the source code in [`researchclaw/agents/benchmark_agent/orchestrator.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/agents/benchmark_agent/orchestrator.py), the architecture follows a strict four-phase execution model:

**SurveyorAgent** ([`researchclaw/agents/benchmark_agent/surveyor.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/agents/benchmark_agent/surveyor.py), lines 48-107): Queries a local knowledge base, searches the HuggingFace Hub, and falls back to LLM reasoning to discover candidate datasets and baselines matching your research topic.

**SelectorAgent** ([`researchclaw/agents/benchmark_agent/selector.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/agents/benchmark_agent/selector.py)): Filters raw candidates by enforcing tier limits, minimum benchmark counts, and cache preferences. The selector ensures you have at least `min_benchmarks` datasets and `min_baselines` competing implementations before proceeding.

**AcquirerAgent** ([`researchclaw/agents/benchmark_agent/acquirer.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/agents/benchmark_agent/acquirer.py)): Generates concrete Python code snippets including `data_loader_code`, `baseline_code`, `setup_code`, and `requirements` strings for each selected item. This code is later injected directly into the experiment implementation.

**ValidatorAgent** ([`researchclaw/agents/benchmark_agent/validator.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/agents/benchmark_agent/validator.py)): Runs lightweight sandbox validation on generated snippets, checking syntax, import whitelist compliance, and NaN/Inf guards. If validation fails, the orchestrator automatically retries the Acquire phase.

The **BenchmarkOrchestrator** ([`researchclaw/agents/benchmark_agent/orchestrator.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/agents/benchmark_agent/orchestrator.py), lines 63-84) coordinates these phases and implements a retry loop (default `max_iterations: 2`) between the Acquirer and Validator for self-healing code generation.

## Configuration Options

The BenchmarkAgent is controlled via the `BenchmarkAgentConfig` dataclass in [`researchclaw/config.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/config.py) (lines 458-475). All settings are nested under the `benchmark_agent:` key in your [`config.arc.yaml`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/config.arc.yaml):

```yaml
benchmark_agent:
  enabled: true               # Master toggle for the entire sub-pipeline

  enable_hf_search: true      # Allow HuggingFace Hub queries

  max_hf_results: 10          # Maximum datasets from HF per query

  enable_web_search: false    # Optional Google Scholar search

  max_web_results: 5
  web_search_min_local: 3     # Skip web search if ≥3 local benchmarks exist

  tier_limit: 2               # 1=pre-cached, 2=downloadable, 3=large external

  min_benchmarks: 1           # Minimum datasets required

  min_baselines: 2            # Minimum baseline implementations required

  prefer_cached: true         # Prioritize locally cached datasets

  max_iterations: 2           # Retry budget for Acquire→Validate loop

```

**Key enforcement rules:**
- The **tier_limit** parameter (1-3) controls dataset eligibility. Tier 1 items exist in the local knowledge base ([`researchclaw/data/benchmark_knowledge.yaml`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/data/benchmark_knowledge.yaml)), while tier 3 requires large external downloads.
- When **prefer_cached** is true, the SelectorAgent weights cached datasets higher during the selection phase.
- If validation fails, the orchestrator retries up to `max_iterations` before failing the stage.

## Running the BenchmarkAgent

### Basic CLI Usage

Initialize a configuration and run the full pipeline:

```bash
git clone https://github.com/aiming-lab/AutoResearchClaw.git
cd AutoResearchClaw
python3 -m venv .venv && source .venv/bin/activate
pip install -e .

# Create configuration (or copy config.researchclaw.example.yaml)

researchclaw init

# Execute with auto-approval

export OPENAI_API_KEY="sk-..."
researchclaw run --topic "Few-shot graph classification with GNNs" --auto-approve

```

During Stage 9, the console displays:

```

BenchmarkAgent starting for: Few-shot graph classification …
Phase 1: Surveying benchmarks … (found 3 HF datasets, 0 LLM fallback)
Phase 2: Selecting benchmarks and baselines … (selected 2 benchmarks, 2 baselines)
Phase 3: Acquiring code (iteration 1/2) …
Phase 4: Validating code (iteration 1/2) … Validation passed

```

The final `BenchmarkPlan` is serialized to [`artifacts/rc-.../stage-09/benchmark_plan.json`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/artifacts/rc-.../stage-09/benchmark_plan.json).

### Programmatic Access

Access generated code after a run completes:

```python
from researchclaw.pipeline import Runner
from pathlib import Path
import json

runner = Runner("config.arc.yaml")
artifacts = runner.run(topic="Self-supervised vision", auto_approve=True)

# Load the generated plan

plan_path = Path(artifacts) / "stage-09" / "benchmark_plan.json"
plan = json.loads(plan_path.read_text())

print(plan["data_loader_code"])
print(plan["baseline_code"])
print(plan["requirements"])

```

## Pipeline Integration

The BenchmarkAgent integrates at specific points in the 23-stage pipeline:

**Stage 9 – Experiment Design** ([`researchclaw/pipeline/stage_impls/_experiment_design.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/pipeline/stage_impls/_experiment_design.py), lines 345-443): The orchestrator executes here, producing the `BenchmarkPlan` dataclass that includes selected datasets, generated code, and validation status.

**Stage 10 – Code Generation** ([`researchclaw/pipeline/stage_impls/_code_generation.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/pipeline/stage_impls/_code_generation.py), lines 356-383): The plan's `to_prompt_block()` method appends `data_loader_code` and `baseline_code` directly into the code-generation prompt, ensuring the generated experiment scripts use the selected benchmarks.

**Stage 22 – Export/Publish**: The `requirements` field from the BenchmarkPlan merges into the final reproducibility package.

If you disable the agent (`enabled: false`), the experiment-design stage falls back to any manually supplied `experiment_plan` placed in the stage directory.

## Debugging and Customization

**Inspecting intermediate artifacts:** Each phase writes debuggable JSON to the stage directory:
- [`survey_results.json`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/survey_results.json) (raw candidates from Surveyor)
- [`selection_results.json`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/selection_results.json) (filtered choices from Selector)
- [`acquisition_0.json`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/acquisition_0.json) (code generation attempt)
- [`validation_0.json`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/validation_0.json) (sandbox results)

**Extending domain knowledge:** Add new datasets to [`researchclaw/data/benchmark_knowledge.yaml`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/data/benchmark_knowledge.yaml). The SurveyorAgent loads this at runtime via `_KNOWLEDGE_PATH` to discover tier-1 items without API calls.

**Custom selectors:** Subclass `SelectorAgent` and reference your implementation via `benchmark_agent.selector_cls` in the config for domain-specific filtering logic.

## Summary

- The **BenchmarkAgent** automates dataset selection through a four-phase pipeline (Survey-Select-Acquire-Validate) implemented in `researchclaw/agents/benchmark_agent/`.
- Configuration is centralized in [`researchclaw/config.py`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/researchclaw/config.py) (lines 458-475) and controlled via the `benchmark_agent:` YAML block.
- The orchestrator retries failed code generation up to `max_iterations` times to ensure robustness.
- Generated code is injected into Stage 10 (Code Generation) via the `BenchmarkPlan.to_prompt_block()` method.
- All artifacts persist to disk for auditability and reproducibility debugging.

## Frequently Asked Questions

### How do I disable the BenchmarkAgent and use my own datasets?

Set `enabled: false` under the `benchmark_agent:` section in your [`config.arc.yaml`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/config.arc.yaml). When disabled, Stage 9 skips the orchestrator and relies on any manually placed `experiment_plan` in the artifacts directory, allowing you to specify custom datasets and baselines outside the automated selection flow.

### What does the tier_limit parameter control?

The `tier_limit` (values 1-3) restricts which datasets the SelectorAgent considers. Tier 1 includes only pre-cached datasets from the local knowledge base, tier 2 allows downloadable HuggingFace Hub datasets, and tier 3 permits large external downloads. Setting `tier_limit: 1` guarantees offline reproducibility.

### How does the validation retry mechanism work?

If the ValidatorAgent detects syntax errors, unauthorized imports, or runtime exceptions in the generated code, it signals failure to the BenchmarkOrchestrator. The orchestrator then discards the failed attempt and re-runs the Acquirer-Validator loop with updated context, consuming one retry from the `max_iterations` budget (default 2) before either succeeding or failing the stage.

### Where is the BenchmarkPlan stored after execution?

The final `BenchmarkPlan` is serialized as [`benchmark_plan.json`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/benchmark_plan.json) in the Stage 9 artifacts directory (e.g., [`artifacts/rc-2026-.../stage-09/benchmark_plan.json`](https://github.com/aiming-lab/AutoResearchClaw/blob/main/artifacts/rc-2026-.../stage-09/benchmark_plan.json)). This file contains the complete data loader code, baseline implementations, requirements list, and selection rationale for downstream stages and reproducibility audits.