How to Set Up BenchmarkAgent for Automated Dataset Selection in AutoResearchClaw

The BenchmarkAgent in AutoResearchClaw automates dataset and baseline selection through a four-stage pipeline (Survey → Select → Acquire → Validate) that generates ready-to-use data loader code during Stage 9 of the research pipeline.

AutoResearchClaw transforms single research ideas into full conference-ready papers by orchestrating a 23-stage automated pipeline. A critical component of this system is the BenchmarkAgent, which handles intelligent dataset selection and baseline generation during the experiment design phase, eliminating the need for manual dataset crawling and configuration.

Architectural Overview

The BenchmarkAgent operates as a sub-pipeline within Stage 9 (Experiment Design), coordinating four specialized sub-agents to produce a complete BenchmarkPlan. According to the source code in researchclaw/agents/benchmark_agent/orchestrator.py, the architecture follows a strict four-phase execution model:

SurveyorAgent (researchclaw/agents/benchmark_agent/surveyor.py, lines 48-107): Queries a local knowledge base, searches the HuggingFace Hub, and falls back to LLM reasoning to discover candidate datasets and baselines matching your research topic.

SelectorAgent (researchclaw/agents/benchmark_agent/selector.py): Filters raw candidates by enforcing tier limits, minimum benchmark counts, and cache preferences. The selector ensures you have at least min_benchmarks datasets and min_baselines competing implementations before proceeding.

AcquirerAgent (researchclaw/agents/benchmark_agent/acquirer.py): Generates concrete Python code snippets including data_loader_code, baseline_code, setup_code, and requirements strings for each selected item. This code is later injected directly into the experiment implementation.

ValidatorAgent (researchclaw/agents/benchmark_agent/validator.py): Runs lightweight sandbox validation on generated snippets, checking syntax, import whitelist compliance, and NaN/Inf guards. If validation fails, the orchestrator automatically retries the Acquire phase.

The BenchmarkOrchestrator (researchclaw/agents/benchmark_agent/orchestrator.py, lines 63-84) coordinates these phases and implements a retry loop (default max_iterations: 2) between the Acquirer and Validator for self-healing code generation.

Configuration Options

The BenchmarkAgent is controlled via the BenchmarkAgentConfig dataclass in researchclaw/config.py (lines 458-475). All settings are nested under the benchmark_agent: key in your config.arc.yaml:

benchmark_agent:
  enabled: true               # Master toggle for the entire sub-pipeline

  enable_hf_search: true      # Allow HuggingFace Hub queries

  max_hf_results: 10          # Maximum datasets from HF per query

  enable_web_search: false    # Optional Google Scholar search

  max_web_results: 5
  web_search_min_local: 3     # Skip web search if ≥3 local benchmarks exist

  tier_limit: 2               # 1=pre-cached, 2=downloadable, 3=large external

  min_benchmarks: 1           # Minimum datasets required

  min_baselines: 2            # Minimum baseline implementations required

  prefer_cached: true         # Prioritize locally cached datasets

  max_iterations: 2           # Retry budget for Acquire→Validate loop

Key enforcement rules:

• The tier_limit parameter (1-3) controls dataset eligibility. Tier 1 items exist in the local knowledge base (researchclaw/data/benchmark_knowledge.yaml), while tier 3 requires large external downloads.
• When prefer_cached is true, the SelectorAgent weights cached datasets higher during the selection phase.
• If validation fails, the orchestrator retries up to max_iterations before failing the stage.

Running the BenchmarkAgent

Basic CLI Usage

Initialize a configuration and run the full pipeline:

git clone https://github.com/aiming-lab/AutoResearchClaw.git
cd AutoResearchClaw
python3 -m venv .venv && source .venv/bin/activate
pip install -e .

# Create configuration (or copy config.researchclaw.example.yaml)

researchclaw init

# Execute with auto-approval

export OPENAI_API_KEY="sk-..."
researchclaw run --topic "Few-shot graph classification with GNNs" --auto-approve

During Stage 9, the console displays:

BenchmarkAgent starting for: Few-shot graph classification …
Phase 1: Surveying benchmarks … (found 3 HF datasets, 0 LLM fallback)
Phase 2: Selecting benchmarks and baselines … (selected 2 benchmarks, 2 baselines)
Phase 3: Acquiring code (iteration 1/2) …
Phase 4: Validating code (iteration 1/2) … Validation passed

The final BenchmarkPlan is serialized to artifacts/rc-.../stage-09/benchmark_plan.json.

Programmatic Access

Access generated code after a run completes:

from researchclaw.pipeline import Runner
from pathlib import Path
import json

runner = Runner("config.arc.yaml")
artifacts = runner.run(topic="Self-supervised vision", auto_approve=True)

# Load the generated plan

plan_path = Path(artifacts) / "stage-09" / "benchmark_plan.json"
plan = json.loads(plan_path.read_text())

print(plan["data_loader_code"])
print(plan["baseline_code"])
print(plan["requirements"])

Pipeline Integration

The BenchmarkAgent integrates at specific points in the 23-stage pipeline:

Stage 9 – Experiment Design (researchclaw/pipeline/stage_impls/_experiment_design.py, lines 345-443): The orchestrator executes here, producing the BenchmarkPlan dataclass that includes selected datasets, generated code, and validation status.

Stage 10 – Code Generation (researchclaw/pipeline/stage_impls/_code_generation.py, lines 356-383): The plan's to_prompt_block() method appends data_loader_code and baseline_code directly into the code-generation prompt, ensuring the generated experiment scripts use the selected benchmarks.

Stage 22 – Export/Publish: The requirements field from the BenchmarkPlan merges into the final reproducibility package.

If you disable the agent (enabled: false), the experiment-design stage falls back to any manually supplied experiment_plan placed in the stage directory.

Debugging and Customization

Inspecting intermediate artifacts: Each phase writes debuggable JSON to the stage directory:

• survey_results.json (raw candidates from Surveyor)
• selection_results.json (filtered choices from Selector)
• acquisition_0.json (code generation attempt)
• validation_0.json (sandbox results)

Extending domain knowledge: Add new datasets to researchclaw/data/benchmark_knowledge.yaml. The SurveyorAgent loads this at runtime via _KNOWLEDGE_PATH to discover tier-1 items without API calls.

Custom selectors: Subclass SelectorAgent and reference your implementation via benchmark_agent.selector_cls in the config for domain-specific filtering logic.

Summary

• The BenchmarkAgent automates dataset selection through a four-phase pipeline (Survey-Select-Acquire-Validate) implemented in researchclaw/agents/benchmark_agent/.
• Configuration is centralized in researchclaw/config.py (lines 458-475) and controlled via the benchmark_agent: YAML block.
• The orchestrator retries failed code generation up to max_iterations times to ensure robustness.
• Generated code is injected into Stage 10 (Code Generation) via the BenchmarkPlan.to_prompt_block() method.
• All artifacts persist to disk for auditability and reproducibility debugging.

Frequently Asked Questions

How do I disable the BenchmarkAgent and use my own datasets?

Set enabled: false under the benchmark_agent: section in your config.arc.yaml. When disabled, Stage 9 skips the orchestrator and relies on any manually placed experiment_plan in the artifacts directory, allowing you to specify custom datasets and baselines outside the automated selection flow.

What does the tier_limit parameter control?

The tier_limit (values 1-3) restricts which datasets the SelectorAgent considers. Tier 1 includes only pre-cached datasets from the local knowledge base, tier 2 allows downloadable HuggingFace Hub datasets, and tier 3 permits large external downloads. Setting tier_limit: 1 guarantees offline reproducibility.

How does the validation retry mechanism work?

If the ValidatorAgent detects syntax errors, unauthorized imports, or runtime exceptions in the generated code, it signals failure to the BenchmarkOrchestrator. The orchestrator then discards the failed attempt and re-runs the Acquirer-Validator loop with updated context, consuming one retry from the max_iterations budget (default 2) before either succeeding or failing the stage.

Where is the BenchmarkPlan stored after execution?

The final BenchmarkPlan is serialized as benchmark_plan.json in the Stage 9 artifacts directory (e.g., artifacts/rc-2026-.../stage-09/benchmark_plan.json). This file contains the complete data loader code, baseline implementations, requirements list, and selection rationale for downstream stages and reproducibility audits.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}