Configuring CodeAgent v2 with Hard Validation in AutoResearchClaw: A Complete Guide

Enable hard validation in AutoResearchClaw by setting hard_validation=True in CodeAgentConfig, which triggers AST-based syntax checks and a targeted LLM repair loop before sandbox execution.

AutoResearchClaw’s CodeAgent v2 is a multi-phase LLM-driven code-generation engine that plans, generates, and repairs research code. Hard validation serves as a critical quality gate between generation and execution, using static analysis to catch syntax errors, import issues, and structural problems before they hit the runtime environment.

What Is Hard Validation?

Hard validation is Phase 2.5 of the CodeAgent v2 pipeline. According to the implementation in researchclaw/pipeline/code_agent.py, this step runs immediately after sequential file generation to validate the entire codebase using Python’s ast module.

The system categorizes findings into two severity levels:

• Critical issues: Syntax errors, missing imports, undefined variables, missing if __name__ == "__main__": guards, and class quality violations. These trigger an automatic repair cycle.
• Warnings: Style concerns or trivial computations that are logged but do not block execution.

When critical issues are detected, the agent invokes _hard_validate_and_repair() (lines 567-605 in code_agent.py), which enters a targeted repair loop limited by hard_validation_max_repairs.

Configuration Methods

You can enable hard validation through three interfaces: direct Python instantiation, YAML configuration, or runtime mutation.

Python API Configuration

The most explicit method creates a CodeAgentConfig instance and embeds it within ExperimentConfig (defined in researchclaw/config.py):

from researchclaw.config import ExperimentConfig, CodeAgentConfig

# Enable CodeAgent v2 with hard validation and 3 repair attempts

config = ExperimentConfig(
    code_agent=CodeAgentConfig(
        enabled=True,
        hard_validation=True,
        hard_validation_max_repairs=3,
        sequential_generation=True,
        architecture_planning=True
    )
)

Pass this config object to run_experiment() or the CLI entry point.

YAML and CLI Configuration

For batch experiments or reproducible workflows, configure via YAML:

# experiment.yaml

code_agent:
  enabled: true
  hard_validation: true          # Activate AST validation gates

  hard_validation_max_repair: 5  # Allow up to 5 repair cycles

  sequential_generation: true
  architecture_planning: true

Execute with:

python -m researchclaw.cli --config experiment.yaml

The ExperimentConfig.code_agent field (lines 61-62 in researchclaw/config.py) automatically maps these YAML keys to the CodeAgentConfig dataclass.

Runtime Overrides

For interactive development or Jupyter notebooks, modify an existing configuration using replace():

exp_cfg = ExperimentConfig()
exp_cfg = exp_cfg.replace(
    code_agent=exp_cfg.code_agent.replace(
        hard_validation=False,  # Disable for faster prototyping

        hard_validation_max_repairs=1
    )
)

The Validation and Repair Pipeline

Understanding the internal mechanics helps tune the hard_validation_max_repairs parameter effectively.

AST-Based Checks

The _hard_validate() method (lines 707-784 in researchclaw/pipeline/code_agent.py) parses each generated Python file and runs validators from researchclaw/experiment/validator.py (lines 497-590):

• Syntax validation: ast.parse() failures are always critical.
• Class quality: Detects empty subclasses, duplicate implementations, or improper nn.Module usage.
• Complexity analysis: Flags hard-coded metrics that bypass actual computation.
• API correctness: Validates that all referenced names exist in scope or imports.
• Import consistency: Cross-file imports must resolve to existing symbols.
• Entry point guard: Missing if __name__ == "__main__": in main.py is treated as critical.

The Repair Loop

When _hard_validate() returns critical issues, _repair_critical_issues() (lines 722-734) constructs a detailed prompt containing:

1. The original architecture blueprint.
2. The current file contents.
3. A enumerated list of critical errors.

The LLM responds with corrected code blocks formatted as ```filename:xxx.py ```. The agent merges these updates and reruns validation until either zero critical issues remain or the repair counter reaches hard_validation_max_repairs.

Complete Usage Example

Below is a minimal script that generates a CNN training script with hard validation enabled:

import pathlib
from researchclaw.cli import run_experiment
from researchclaw.config import ExperimentConfig, CodeAgentConfig

# Configure with hard validation active

cfg = ExperimentConfig(
    code_agent=CodeAgentConfig(
        enabled=True,
        hard_validation=True,
        hard_validation_max_repairs=4,
        sequential_generation=True
    )
)

# Research topic

topic = "Train a CNN on CIFAR-10 with early stopping"

# Execute pipeline

result = run_experiment(
    topic=topic,
    config=cfg,
    work_dir=pathlib.Path("./validation_demo")
)

# Inspect results

print("Generated files:", list(result.files.keys()))
print("Validation log:")
for entry in result.validation_log:
    print(entry)

Typical output shows the validation gate in action:

[CodeAgent] Phase 2.5: Hard validation gates
  CRITICAL: [main.py] Missing `if __name__ == "__main__":` block
  WARNING: [model.py] Trivial computation detected
  Targeted repair for critical issues
  Repair updated 1 file(s): main.py
  Hard validation passed (1 warning(s), attempt 1)

Summary

• Hard validation in AutoResearchClaw’s CodeAgent v2 acts as a static analysis gate between generation and execution, catching syntax and structural errors early.
• Configure it via CodeAgentConfig(hard_validation=True, hard_validation_max_repairs=N) in Python or YAML.
• Critical issues found by _hard_validate() in researchclaw/pipeline/code_agent.py trigger a targeted repair loop using _repair_critical_issues().
• Validation helpers in researchclaw/experiment/validator.py check for AST validity, import consistency, class quality, and the presence of main.py guards.
• Higher hard_validation_max_repairs values increase robustness at the cost of additional LLM token usage and latency.

Frequently Asked Questions

What types of errors does hard validation catch?

Hard validation catches syntax errors, missing imports, undefined variables, cross-file import inconsistencies, and missing __main__ guards in main.py. These are classified as critical issues because they would cause immediate runtime failures. The system also logs warnings for style issues and trivial computations, but these do not trigger repair cycles.

How many repair attempts should I allow?

The optimal hard_validation_max_repairs value depends on your LLM model and code complexity. For GPT-4 class models, 3-5 attempts typically resolve syntax and import issues without excessive cost. If you observe repeated failures beyond 5 iterations, the architectural blueprint may be underspecified, and you should review the validation logs rather than increasing the limit indefinitely.

Can I disable hard validation for faster iteration?

Yes. Set hard_validation=False in CodeAgentConfig or override it at runtime. This skips Phase 2.5 entirely, sending generated code directly to the execution-and-repair loop. While this reduces latency, you risk encountering avoidable runtime crashes that hard validation would have caught during the static analysis phase.

Where are the validation rules defined?

The concrete AST checks are implemented in researchclaw/experiment/validator.py (lines 497-590). These include check_class_quality, check_code_complexity, check_api_correctness, and scoping validators. The orchestration logic that invokes these checks resides in _hard_validate() within researchclaw/pipeline/code_agent.py (lines 707-784).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}