How to Troubleshoot Experiment Self-Healing and Repair Cycles in AutoResearchClaw

AutoResearchClaw automatically detects experimental failures through structured diagnosis, generates targeted repairs via LLM or OpenCode within a Docker sandbox, and promotes only higher-quality results back to the main pipeline after scoring them against the original.

AutoResearchClaw's self-healing mechanism transforms failed experiments into publication-ready research through automated diagnosis and repair cycles. Understanding how to troubleshoot these repair cycles is essential when experiments stall at the technical_report quality level or fail to produce valid experiment_summary.json files. This guide breaks down the internal architecture, file paths, and debugging strategies for the repair pipeline implemented in the aiming-lab/AutoResearchClaw repository.

Understanding the Self-Healing Architecture

The repair system consists of specialized components that work sequentially to diagnose, patch, and validate experiments. Each component is implemented in specific source files within the repository.

Experiment Diagnosis Engine

The ExperimentDiagnosis class in researchclaw/pipeline/experiment_diagnosis.py (lines 44-77) serves as the entry point for repair cycles. This component analyzes stdout, stderr, runtime metrics, and the experiment plan to produce a structured list of deficiencies. These deficiency objects categorize failures into types such as MISSING_DEPENDENCY, GPU_OOM, and TIME_GUARD_DOMINANT.

Repair Prompt Construction and Execution

The build_repair_prompt function in researchclaw/pipeline/experiment_repair.py (lines 88-94) transforms a diagnosis into a detailed LLM prompt. This prompt includes:

• The diagnosed deficiencies
• A scope-reduction guide for time-constrained repairs
• Current source files requiring modification

The repair executor then calls _get_repaired_code or _repair_via_llm (lines 158-227) to invoke OpenCode (if enabled) or fall back to a plain LLM chat. These methods extract Python code blocks from the response and merge them with the original experiment code.

Sandbox Execution and Quality Reassessment

After patching, _run_experiment_in_sandbox (lines 771-802) executes the modified experiment inside a Docker-based sandbox and returns new stdout, stderr, and metrics. The system then re-runs assess_experiment_quality on the new experiment_summary.json to determine if the repair succeeded.

The select_best_results function (lines 96-128) scores every version using _summary_quality_score, comparing metrics richness, completed conditions, and primary metric presence to select the optimal result.

The Experiment Repair Cycle Flow

According to the implementation in researchclaw/pipeline/runner.py and experiment_repair.py, the repair cycle follows this exact sequence:

1. Stage 14 produces the initial experiment_summary.json
2. Stage 15 runs assess_experiment_quality
3. If quality is insufficient, diagnose_experiment identifies specific deficiencies
4. build_repair_prompt constructs the repair instructions
5. The repair executor generates patched files in stage-14_repair_vN/experiment/ directories
6. _run_experiment_in_sandbox executes the patched version
7. Quality is reassessed on the new summary
8. The loop repeats up to MAX_REPAIR_CYCLES (default 3) or until reaching full-paper mode
9. select_best_results chooses the highest-scoring summary
10. The promotion guard validates the repair against the original before overwriting stage-14/experiment_summary.json

The pipeline orchestration logic in _run_experiment_repair (runner.py lines 46-78) manages the handoff between diagnosis and repair execution.

Common Failure Points and Debugging Strategies

Symptom	Root Cause	Debug Strategy
ModuleNotFoundError: No module named 'xyz'	Missing dependency (DeficiencyType.MISSING_DEPENDENCY)	Check the generated repair prompt for requirements.txt modifications; manually verify dependencies in the sandbox environment
CUDA out of memory	GPU OOM (DeficiencyType.GPU_OOM)	Inspect the prompt's suggested fixes for batch size reduction or gradient checkpointing; verify GPU memory limits in the Docker sandbox
>50% of conditions skipped	Time-guard dominance (DeficiencyType.TIME_GUARD_DOMINANT)	Review the automatically injected Scope Reduction section in the repair prompt, which suggests keeping only baseline + proposed + one ablation and reducing epochs
Missing experiment_summary.json after repair	Sandbox crash or timeout	Examine stage-14_repair_vN/sandbox/*.json for stdout/stderr; increase repair_cfg.timeout_sec_per_cycle in configuration
Mode remains technical_report after cycles	All repairs scored lower than original	Analyze experiment_repair_result.json → cycle_history[i].diagnosis_summary for deficiency patterns; consider enabling OpenCode via experiment.repair.use_opencode: true or adjusting hyper-parameters

The Promotion Guard and BUG-198 Protection

The promotion guard in pipeline/runner.py (lines 70-100) contains a critical safety mechanism documented as BUG-198:

Only promote if the repair summary is RICHER than the existing stage-14 summary. The repair loop can produce empty summaries (metrics: {}, 0 conditions) which would overwrite enriched data from the analysis stage.

This guard compares scores computed by _summary_quality_score, where higher scores indicate more completed conditions, present primary metrics, and richer metric keys. If the repair score does not exceed the original, the canonical stage-14/experiment_summary.json remains unchanged. This prevents downstream stages (sanitizer, paper verifier) from receiving degraded or empty data sets.

Practical Code Examples for Manual Intervention

Inspecting Repair Artifacts

After a pipeline run completes, examine the self-healing process through the generated artifacts:

# View the complete repair cycle history

cat artifacts/rc-20240528-123456/experiment_repair_result.json | jq

# Locate the best repaired code version

ls artifacts/rc-20240528-123456/stage-14_repair_v*/experiment/

Manual Repair Prompt Generation

For debugging specific failures without running the full pipeline:

from pathlib import Path
import json
from researchclaw.pipeline.experiment_diagnosis import diagnose_experiment
from researchclaw.pipeline.experiment_repair import build_repair_prompt

run_dir = Path("artifacts/rc-20240528-123456")
summary = json.loads((run_dir / "stage-14/experiment_summary.json").read_text())
plan = json.loads((run_dir / "stage-09/experiment_design.json").read_text())

# Run diagnosis with captured output

diagnosis = diagnose_experiment(
    experiment_summary=summary,
    experiment_plan=plan,
    stdout="",
    stderr=""
)

# Gather current source files

code = {
    p.name: p.read_text()
    for p in (run_dir / "stage-10/experiment").glob("*.py")
}

# Generate the repair prompt

prompt = build_repair_prompt(
    diagnosis, 
    code, 
    experiment_plan=plan, 
    time_budget_sec=2400
)

print(prompt)  # Feed this directly to an LLM for testing

Extracting Code from LLM Responses

When working with manual LLM interventions, use the internal extraction utility:

from researchclaw.pipeline.experiment_repair import _extract_code_blocks

llm_response = """```python main.py
import torch

# fixed imports

torch==2.2.0
numpy>=1.24

```"""

fixed_files = _extract_code_blocks(llm_response)

# Returns: {"main.py": "import torch\n# fixed imports", 

#          "requirements.txt": "torch==2.2.0\nnumpy>=1.24"}

Configuring Repair Cycle Limits

Override the default MAX_REPAIR_CYCLES = 3 in your configuration file:

experiment:
  repair:
    enabled: true
    max_cycles: 5
    timeout_sec_per_cycle: 900
    use_opencode: true

Summary

• Structured diagnosis in experiment_diagnosis.py categorizes failures into specific deficiency types (dependencies, OOM, time-guards) that drive targeted repairs
• The repair loop runs up to 3 cycles by default, executing patched code in Docker sandboxes and re-assessing quality through _summary_quality_score
• BUG-198 protection prevents the promotion of empty or degraded summaries, ensuring only richer results overwrite the canonical stage-14 data
• Sandbox outputs in stage-14_repair_vN/sandbox/ contain the debugging data needed when experiments crash or time out during repair attempts
• Manual debugging is supported through build_repair_prompt and _extract_code_blocks for isolated testing of repair strategies

Frequently Asked Questions

Why does my experiment remain in technical_report mode after repair cycles complete?

When all repair cycles finish but the quality mode remains technical_report, the promotion guard (BUG-198) has rejected all repairs because they scored lower than the original summary. Check experiment_repair_result.json → cycle_history to view each cycle's diagnosis summary. The repair likely failed to address the root deficiencies, or the original experiment had incomplete but higher-scoring metrics than the attempted fixes.

How do I manually trigger or debug a specific repair cycle?

Import the diagnosis and repair functions directly in a Python REPL as shown in the manual prompt generation example above. This allows you to inspect the exact prompt sent to the LLM, modify the source code inputs, and test different repair strategies without re-running the entire pipeline from stage 1.

What prevents repair cycles from overwriting good data with empty results?

The promotion guard in pipeline/runner.py implements BUG-198 protection by comparing scores via _summary_quality_score. This function evaluates metric richness, condition completion, and primary metric presence. Only summaries with strictly higher scores than the original can overwrite stage-14/experiment_summary.json, protecting downstream stages from empty dictionaries or zero-condition outputs.

How do I resolve persistent missing dependency errors in the repair loop?

When DeficiencyType.MISSING_DEPENDENCY appears repeatedly, verify that the repair prompt correctly identifies the missing package in requirements.txt. If using OpenCode (use_opencode: true), ensure the sandbox environment has network access for pip installations. For manual fixes, update stage-10/experiment/requirements.txt before the repair cycle begins, or increase the time_budget_sec parameter to allow dependency installation during the sandbox execution.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}