How to Set Up Incremental HEP Physics Experiments in AutoResearchClaw

Enable the --incremental-experiment CLI flag or set experiment.collider_agent.incremental: true in your configuration to preserve existing Stage 12 collider-simulation artifacts and layer new results on top of versioned snapshots.

AutoResearchClaw automates end-to-end high-energy physics (HEP) research pipelines—from topic scoping to final paper generation. When you need to iterate on collider simulations without discarding days of computation, you can set up incremental HEP physics experiments to snapshot your workspace, generate delta prompts, and merge new results with previous runs.

Enabling Incremental Mode

AutoResearchClaw provides three methods to activate incremental experimentation for the collider agent. When enabled, the system checks for existing models/ or events/ directories in the Stage 12 workspace before determining whether to trigger the incremental path.

CLI Flag Method

The fastest way to enable incremental mode is passing the --incremental-experiment argument when invoking the CLI. In researchclaw/cli.py (lines 56–61), this flag updates the runtime configuration to set experiment.collider_agent.incremental to True.

researchclaw run \
    --config config.researchclaw.yaml \
    --profile hep_ph \
    --incremental-experiment \
    --auto-approve

Configuration File Method

You can persist the incremental setting in your YAML configuration under the experiment.collider_agent namespace. The ColliderAgentConfig data class in researchclaw/config.py (lines 33–38) defines the incremental boolean field, which defaults to False.

experiment:
  mode: collider_agent
  collider_agent:
    incremental: true   # Preserve and build upon existing artifacts

Programmatic Method

For scripted workflows, toggle the flag in-memory using Python’s dataclasses.replace before invoking the runner:

from researchclaw.config import RCConfig
import dataclasses

cfg = RCConfig.load("config.yaml")
cfg = dataclasses.replace(
    cfg,
    experiment=dataclasses.replace(
        cfg.experiment,
        collider_agent=dataclasses.replace(
            cfg.experiment.collider_agent,
            incremental=True,
        ),
    ),
)

# Launch the run

from researchclaw.cli import build_parser, cmd_run
parser = build_parser()
args = parser.parse_args(["run", "--config", "config.yaml"])
cmd_run(args)   # Respects the in-memory incremental flag

Stage 12 Execution and Snapshot Management

When experiment.mode equals "collider_agent", the pipeline invokes _execute_experiment_run in researchclaw/pipeline/stage_impls/_execution.py (lines 73–84). If incremental mode is active and the workspace contains existing artifacts, the system performs the following actions:

1. Versioned Snapshot Creation – The current stage-12 directory is copied to stage-12_v<N> (where N increments with each re-entry).
2. Storage Monitoring – A warning emits when the cumulative footprint of all snapshots exceeds 20 GB (lines 26–34).

The script run_hep_pipeline.sh (lines 33–36) demonstrates this behavior in production HEP workflows, automatically preserving prior simulation states before running new iterations.

Delta Prompt Construction and Workspace Preparation

The ColliderAgentSandbox._prepare_workspace method in researchclaw/experiment/collider_agent_sandbox.py orchestrates the incremental handoff to Claude Code. When incremental mode is detected, the system executes a three-phase prompt assembly (lines 17–23):

1. CONTINUATION CONTEXT – Metadata describing the current experimental state
2. PRIOR PLAN – The previous collider_plan.md copied to collider_plan.prev.md
3. NEW / ADDITIONAL TASKS – The specific delta or extension to execute

The merged prompt is written back to collider_plan.md and passed to the Claude Code CLI (lines 99–106). After the sandbox completes, it writes a results.json file that merges with the previous snapshot using a "new keys win, old keys kept" strategy (lines 66–73), ensuring downstream stages (13 through final) see the combined dataset.

Inspecting Incremental Artifacts

After running incremental iterations, you can verify the snapshot chain and inspect metadata:

# List versioned snapshots

ls artifacts/rc-20241015-123456/stage-12_v*

# Expected output:

# stage-12_v1  stage-12_v2

# View snapshot metadata

cat artifacts/rc-20241015-123456/stage-12_v1/INCREMENTAL_SNAPSHOT.txt

The INCREMENTAL_SNAPSHOT.txt file records the timestamp, trigger reason, and prior metrics such as primary_metric and cross_section_pb, providing an audit trail for each experimental iteration.

Summary

• Activation: Use --incremental-experiment, YAML config, or Python API to set ColliderAgentConfig.incremental = True
• Snapshotting: Stage 12 automatically versions into stage-12_v<N> folders when existing models/ or events/ are detected
• Delta Workflow: The sandbox generates three-part prompts (Continuation Context, Prior Plan, New Tasks) to guide Claude Code
• Storage: Monitor total snapshot size; warnings trigger at 20 GB cumulative footprint
• Results Merging: New results.json entries override old ones while preserving untouched keys for downstream stages

Frequently Asked Questions

What triggers the incremental mode in AutoResearchClaw?

Incremental mode activates when the incremental boolean is True in ColliderAgentConfig and the existing Stage 12 workspace contains models/ or events/ artifacts. The pipeline logic in researchclaw/pipeline/stage_impls/_execution.py performs this check before creating versioned snapshots.

How are delta prompts structured for the collider agent?

The ColliderAgentSandbox builds prompts containing three distinct blocks: CONTINUATION CONTEXT (current state), PRIOR PLAN (copied from collider_plan.prev.md), and NEW / ADDITIONAL TASKS (the incremental objective). This structured prompt allows Claude Code to understand the experimental history without re-processing the entire prior workspace.

What happens when snapshot storage exceeds 20 GB?

The system emits a warning via the footprint check in researchclaw/pipeline/stage_impls/_execution.py (lines 26–34). While the pipeline continues executing, you should monitor disk usage or archive older stage-12_v<N> directories to prevent storage exhaustion.

Can I merge results from multiple incremental iterations?

Yes. Each incremental run produces a results.json that merges with the previous snapshot using a strategy where new keys override old ones and existing keys are retained. This merged result propagates to subsequent pipeline stages (13 through final paper generation), enabling cumulative research progress.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}