Configuring SSH Remote Sandbox Execution with GPU Support in AutoResearchClaw

AutoResearchClaw enables GPU-accelerated experiments on remote SSH hosts through the SshRemoteConfig class, which maps specific GPU indices to CUDA_VISIBLE_DEVICES or Docker --gpus arguments without modifying experiment code.

AutoResearchClaw executes machine learning experiments inside isolated sandboxes. The SSH remote sandbox implementation allows you to run code on any SSH-accessible GPU server, from lab workstations to cloud instances, while the framework handles code staging, environment setup, and result collection automatically.

Understanding the SSH Remote Sandbox Architecture

The sandbox system separates configuration from execution. When you set mode="ssh_remote" in your experiment configuration, the factory in researchclaw/experiment/factory.py (lines 45‑62) instantiates a SshRemoteSandbox rather than a local sandbox. This class orchestrates the entire remote workflow: uploading code via scp, creating unique remote directories, executing experiments, parsing metrics, and cleaning up temporary files.

Under the hood, researchclaw/servers/ssh_executor.py provides asynchronous rsync and ssh helpers that ensure reliable file transfer and command execution across network instability.

Configuring GPU Access via SshRemoteConfig

All SSH-specific settings live in the SshRemoteConfig dataclass defined in researchclaw/config.py (lines 31‑41). This configuration bridges your local experiment definition with the remote GPU resources.

Essential Configuration Fields

The following fields control GPU visibility and remote environment setup:

• gpu_ids – A tuple of integer indices (e.g., (0, 1)) specifying which GPUs the experiment may access. The sandbox translates these into environment variables or Docker arguments.
• host – Target hostname or IP address of the GPU server.
• remote_workdir – Absolute path on the remote host where temporary experiment files reside (e.g., /tmp/rc_experiments).
• setup_commands – Optional shell commands executed before the experiment starts, such as module load cuda/12.0 or conda activation.
• use_docker – Boolean flag; when True, the experiment runs inside a Docker container on the remote host with GPU passthrough enabled.

Authentication and Connectivity

The sandbox supports key-based authentication via the key_path field (typically ~/.ssh/id_rsa). Before launching expensive workloads, validate connectivity using SshRemoteSandbox.check_ssh_available (implemented in researchclaw/experiment/ssh_sandbox.py, lines 37‑55), which verifies SSH access and basic command execution without staging files.

How GPU Mapping Works Under the Hood

The SshRemoteSandbox class in researchclaw/experiment/ssh_sandbox.py handles GPU assignment differently depending on whether Docker is enabled.

Bare Metal Execution with CUDA_VISIBLE_DEVICES

When use_docker=False, the sandbox injects GPU restrictions directly into the shell environment. The method _build_bare_exec_cmd (lines 88‑92) constructs commands prefaced with CUDA_VISIBLE_DEVICES=<ids>, ensuring only the specified GPUs are visible to PyTorch or TensorFlow.

Docker-based Execution with --gpus flag

When use_docker=True, the sandbox instead passes GPU IDs to the Docker runtime. The method _build_docker_exec_cmd (lines 44‑51) appends --gpus device=<id1>,<id2> to the docker run invocation. The container inherits the same GPU mapping while maintaining environment isolation.

Remote Hardware Detection

Before the pipeline schedules a GPU-dependent experiment, AutoResearchClaw probes the remote host to verify available accelerators. The detect_hardware function calls _detect_nvidia_remote in researchclaw/hardware.py (lines 98‑138), which runs nvidia-smi over SSH to query GPU names, VRAM capacity, and driver versions. This information builds a HardwareProfile that ensures experiments land on nodes with sufficient resources.

Implementation Examples

The following patterns demonstrate common configurations for GPU-enabled remote execution.

# Define a remote sandbox with GPU IDs 0 and 1

from researchclaw.config import Config, SshRemoteConfig

cfg = Config(
    experiment=Config.ExperimentConfig(
        mode="ssh_remote",
        ssh_remote=SshRemoteConfig(
            host="gpu.lab.edu",
            user="alice",
            key_path="~/.ssh/id_rsa",
            gpu_ids=(0, 1),                # expose GPUs 0 and 1

            remote_workdir="/tmp/rc_experiments",
            setup_commands=("module load cuda/12.0",),  # optional init

            use_docker=False,              # run Python directly

        ),
    )
)

# Run a short experiment (the pipeline will pick the sandbox automatically)

from researchclaw.pipeline import run_experiment

run_experiment(
    config=cfg,
    code="""
import torch
print("GPU count:", torch.cuda.device_count())
print("GPU 0 name:", torch.cuda.get_device_name(0))
""",
    timeout_sec=120,
)

# Using Docker on the remote host (GPU IDs are passed to Docker)

cfg.experiment.ssh_remote = cfg.experiment.ssh_remote.__class__(
    host="gpu.lab.edu",
    gpu_ids=(0, 1),
    use_docker=True,
    docker_image="pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime",
)

# The same `run_experiment` call now runs inside the Docker container,

# still respecting the selected GPUs.

# Verify SSH connectivity before launching a large job

from researchclaw.experiment.ssh_sandbox import SshRemoteSandbox

ok, msg = SshRemoteSandbox.check_ssh_available(cfg.experiment.ssh_remote)
print(msg)   # → “SSH connection to gpu.lab.edu OK”

Summary

• SSH Remote Sandbox enables experiment execution on any SSH-accessible GPU server without code changes.
• SshRemoteConfig (researchclaw/config.py lines 31‑41) defines GPU indices via gpu_ids, Docker usage via use_docker, and environment setup via setup_commands.
• GPU isolation occurs through CUDA_VISIBLE_DEVICES for bare-metal runs (_build_bare_exec_cmd, lines 88‑92) or --gpus flags for Docker runs (_build_docker_exec_cmd, lines 44‑51).
• Hardware detection (_detect_nvidia_remote, lines 98‑138) queries remote GPU capacity before scheduling.
• Pre-flight validation via check_ssh_available (lines 37‑55) prevents failed launches due to connectivity issues.

Frequently Asked Questions

How do I specify which GPUs to use on the remote host?

Set the gpu_ids tuple in SshRemoteConfig to the indices of the desired GPUs, such as (0,) for the first GPU or (0, 1) for the first two. The sandbox automatically exports CUDA_VISIBLE_DEVICES=0,1 on the remote host or passes --gpus device=0,1 to Docker, ensuring your experiment only sees those specific devices.

Can I use Docker containers with GPU support in the SSH sandbox?

Yes. Set use_docker=True in SshRemoteConfig and provide a docker_image string. According to the source code in researchclaw/experiment/ssh_sandbox.py (lines 44‑51), the sandbox constructs a docker run --gpus device=<ids> command that maps the configured gpu_ids into the container while maintaining full isolation from the host environment.

How does AutoResearchClaw verify GPU availability before running experiments?

Before scheduling, the pipeline calls detect_hardware, which invokes _detect_nvidia_remote in researchclaw/hardware.py (lines 98‑138). This function executes nvidia-smi over SSH to retrieve GPU names and VRAM statistics, building a HardwareProfile that matches experiments to nodes with adequate resources.

What authentication methods are supported for SSH remote sandboxes?

AutoResearchClaw uses key-based authentication configured via the key_path field in SshRemoteConfig, typically pointing to a private key like ~/.ssh/id_rsa. The check_ssh_available method validates the connection before any files are transferred, ensuring the private key has proper permissions and network connectivity exists before experiment staging begins.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}