How Background-Agents Implements a Circuit Breaker Mechanism for Sandbox Failures

Background-Agents uses a classic circuit breaker pattern with a default threshold of 3 consecutive failures within a 5-minute window to prevent cascading sandbox spawn failures.

The ColeMurray/background-agents repository protects its sandbox-spawning pipeline with a robust circuit breaker mechanism that throttles rapid retry attempts after repeated permanent failures. This pattern tracks consecutive spawn failures in a state object and blocks new requests when thresholds are exceeded, giving underlying infrastructure time to recover while preventing endless spawn loops. The implementation separates pure decision logic from side effects, making it both testable and resilient.

Circuit Breaker State and Configuration

Tracking Failures with CircuitBreakerState

Each sandbox record stores a CircuitBreakerState object that maintains the failure count and timestamp. In packages/control-plane/src/sandbox/lifecycle/decisions.ts (lines 35-42), the state interface captures two critical fields:

• failureCount: The number of consecutive spawn failures
• lastFailureTime: The timestamp of the most recent failure

Configurable Thresholds and Time Windows

The CircuitBreakerConfig defines the policy boundaries. The DEFAULT_CIRCUIT_BREAKER_CONFIG in packages/control-plane/src/sandbox/lifecycle/decisions.ts (lines 58-61) establishes:

• threshold: 3 failures
• windowMs: 300,000 milliseconds (5 minutes)

These defaults balance sensitivity with stability, ensuring temporary glitches do not immediately block spawning while still protecting against persistent issues.

Decision Logic: Evaluating the Circuit Breaker

The pure function evaluateCircuitBreaker(state, config, now) in packages/control-plane/src/sandbox/lifecycle/decisions.ts (lines 75-85) determines whether to allow or block spawn requests. It follows three discrete steps:

1. Resets the failure count if the time since the last failure exceeds windowMs
2. Opens the circuit when failureCount >= threshold, returning a waitTimeMs until retry
3. Allows the spawn to proceed if within limits

import {
  evaluateCircuitBreaker,
  DEFAULT_CIRCUIT_BREAKER_CONFIG,
  type CircuitBreakerState,
} from "@open-inspect/control-plane/sandbox/lifecycle/decisions";

async function canSpawnSandbox(state: CircuitBreakerState): Promise<boolean> {
  const now = Date.now();
  const decision = evaluateCircuitBreaker(
    state,
    DEFAULT_CIRCUIT_BREAKER_CONFIG,
    now,
  );

  if (!decision.shouldProceed) {
    console.warn(
      `Circuit breaker open – wait ${decision.waitTimeMs!} ms before retrying`,
    );
    return false;
  }

  if (decision.shouldReset) {
    // Reset failure count in DB (not shown)
  }

  return true;
}

Permanent vs. Transient Error Handling

The sandbox lifecycle manager distinguishes between error types to avoid false positives. According to the source code in packages/control-plane/src/sandbox/lifecycle/manager.ts (line 569), the circuit breaker only increments for permanent errors such as misconfiguration, deliberately ignoring transient network blips.

Unit tests in packages/control-plane/src/sandbox/lifecycle/manager.test.ts (line 930) verify that transient errors do not increment the failure counter. This selective counting ensures temporary infrastructure hiccups do not trigger unnecessary circuit breaks.

import { SandboxProviderError } from "./provider";

try {
  await sandboxProvider.spawn(...);
} catch (err) {
  if (err instanceof SandboxProviderError && err.errorType === "permanent") {
    await db.incrementCircuitBreaker(sessionId);
  }
  // Transient errors are logged but do not affect the breaker
}

Summary

• CircuitBreakerState tracks failureCount and lastFailureTime for each sandbox record in the database
• Default configuration allows 3 failures within a 5-minute window before opening the circuit
• Pure decision function evaluateCircuitBreaker handles state transitions without side effects, enabling deterministic testing
• Permanent error filtering prevents transient network issues from triggering false breaks, as enforced in manager.ts
• Implementation files located in packages/control-plane/src/sandbox/lifecycle/decisions.ts and manager.ts

Frequently Asked Questions

What triggers the circuit breaker to open?

The circuit opens when failureCount exceeds the configured threshold (default 3) within the windowMs period (default 5 minutes), as implemented in evaluateCircuitBreaker in decisions.ts.

How long does the circuit breaker stay open?

The circuit remains open until the time since the last failure exceeds the windowMs value (5 minutes by default), at which point the failure count resets and spawning resumes.

Why does the mechanism ignore transient errors?

The lifecycle manager only increments the counter for permanent errors (e.g., misconfiguration) and ignores transient network blips, preventing temporary infrastructure issues from triggering unnecessary cooldown periods.

Where is the circuit breaker logic implemented?

The core logic resides in packages/control-plane/src/sandbox/lifecycle/decisions.ts (state definitions and decision function) and packages/control-plane/src/sandbox/lifecycle/manager.ts (error classification and state updates).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}