Circuit Breaker Levels and Per-Beat Recovery in Munder Difflin

Munder Difflin implements a four-step circuit breaker ladder (healthy, steering, constrained, stopped) that escalates agents one level per heartbeat when trip conditions are met and automatically recovers one level per healthy beat when all signals clear.

Munder Difflin’s circuit breaker protects AI agents from runaway loops, excessive token velocity, and cost overruns through deterministic state management evaluated on every system heartbeat. Unlike binary kill switches, this architecture provides graduated interventions that allow agents to rehabilitate automatically when failure conditions resolve. The implementation in src/main/breaker.ts defines distinct circuit breaker levels and step-wise recovery mechanics that balance safety with operational continuity.

The Four Circuit Breaker Levels Defined

The circuit breaker maintains four discrete states defined at lines 28-31 of src/main/breaker.ts. Each level represents a specific intervention severity:

• Healthy: Default state indicating normal operation. The agent executes without restrictions.
• Steering: Mild intervention state triggered by initial trip conditions. The system issues corrective "steer" messages to nudge the agent back on track.
• Constrained: Restricted operation state escalated from steering. The agent faces stronger limitations such as reduced tool access or rate limiting.
• Stopped: Terminal state where execution halts completely. The agent is killed and archived, escalated from constrained when hard-stop conditions fire.

The levels form a one-way ladder during healthy operation, with recovery moving downward toward healthy and escalation moving upward toward stopped.

How Escalation Works During Each Beat

During every heartbeat, the CircuitBreaker.tick() method evaluates all agents through the evaluate() function. If any agent exceeds configured thresholds, the breaker promotes that agent exactly one level higher (capped by the hardStop configuration).

Trip conditions that trigger escalation include:

• Repeated identical tool calls exceeding repeatedToolLimit
• Consecutive API error storms where errorCount ≥ errorStormLimit
• Per-agent token budget exhaustion beyond agentTokenCaps[agentId]
• Floor-wide cost or token cap violations (blaming the top spender)
• Token velocity spikes surpassing tokenVelocityPerMin
• Stagnation detection when noProgressBeats ≥ NO_PROGRESS_BEATS

The promotion logic resides at lines 66-68 of breaker.ts, where the breaker increments the level rank when any trip condition evaluates true:

// When a trip condition is detected (lines 66-68)
const nextLevel = escalationPolicy.promote(currentLevel);

Per-Beat Recovery and De-escalation

When an agent completes a heartbeat with no trip conditions active, the circuit breaker automatically recovers one level toward healthy. This de-escalation occurs in the same evaluation cycle as promotion checks, as implemented at lines 66-68:

// src/main/breaker.ts
const target = LEVELS[Math.max(rank(s.level) - 1, 0)]; // recover one level

Key recovery characteristics include:

• Step-wise movement: Agents move only one rung per beat, preventing oscillation between stopped and healthy
• Signal dependency: Recovery requires all failure signals to clear simultaneously (no repeated tools, stabilized velocity, observable progress)
• Transition tracking: The changed flag in the returned BreakerDecision indicates level transitions, enabling the orchestrator to emit corrective messages only when states actually shift

This design ensures that recovery is gradual and verifies sustained healthy behavior before restoring full capabilities.

Complete Lifecycle Implementation

The full per-beat lifecycle flows through src/main/index.ts integration and follows this deterministic sequence:

1. Signal collection aggregates tool usage, errors, token consumption, and progress flags
2. Evaluation via evaluate() checks each agent against threshold configurations loaded from src/main/config.ts
3. Escalation promotes one level if any trip condition fires (up to constrained or stopped based on hardStop)
4. Recovery demotes one level if no conditions are met
5. Action emission generates BreakerAction types (steer, constrain, stop, or none) based on the new level

The following example demonstrates instantiating the breaker with live configuration and processing a 30-second heartbeat:

import { CircuitBreaker } from './breaker';
import { getConfig } from './config';

// Instantiate with live config getter
const breaker = new CircuitBreaker(() => getConfig().circuitBreaker ?? {});

// Simulate heartbeat every 30 seconds
setInterval(() => {
  const now = Date.now();
  const inputs = [
    {
      agentId: 'creed-xyz',
      sample: { 
        input: 200, 
        output: 500, 
        cacheRead: 0, 
        cacheCreation: 0, 
        usd: 0.12, 
        ts: now 
      },
      progressing: true,
    },
    // ...additional agents
  ];
  
  const decisions = breaker.tick(inputs, now);
  decisions.forEach(d => {
    console.log(`${d.state.agentId} → ${d.state.level} (${d.state.reason})`);
    if (d.action !== 'none') {
      sendBreakerMessage(d.state.agentId, d.action);
    }
  });
}, 30_000);

Summary

• Munder Difflin implements four circuit breaker levels (healthy, steering, constrained, stopped) defined in src/main/breaker.ts
• Escalation occurs one level per beat when agents trigger trip conditions such as repeated tool calls, error storms, or token overruns
• Recovery automatically de-escalates one level per healthy beat when all failure signals clear, preventing abrupt state oscillations
• The ladder architecture balances gradual intervention with eventual termination, allowing rehabilitation while guaranteeing hard stops for runaway agents
• Configuration thresholds including repeatedToolLimit, tokenVelocityPerMin, and NO_PROGRESS_BEATS tune the sensitivity in src/main/config.ts

Frequently Asked Questions

How many levels can an agent move in a single heartbeat?

An agent can move exactly one level per heartbeat in either direction. The step-wise design at lines 66-68 of breaker.ts uses Math.max(rank(s.level) - 1, 0) for recovery and single-step promotion for escalation, preventing agents from jumping directly from healthy to stopped or vice versa.

What prevents the circuit breaker from rapidly oscillating between levels?

The changed flag in BreakerDecision tracks actual state transitions, while the requirement for all trip conditions to clear simultaneously ensures sustained healthy behavior before recovery. The tick() method evaluates all six failure signals (tool repeats, errors, token caps, velocity, progress) and only recovers when the agent demonstrates complete metric stability.

Where are the threshold limits configured for trip conditions?

Thresholds are defined in src/main/config.ts through the CircuitBreakerConfig interface. Key parameters include repeatedToolLimit for identical tool calls, errorStormLimit for consecutive API errors, agentTokenCaps for per-agent budgets, and tokenVelocityPerMin for rate limiting. The breaker accepts a config getter function to enable dynamic threshold updates without restarts.

What happens when an agent reaches the stopped level?

When an agent escalates to stopped (either from constrained progression or hard-stop triggers), execution halts immediately. According to the implementation in src/main/breaker.ts, the orchestrator receives a stop action in the BreakerDecision, triggering kill and archive operations. Recovery from stopped requires manual intervention or agent redeployment rather than automatic de-escalation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}