# Circuit Breaker Levels and Per-Beat Recovery in Munder Difflin

> Explore Munder Difflin's four circuit breaker levels healthy steering constrained stopped and understand per beat recovery. Learn how agents escalate and recover automatically.

- Repository: [Chaitanya Giri/munder-difflin](https://github.com/chaitanyagiri/munder-difflin)
- Tags: deep-dive
- Published: 2026-08-29

---

**Munder Difflin implements a four-step circuit breaker ladder (healthy, steering, constrained, stopped) that escalates agents one level per heartbeat when trip conditions are met and automatically recovers one level per healthy beat when all signals clear.**

Munder Difflin’s circuit breaker protects AI agents from runaway loops, excessive token velocity, and cost overruns through deterministic state management evaluated on every system heartbeat. Unlike binary kill switches, this architecture provides graduated interventions that allow agents to rehabilitate automatically when failure conditions resolve. The implementation in [`src/main/breaker.ts`](https://github.com/chaitanyagiri/munder-difflin/blob/main/src/main/breaker.ts) defines distinct circuit breaker levels and step-wise recovery mechanics that balance safety with operational continuity.

## The Four Circuit Breaker Levels Defined

The circuit breaker maintains four discrete states defined at **lines 28-31** of [`src/main/breaker.ts`](https://github.com/chaitanyagiri/munder-difflin/blob/main/src/main/breaker.ts). Each level represents a specific intervention severity:

- **Healthy**: Default state indicating normal operation. The agent executes without restrictions.
- **Steering**: Mild intervention state triggered by initial trip conditions. The system issues corrective "steer" messages to nudge the agent back on track.
- **Constrained**: Restricted operation state escalated from steering. The agent faces stronger limitations such as reduced tool access or rate limiting.
- **Stopped**: Terminal state where execution halts completely. The agent is killed and archived, escalated from constrained when hard-stop conditions fire.

The levels form a one-way ladder during healthy operation, with recovery moving downward toward `healthy` and escalation moving upward toward `stopped`.

## How Escalation Works During Each Beat

During every heartbeat, the `CircuitBreaker.tick()` method evaluates all agents through the `evaluate()` function. If any agent exceeds configured thresholds, the breaker **promotes** that agent exactly one level higher (capped by the `hardStop` configuration).

Trip conditions that trigger escalation include:

- **Repeated identical tool calls** exceeding `repeatedToolLimit`
- **Consecutive API error storms** where `errorCount ≥ errorStormLimit`
- **Per-agent token budget exhaustion** beyond `agentTokenCaps[agentId]`
- **Floor-wide cost or token cap violations** (blaming the top spender)
- **Token velocity spikes** surpassing `tokenVelocityPerMin`
- **Stagnation detection** when `noProgressBeats ≥ NO_PROGRESS_BEATS`

The promotion logic resides at **lines 66-68** of [`breaker.ts`](https://github.com/chaitanyagiri/munder-difflin/blob/main/breaker.ts), where the breaker increments the level rank when any trip condition evaluates true:

```typescript
// When a trip condition is detected (lines 66-68)
const nextLevel = escalationPolicy.promote(currentLevel);

```

## Per-Beat Recovery and De-escalation

When an agent completes a heartbeat with **no trip conditions** active, the circuit breaker automatically **recovers** one level toward `healthy`. This de-escalation occurs in the same evaluation cycle as promotion checks, as implemented at **lines 66-68**:

```typescript
// src/main/breaker.ts
const target = LEVELS[Math.max(rank(s.level) - 1, 0)]; // recover one level

```

Key recovery characteristics include:

- **Step-wise movement**: Agents move only one rung per beat, preventing oscillation between `stopped` and `healthy`
- **Signal dependency**: Recovery requires all failure signals to clear simultaneously (no repeated tools, stabilized velocity, observable progress)
- **Transition tracking**: The `changed` flag in the returned `BreakerDecision` indicates level transitions, enabling the orchestrator to emit corrective messages only when states actually shift

This design ensures that recovery is gradual and verifies sustained healthy behavior before restoring full capabilities.

## Complete Lifecycle Implementation

The full per-beat lifecycle flows through [`src/main/index.ts`](https://github.com/chaitanyagiri/munder-difflin/blob/main/src/main/index.ts) integration and follows this deterministic sequence:

1. **Signal collection** aggregates tool usage, errors, token consumption, and progress flags
2. **Evaluation** via `evaluate()` checks each agent against threshold configurations loaded from [`src/main/config.ts`](https://github.com/chaitanyagiri/munder-difflin/blob/main/src/main/config.ts)
3. **Escalation** promotes one level if any trip condition fires (up to `constrained` or `stopped` based on `hardStop`)
4. **Recovery** demotes one level if no conditions are met
5. **Action emission** generates `BreakerAction` types (`steer`, `constrain`, `stop`, or `none`) based on the new level

The following example demonstrates instantiating the breaker with live configuration and processing a 30-second heartbeat:

```typescript
import { CircuitBreaker } from './breaker';
import { getConfig } from './config';

// Instantiate with live config getter
const breaker = new CircuitBreaker(() => getConfig().circuitBreaker ?? {});

// Simulate heartbeat every 30 seconds
setInterval(() => {
  const now = Date.now();
  const inputs = [
    {
      agentId: 'creed-xyz',
      sample: { 
        input: 200, 
        output: 500, 
        cacheRead: 0, 
        cacheCreation: 0, 
        usd: 0.12, 
        ts: now 
      },
      progressing: true,
    },
    // ...additional agents
  ];
  
  const decisions = breaker.tick(inputs, now);
  decisions.forEach(d => {
    console.log(`${d.state.agentId} → ${d.state.level} (${d.state.reason})`);
    if (d.action !== 'none') {
      sendBreakerMessage(d.state.agentId, d.action);
    }
  });
}, 30_000);

```

## Summary

- Munder Difflin implements **four circuit breaker levels** (`healthy`, `steering`, `constrained`, `stopped`) defined in [`src/main/breaker.ts`](https://github.com/chaitanyagiri/munder-difflin/blob/main/src/main/breaker.ts)
- **Escalation** occurs one level per beat when agents trigger trip conditions such as repeated tool calls, error storms, or token overruns
- **Recovery** automatically de-escalates one level per healthy beat when all failure signals clear, preventing abrupt state oscillations
- The ladder architecture balances **gradual intervention** with **eventual termination**, allowing rehabilitation while guaranteeing hard stops for runaway agents
- Configuration thresholds including `repeatedToolLimit`, `tokenVelocityPerMin`, and `NO_PROGRESS_BEATS` tune the sensitivity in [`src/main/config.ts`](https://github.com/chaitanyagiri/munder-difflin/blob/main/src/main/config.ts)

## Frequently Asked Questions

### How many levels can an agent move in a single heartbeat?

An agent can move **exactly one level** per heartbeat in either direction. The step-wise design at lines 66-68 of [`breaker.ts`](https://github.com/chaitanyagiri/munder-difflin/blob/main/breaker.ts) uses `Math.max(rank(s.level) - 1, 0)` for recovery and single-step promotion for escalation, preventing agents from jumping directly from `healthy` to `stopped` or vice versa.

### What prevents the circuit breaker from rapidly oscillating between levels?

The **changed** flag in `BreakerDecision` tracks actual state transitions, while the requirement for **all** trip conditions to clear simultaneously ensures sustained healthy behavior before recovery. The `tick()` method evaluates all six failure signals (tool repeats, errors, token caps, velocity, progress) and only recovers when the agent demonstrates complete metric stability.

### Where are the threshold limits configured for trip conditions?

Thresholds are defined in [`src/main/config.ts`](https://github.com/chaitanyagiri/munder-difflin/blob/main/src/main/config.ts) through the `CircuitBreakerConfig` interface. Key parameters include `repeatedToolLimit` for identical tool calls, `errorStormLimit` for consecutive API errors, `agentTokenCaps` for per-agent budgets, and `tokenVelocityPerMin` for rate limiting. The breaker accepts a config getter function to enable dynamic threshold updates without restarts.

### What happens when an agent reaches the `stopped` level?

When an agent escalates to `stopped` (either from `constrained` progression or hard-stop triggers), execution halts immediately. According to the implementation in [`src/main/breaker.ts`](https://github.com/chaitanyagiri/munder-difflin/blob/main/src/main/breaker.ts), the orchestrator receives a `stop` action in the `BreakerDecision`, triggering kill and archive operations. Recovery from `stopped` requires manual intervention or agent redeployment rather than automatic de-escalation.