How the Munder Difflin Circuit Breaker Recovers from Tripped States

The Munder Difflin circuit breaker recovers incrementally by decrementing the agent's severity level one step toward "healthy" on each heartbeat tick where no trip conditions are met, gradually restoring full operation through the evaluation cycle in src/main/breaker.ts.

The Munder Difflin circuit breaker implements a graduated recovery mechanism that prevents abrupt state resets by requiring multiple healthy evaluations to return an agent to full capacity. Located in the open-source chaitanyagiri/munder-difflin repository, this system processes recovery candidates on every "breaker beat," stepping down through severity levels only when telemetry confirms the underlying instability has resolved.

Incremental Level Stepping in src/main/breaker.ts

The core recovery logic executes during the breaker.tick() evaluation cycle. When the system detects no tripping conditions, it enters the recovery branch at lines 62-68:

} else {
  // recovery – move one level down, never below ‘healthy’
  target = LEVELS[Math.max(rank(s.level) - 1, 0)]; // recover one level
}

This incremental decrement ensures an agent transitions from constrained to steering on the next healthy tick, then to healthy on the subsequent tick. The rank() function maps string severity levels to numeric indices, while Math.max(rank(s.level) - 1, 0) guarantees the level never drops below the healthy baseline (index 0). This staged approach prevents thrashing by requiring sustained stability rather than allowing instant recovery after a single benign observation.

Telemetry Signals That Enable Recovery

Before the decrement logic can execute, the breaker must clear the counters that triggered the original escalation. The system monitors specific telemetry signals to reset internal trip conditions:

Distinct Tool Call Detection

When an agent invokes a new combination of toolName and toolInput parameters, the recordToolUse method resets repetition counters. According to lines 57-66 in breaker.ts, this distinct tool call updates lastDistinctToolAt and zeroes both repeatCount and errorCount, removing the quantitative conditions that forced the level upward.

Progress Indicators

The progressing boolean flag in the heartbeat sample indicates forward movement, such as file modification time changes. Lines 33-42 of the evaluation block clear the noProgressBeats debounce counter when this flag is true, signaling that the agent has resumed productive work and halting further escalation.

Compaction Completion

When compaction ends, the system shortens the exemption window (lines 87-90), re-enabling normal velocity checks. This signal indicates resource pressure has eased, permitting the breaker to evaluate standard metrics against thresholds again.

Immediate Recovery When Disabled

If administrators disable the circuit breaker via configuration (cfg.enabled === false), the system bypasses gradual stepping entirely. Lines 16-24 in breaker.ts force every agent to the healthy state immediately, wiping any lingering level assignments or trip reasons. This provides an emergency override for scenarios requiring instant restoration of full agent capabilities.

Practical Recovery Implementation

The following TypeScript example demonstrates the transition from a tripped state back to healthy operation through distinct tool usage:

import { CircuitBreaker } from './breaker';

// Initialize breaker with configuration
const breaker = new CircuitBreaker(() => ({
  enabled: true,
  hardStop: false,
  repeatedToolLimit: 8,
  errorStormLimit: 5,
  tokenVelocityPerMin: 60_000,
}));

// Simulate tripped state via repeated identical tool calls
breaker.recordToolUse('agent-42', 'search', { query: 'foo' });
breaker.recordToolUse('agent-42', 'search', { query: 'foo' });
// ... repeat until repeatCount ≥ repeatedToolLimit

// Escalation tick moves to next severity level
let decisions = breaker.tick(
  [{ agentId: 'agent-42', sample: null, progressing: false }],
  Date.now()
);
console.log(decisions[0].state.level); // "steering" or "constrained"

// Distinct tool call resets internal counters
breaker.recordToolUse('agent-42', 'write', { text: 'new content' });

// Recovery tick decrements level one step
decisions = breaker.tick(
  [{ agentId: 'agent-42', sample: null, progressing: false }],
  Date.now()
);
console.log(decisions[0].state.level); // Steps toward "healthy"

Summary

• Gradual stepping: Recovery decrements the LEVELS array index by exactly one per healthy tick, transitioning constrained → steering → healthy without skips.
• Counter clearing: Distinct tool calls, progress flags, and compaction events reset repeatCount, errorCount, and noProgressBeats to eliminate trip conditions.
• Safety floors: The Math.max(rank(s.level) - 1, 0) calculation ensures levels never descend below healthy (index 0).
• Emergency override: Setting enabled: false forces immediate return to healthy state according to lines 16-24 in breaker.ts.
• Tick-driven evaluation: All recovery state transitions occur exclusively during the breaker.tick() execution in src/main/breaker.ts.

Frequently Asked Questions

How many ticks does full recovery require?

Full recovery requires one healthy tick per severity level. An agent at constrained requires two consecutive healthy evaluations to reach healthy, while steering requires one. The total wall-clock time depends on the heartbeat interval configured in src/main/index.ts.

What triggers immediate recovery to the healthy state?

Configuration disables trigger instant recovery. When cfg.enabled evaluates to false, lines 16-24 of src/main/breaker.ts immediately reset all agents to healthy, bypassing the incremental stepping mechanism entirely and clearing all trip reasons.

Can the recovery rate be accelerated or configured?

The repository does not expose a recovery acceleration parameter. The step-down rate is hardcoded to one level per tick in the else branch of the evaluation logic (lines 62-68). Operators seeking faster recovery must either disable the breaker entirely or ensure agents generate distinct tool calls or progress signals to clear trip counters immediately.

Where is the circuit breaker recovery logic located?

The primary recovery implementation resides in src/main/breaker.ts, specifically lines 62-68 where the target level calculation decrements the rank. Supporting configuration definitions exist in src/main/config.ts, while the heartbeat integration and tick scheduling occur in src/main/index.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}