How the Reasonix Recovery System Manages Automatic Error Recovery in DeepSeek-Reasonix

The Reasonix recovery system manages automatic error recovery through a coordinated pipeline that persists checkpoints to side-car JSON files, evaluates retry policies against configured budgets, and escalates to human review when automatic attempts exhaust their limits.

The Reasonix recovery system is the error-handling backbone of the DeepSeek-Reasonix project, providing resilient session continuity across transient failures. According to the esengine/DeepSeek-Reasonix source code, the implementation centers on the internal/recovery package, which orchestrates checkpoints, policy rules, and budget gates to decide when to retry automatically versus when to pause for user input.

Core Components of the Reasonix Recovery System

Recovery State and Checkpointing (state.go)

The Recovery State component in internal/recovery/state.go persists a lightweight checkpoint to a *.recovery.json side-car file adjacent to the normal session transcript. This side-car records the last successful step and any pending retry metadata, enabling the engine to resume after a crash or retry pause.

Recovery Rules and Policy Enforcement (rules.go)

Located in internal/recovery/rules.go, the Recovery Rules encapsulate policy decisions such as budget limits, maximum retries, and error-type allowances. The system consults these rules before each automatic retry to determine whether to continue, pause, or abort the operation.

Decision Engine and Error Fingerprinting (decision.go)

The Decision Engine in internal/recovery/decision.go combines the current session context with the rule set to produce concrete actions: continue, pause, or abort. It integrates a fingerprint of the failure to avoid repeating the same unresolvable error, preventing wasted cycles on permanent failures.

Budget Enforcement and Throttling (gate.go)

The Gate component in internal/recovery/gate.go enforces retry budgets—including total retry time and number of attempts—and throttles retries to prevent runaway loops. If the budget exhausts, the gate forces a pause and surfaces a user-visible notice via the serving layer.

Human-in-the-Loop Review (reviewer.go)

When automatic retries stall, the Reviewer in internal/recovery/reviewer.go presents a concise prompt to the user. The prompt includes the recovery hint embedded in error messages (verified in write_tool_test.go), guiding the user to supply missing information or change direction.

Telemetry and Observability (sink.go)

All recovery events emit detailed metrics via internal/telemetry/sink.go. The telemetry records event types such as failure, rule_continue, and human_prompt, enabling operational tuning and behavioral auditing.

Automatic Error Recovery Workflow

The Reasonix recovery system operates through a seven-stage pipeline:

1. Error Detection – When a tool call or internal step fails, the error wraps with a recovery hint that categorizes the failure type.

2. Checkpoint Creation – The current session writes its state to <session>.recovery.json, preserving the SessionRecoveryState documented in session.go.

3. Rule Evaluation – The decision module consults rules.go to verify if the error type qualifies for automatic retry and whether the retry budget permits another attempt.

4. Budget Check – The gate.go module validates that cumulative retry costs remain within configured limits. If limits are exceeded, the system transitions to the paused state.

5. Automatic Retry – If allowed, the engine re-executes the failing step and updates the recovery side-car with the new attempt count.

6. Human Review – If the failure persists beyond automatic attempts, the reviewer presents a UI notice (defined in internal/serve/serve.go under the recovery_paused string) prompting the user to continue or modify the request.

7. Telemetry Emission – Every stage reports events to telemetry/sink.go for monitoring and future system improvements.

Code Implementation Examples

The following patterns demonstrate how to integrate with the Reasonix recovery system in your own handlers:

// Example: Creating a recovery checkpoint when a tool call fails
func handleToolError(err error, sess *session.Session) error {
    if !isRecoverable(err) {
        return err
    }
    // Persist checkpoint for automatic retry
    recPath := sess.RecoveryPath()
    if err := sess.SaveRecovery(recPath); err != nil {
        return fmt.Errorf("failed to persist recovery state: %w", err)
    }
    // Trigger the decision engine
    action, hint := recovery.Decide(err, sess)
    switch action {
    case recovery.Continue:
        return retryToolCall(sess)
    case recovery.Pause:
        return fmt.Errorf("%s – %s", hint, "recovery_paused")
    default:
        return err
    }
}

// Example: Decision logic using rules and gate
func Decide(err error, sess *session.Session) (Action, string) {
    // 1. Apply policy rules
    if !rules.Allowed(err) {
        return Abort, "error not recoverable"
    }
    // 2. Check budget via gate
    if !gate.Allowed(sess) {
        return Pause, "Automatic retries paused"
    }
    // 3. Otherwise continue automatically
    return Continue, ""
}

Summary

• The Reasonix recovery system persists session state to *.recovery.json side-car files via internal/recovery/state.go, enabling crash recovery.
• Budget gates in internal/recovery/gate.go prevent infinite retry loops by enforcing time and attempt limits.
• The decision engine (internal/recovery/decision.go) uses error fingerprinting to avoid retrying unresolvable failures.
• When automatic recovery exhausts its budget, the reviewer component prompts users through the UI defined in internal/serve/serve.go.
• All recovery events emit telemetry through internal/telemetry/sink.go for observability and tuning.

Frequently Asked Questions

What file format does the Reasonix recovery system use for checkpoints?

The system persists checkpoints to *.recovery.json side-car files stored alongside the normal session transcript. These JSON files capture the last successful step and pending retry metadata, as implemented in internal/recovery/state.go.

How does the Reasonix recovery system prevent infinite retry loops?

The Gate component in internal/recovery/gate.go enforces budget limits on total retry time and attempt counts. Additionally, the Decision Engine in internal/recovery/decision.go fingerprints failures to detect recurring unresolvable errors, forcing a pause rather than repeated attempts.

When does the Reasonix recovery system escalate to human review?

Escalation occurs when the Gate exhausts its configured budget or when the Decision Engine identifies a failure pattern that exceeds automatic retry limits. At this point, the Reviewer in internal/recovery/reviewer.go surfaces a prompt via the web client (handled in internal/serve/serve.go) requesting user input to continue or modify the session.

Where is the pause notice UI implemented in the Reasonix recovery system?

The user-visible pause notice is implemented in internal/serve/serve.go, which references the recovery_paused state string. When the Gate forces a pause or the Reviewer requires input, this serving layer renders the notice that guides users to continue or adjust their request.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}