How Apache Maka's Runtime Event Log Enables Crash Recovery
Apache Maka persists every Agent Run step to a durable SQLite-based Runtime Event Log, enabling deterministic session restoration through immutable event sourcing and safe-boundary continuation planning.
Apache Maka is an open-source agent runtime that treats durability as a first-class concern. By recording every execution side effect—including function calls, responses, permission requests, and model-visibility flags—to a Runtime Event Log, the system creates a single source of truth that survives process restarts. When crashes occur, Apache Maka leverages this ledger to reconstruct session state without external checkpointing or complex distributed coordination.
The Immutable Ledger Foundation
At the core of Apache Maka's crash recovery capability is an immutable event ledger stored in runtime.sqlite. According to the source code in packages/runtime/src/session-manager.ts, each event appended to this ledger contains an immutable identifier, timestamp, execution side effect, and a model-visibility flag that determines LLM exposure. The packages/runtime/src/runtime-kernel.ts file implements the core execution engine responsible for atomically appending these events, ensuring that every state change is durably recorded before side effects are acknowledged.
The Six-Stage Crash Recovery Pipeline
When a process crashes or is interrupted, Apache Maka executes a deterministic pipeline to restore session integrity. This workflow is implemented across several specialized modules in packages/runtime/src/.
1. Failure Classification and Recovery Decision
The recovery process begins in packages/runtime/src/agent-run-recovery.ts (lines 48-100), where the system inspects the last non-corrupt event to classify the failure type. The logic determines whether the run stopped mid-stream, during a tool call, or while awaiting user permission. If the run status is not terminal, the function returns an AgentRunRecoveryDecision object describing the failure provenance and context required for reconstruction.
2. Back-filling Missing Events
Before reconstruction, packages/runtime/src/runtime-event-backfill.ts (lines 29-64) reconstructs any missing events from stored messages. This back-filling process guarantees a complete chronological view of the run, ensuring that gaps in the SQLite ledger do not compromise recovery accuracy.
3. Terminal Fact Verification
The packages/runtime/src/runtime-event-read-model.ts module (lines 21-38) translates the ledger into a terminal fact (e.g., run_completed) and validates that the stored run header matches the terminal event. This verification step prevents recovery attempts on corrupted or mismatched run states.
4. Safe-Boundary Continuation Planning
In packages/runtime/src/runtime-resume.ts (lines 81-118), the system constructs a SafeBoundaryContinuationPlan by reading the immutable prefix of events. This logic checks workspace identity, tool catalog compatibility, background operations status, and other safety predicates. If all checks pass, it returns a safe_replay disposition; otherwise, the plan is parked pending user resolution.
5. Resume Plan Construction
The buildResumePlanFromRuntimeEvents function, located in packages/runtime/src/runtime-resume.ts (lines 89-106), groups function calls with their corresponding responses and flags indeterminate or corrupted tool results. This function derives rejection reasons that would block automatic replay, ensuring only verifiable event sequences proceed to execution.
6. Continuation Execution
Upon plan approval, the system creates a new continuation run with fresh IDs and a continuation claim that ties the new run to the immutable ledger prefix. As implemented in packages/runtime/src/runtime-resume.ts (lines 81-119), the continuation replays recorded events up to the last safe point, guaranteeing that no side effects are lost or duplicated during recovery.
Implementing Runtime Recovery
Developers can programmatically interact with the recovery system using the RuntimeContinuationPlanner and related utilities from @maka/runtime.
Loading the Ledger and Planning Continuation
The following example demonstrates initializing the SQLite event store and requesting a continuation plan:
import { RuntimeContinuationPlanner } from '@maka/runtime';
import { SqliteRuntimeEventStore } from '@maka/core/runtime-event-store';
const eventStore = new SqliteRuntimeEventStore('/path/to/runtime.sqlite');
const planner = new RuntimeContinuationPlanner({
async readSourceRun(sessionId, runId) { /* … */ },
async readImmutableRuntimePrefix(input) { /* … */ },
async findExistingContinuation(sessionId, sourceRunId, highWater) { /* … */ },
newId: () => crypto.randomUUID(),
});
const plan = await planner.plan({
sessionId: 'sess-123',
sourceRunId: 'run-456',
currentCwd: process.cwd(),
sourceWorkspaceIdentity: 'ws-abc',
currentWorkspaceIdentity: 'ws-abc',
backgroundOperationsSettled: true,
availableToolNames: ['read', 'write', 'bash'],
});
if (plan.disposition === 'continue') {
console.log('Safe to resume – continuation IDs:', plan.continuation?.runId);
} else {
console.warn('Cannot resume automatically:', plan.diagnostics);
}
Building Resume Plans from Raw Events
For scenarios requiring direct event inspection after a crash, use buildResumePlanFromRuntimeEvents:
import { buildResumePlanFromRuntimeEvents } from '@maka/runtime';
import { readAllEvents } from './event-store';
const events = await readAllEvents('sess-123', 'run-456');
const resumePlan = buildResumePlanFromRuntimeEvents(events);
if (resumePlan.disposition === 'safe_replay') {
await runtime.replay(resumePlan.replayRuntimeEvents);
} else {
console.log('Manual inspection required. Diagnostics:', resumePlan.diagnostics);
}
Summary
- Immutable SQLite Ledger: Apache Maka records every execution step in
runtime.sqlitewith unique identifiers and timestamps, creating an append-only log that survives process crashes. - Structured Recovery Pipeline: The system classifies failures in
agent-run-recovery.ts, back-fills missing events, verifies terminal facts, and constructs safe-boundary continuation plans. - Deterministic Replay: The
buildResumePlanFromRuntimeEventsfunction groups related calls and responses to enable idempotent replay without duplicating side effects. - Safety-First Resumption:
runtime-resume.tsvalidates workspace identity, tool availability, and background operation status before allowing automatic continuation, parking unsafe plans for manual review. - Programmatic Control: Developers can implement custom recovery workflows using
RuntimeContinuationPlannerand the back-filling utilities in the core runtime package.
Frequently Asked Questions
What data structure does Apache Maka use for the Runtime Event Log?
Apache Maka uses a SQLite database (runtime.sqlite) as the durable store for the Runtime Event Log. Each row represents an immutable event containing an identifier, timestamp, execution side effect, and model-visibility flag. The session-manager.ts and runtime-kernel.ts modules handle persistence and atomic appends to this ledger.
How does Apache Maka determine if a crashed session can be safely resumed?
The system applies safe-boundary predicates in runtime-resume.ts (lines 81-118). It checks that workspace identities match, required tools remain available in the catalog, background operations have settled, and the event ledger prefix is uncorrupted. Only when all predicates pass does it return a safe_replay disposition; otherwise, the plan is parked for manual resolution.
What happens if the Runtime Event Log is corrupted during a crash?
The runtime-event-read-model.ts module (lines 21-38) performs terminal fact verification to validate that the stored run header matches the terminal event. If corruption is detected—either through mismatched headers or indeterminate tool results flagged by buildResumePlanFromRuntimeEvents—the recovery pipeline prevents automatic replay and surfaces diagnostics for manual inspection.
Can the Runtime Event Log replay events without duplicating side effects?
Yes. The continuation execution mechanism ties new runs to the immutable ledger prefix through continuation claims. By replaying only up to the last safe event boundary and assigning fresh run IDs to the continuation, Apache Maka ensures that previously executed side effects are not re-applied, maintaining exactly-once semantics for critical operations.
Have a question about this repo?
These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:
curl -s "https://instagit.com/install.md" Maintain an open-source project? Get it listed too →