Crash Recovery in Apache Maka: Event Sourcing and Atomic Storage Guarantees

Apache Maka guarantees zero data loss during crashes by combining an immutable append-only event log with atomic SQLite transactions and file-system locks, allowing the system to replay events and discard incomplete writes on restart.

Apache Maka is an open-source application framework built on the principle that user work must survive any failure mode, from graceful shutdowns to sudden power losses. The project's crash recovery architecture leverages event sourcing and transactional storage to ensure that every user action, tool invocation, and permission decision remains intact even when the underlying process terminates unexpectedly. This design is implemented across the packages/storage layer and documented in the project's architecture specifications.

The RuntimeEvent Log: Immutable Source of Truth

At the heart of Maka's resilience lies an append-only event log that treats every state change as a RuntimeEvent. According to ARCHITECTURE.md, the UI and all projections—such as the chat view, code editor, and permission panels—are not persisted directly. Instead, they are re-computed from the immutable log each time the application starts.

This event-sourcing pattern ensures that the storage layer maintains a single source of truth. When a crash occurs, the system does not attempt to repair complex in-memory state; it simply replays the log from the beginning to reconstruct the exact user context. The message-admission-store.ts file serves as the central hub for persisting these RuntimeEvents, making it the primary recovery artifact.

Atomic Storage and Lock-Based Consistency

The storage layer implements transactional durability using SQLite combined with advisory file-system locks. In packages/storage/src/model-call-ledger.ts, the code handles scenarios where "a failed upsert, a crash between the two – and that is recoverable." This is achieved by ensuring that writes are atomic: either the entire transaction commits to disk or it leaves no trace.

To detect incomplete transactions, Maka uses lock files. As noted in packages/storage/src/credential-store.ts, a "hard crash (SIGKILL / power loss) mid-write leaves the lock." When the application restarts, the presence of this stale lock signals that the previous process did not complete cleanly. The system then discards any half-written journal files and re-applies the event log from the last known good state.

The Crash Recovery Sequence Step-by-Step

When Apache Maka restarts after a failure, it executes a deterministic four-phase recovery protocol:

1. Detect stale locks. The storage layer scans for *.lock files left by crashed processes, implemented in process-lifetime-file-update-lock.ts.

2. Discard incomplete writes. Any journal files that were not fully committed are ignored, and the lock marker is removed. This logic is verified in packages/storage/src/__tests__/file-session-repository.test.ts, which specifically tests the crash-point "before-rename" scenario.

3. Replay the event log. The immutable RuntimeEvent stream is read from message-admission-store.ts and replayed to rebuild the UI state exactly as it existed before the crash.

4. Resume normal operation. After state reconstruction, a fresh lock is acquired and the application continues. The cleanup logic ensures no resource leaks persist from the previous session, as validated in session-copy-cleanup.test.ts.

Testing Crash Recovery Boundaries with Real-Process Injection

Maka validates its recovery guarantees through an aggressive test suite that injects real-process crashes at precise execution points. The file packages/storage/src/__tests__/sqlite-runtime-crash.test.ts contains tests labeled "SqliteRuntimeStore real-process crash boundaries" that spawn child processes, instruct them to exit abruptly at specific fail-points (such as after-rename), and verify that the parent process can recover correctly.

Similarly, packages/storage/src/__tests__/session-bundle-file-service.test.ts ensures that "a process crash leaves owned staging that can be reclaimed without touching decoys." These tests simulate hard failures by creating orphaned lock files and verifying that the recovery logic correctly identifies and cleans them up without affecting legitimate data.

Practical Implementation Examples

The following patterns demonstrate how to interact with Maka's crash-recovery mechanisms programmatically.

Simulating a Crash and Verifying Recovery

This snippet shows how the test suite spawns a child process that crashes at a controlled fail-point, then validates that the storage layer recovers the event log:

import { spawn } from 'child_process';
import { SessionRepository } from '@maka/storage';
import { assert } from 'node:assert';

// 1️⃣ Start child that crashes after writing a RuntimeEvent
const child = spawn('node', [
  './fixtures/file-session-repository-crash-writer.ts',
  repoRoot,
  'input.json',
  'after-rename'   // fail-point where child exits abruptly
]);

// 2️⃣ Wait for child to signal it reached the fail-point
const crashPoint = await new Promise<string>((resolve, reject) => {
  child.on('message', resolve);
  child.on('error', reject);
});

assert.equal(crashPoint, 'after-rename');

// 3️⃣ Parent opens repository; detects stale lock and recovers
const repo = await SessionRepository.open(repoRoot);

// 4️⃣ Verify recovery of previously written event
const events = await repo.readAllEvents();
assert.ok(events.some(e => 
  e.type === 'userMessage' && e.content === 'Hello world'
));

Recovering from a Hard Crash in Credential Store

When a crash leaves an orphaned lock, the storage layer automatically cleans it up on the next initialization:

import { CredentialStore } from '@maka/storage';
import { mkdir } from 'fs/promises';

// Simulate crash by creating lock that will never be released
await mkdir(`${storePath}.lock`);   // mimics a crashed holder

// On next startup, store detects lock and recovers automatically
const store = await CredentialStore.open(storePath);
await store.put('apiKey', '******');   // normal operation resumes

These examples illustrate two-phase recovery: first cleaning up stale artifacts from the crashed process, then replaying the immutable log to reconstruct state.

Summary

• Apache Maka uses an append-only RuntimeEvent log as the single source of truth, allowing complete state reconstruction after any crash.
• Atomic writes and file-system locks in the packages/storage layer ensure that incomplete transactions are detected and discarded, as implemented in model-call-ledger.ts and credential-store.ts.
• The recovery sequence follows four deterministic phases: stale lock detection, incomplete write cleanup, event log replay, and normal operation resumption.
• Real-process crash injection tests in files like sqlite-runtime-crash.test.ts validate that the system survives SIGKILL, power loss, and mid-write failures without data loss.
• Developers interact with these guarantees through the standard storage APIs, which automatically handle lock cleanup and log replay on initialization.

Frequently Asked Questions

How does Apache Maka detect that a previous session crashed?

The storage layer scans for advisory lock files (e.g., *.lock) on startup. If a lock file exists from a previous process that no longer holds the PID, the system identifies this as a stale lock indicating an unclean shutdown, triggering the recovery protocol in process-lifetime-file-update-lock.ts.

What happens to data that was being written during the crash?

Any in-flight writes are contained within SQLite transactions. If a hard crash occurs mid-write, the transaction never commits, and the associated journal files are discarded during the recovery sequence. The system then replays the immutable RuntimeEvent log from message-admission-store.ts to restore the last committed state.

Can Apache Maka recover from a SIGKILL or power loss?

Yes. The architecture specifically handles hard crashes such as SIGKILL or power failures. By relying on atomic file operations and the append-only event log rather than in-memory state, Maka ensures that no committed data is lost and that partial writes are automatically invalidated and cleaned up on restart.

How does the test suite verify crash recovery without corrupting the development environment?

The test suite uses child process isolation to simulate crashes. Tests in sqlite-runtime-crash.test.ts spawn separate Node.js processes that exit at specific fail-points (like after-rename), leaving behind realistic crash artifacts. The parent process then validates that it can open the repository and recover correctly, ensuring the crash recovery logic works in real-world conditions without risking the parent test runner.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}