How Maka Handles Storage and SQLite Control Planes in `packages/storage`

Maka separates storage management into a storage-root authority that guards directory trees via capabilities and leases, and a SQLite control plane that manages per-root databases with transactional safety, schema migrations, and strict PRAGMA-based access controls.

Apache Maka’s packages/storage module implements a dual-layer architecture that isolates directory-level authority from database persistence. The design centers on a storage-root control plane that validates filesystem identity and enforces exclusive access, while a SQLite control plane handles artifact storage with deterministic schema versioning and atomic transactions.

Storage-Root Authority and Control Plane

The storage-root authority in src/root-authority.ts establishes the foundation of Maka’s storage subsystem. It manages the lifecycle of a storage root directory through marker files, capability objects, and filesystem locks.

Root Discovery and Marker Initialization

When initializing storage, discoverMarkedStorageRoot and resolveStorageRoot locate or create a target directory and generate a hidden marker file .maka-storage-root.json. This marker contains a random rootId plus host-local device and inode identifiers (dev/ino), allowing the system to detect filesystem remounts or corruption. According to the source code, this marker creation and validation logic resides at lines 13-28 of root-authority.ts.

Capabilities and Lease-Based Access

Access to a storage root is mediated through two distinct objects:

• StorageRootCapability: A read-only reference that can be safely shared between components.
• StorageRootLease: Grants exclusive read or write access and binds to a physical lock file (owner.lock) in the control directory.

Control Directory and Filesystem Locking

The prepareStorageRootControlDirectory function creates a private control directory (typically under ~/.cache/maka/runtime-hosts/<rootId> on Linux) with strict POSIX permissions (mode 0o700). When acquiring a lease, acquireStateRootLock uses fs-native-extensions to perform advisory locking on owner.lock. The function assertStableLockArtifact verifies inode and device consistency before and after locking to prevent time-of-check-to-time-of-use (TOCTOU) attacks, as implemented at lines 126-144 of root-authority.ts.

SQLite Control Plane Implementation

While the storage-root authority guards the directory, the SQLite control plane manages actual data persistence through separate database files (e.g., runtime.sqlite, memory.sqlite) located within the root.

Database Initialization and PRAGMA Configuration

Each store follows a consistent initialization pattern in files like src/sqlite-runtime-store.ts. The loadDatabaseSync function lazily loads node:sqlite and creates a DatabaseSync instance. For writable stores, configureSqliteRuntimeDatabase enables foreign keys, sets busy timeouts, and executes schema migrations via migrateSqliteRuntimeDatabase (lines 38-42).

Schema Versioning and Migrations

Every SQLite store embeds a schema version constant (e.g., SQLITE_RUNTIME_SCHEMA_VERSION). During initialization, the store checks the user_version PRAGMA and executes migration scripts if the database version lags behind the expected constant. This ensures that runtime events, long-term memory, and usage data tables remain compatible across Maka updates.

Read-Only Safety and Transactional Integrity

When opened with { readOnly: true }, stores activate PRAGMA query_only = ON and PRAGMA foreign_keys = ON, preventing accidental writes while maintaining referential integrity. All mutations occur within this.transaction(() => …) blocks, providing atomicity and a single injection point for failure testing. The store exposes high-level methods like appendRuntimeEvent and readImmutableRuntimeEvents, internally using prepared statements (db.prepare(...)) and typed row decoders.

Bridging the Control and Data Planes

The two planes interact through lease coupling. An InteractiveRootOwner first obtains a write lease on the storage root, then passes the lease's controlDirectory to SQLite stores. When a store receives a databaseLease option, it reuses the already-opened SQLite connection from the lease rather than opening a new file. This ties the database lifecycle to the storage-root lock, ensuring exclusive filesystem access persists for the duration of database operations (lines 65-70 in sqlite-runtime-store.ts).

Safety Guarantees and Security Mechanisms

Maka enforces several critical invariants through the following mechanisms:

• Root Integrity: assertRootPathIdentity and confirmRootSnapshot compare live filesystem stats against the .maka-storage-root.json marker before operations, detecting remounts or tampering.
• Directory Privacy: ensurePrivateDirectory enforces mode 0o700 and ownership checks, aborting with insecure_control_directory errors if permissions are violated.
• Lock Stability: The system verifies lock file inode/device consistency before and after locking to prevent TOCTOU vulnerabilities.
• Deterministic Repair: If filesystem identity changes (e.g., after backup restoration), replaceRootMarkerIdentity updates the marker at lines 266-280 of root-authority.ts without invalidating stored authority.

Working with Maka Storage APIs

The following examples demonstrate acquiring a storage root lease and interacting with the SQLite runtime store.

// Acquire a writable storage-root lease
import { resolveStorageRoot, tryAcquireInteractiveRootOwner } from '@maka/storage';

const capability = await resolveStorageRoot({ path: '/tmp/maka-root', kind: 'interactive' });
const owner = await tryAcquireInteractiveRootOwner(capability);
if (!owner) throw new Error('Unable to lock storage root');

// Open the runtime SQLite store using the lease
import { createSqliteRuntimeStore } from '@maka/storage';

const runtimeStore = createSqliteRuntimeStore(
  '/tmp/maka-root/runtime.sqlite',
  { databaseLease: owner },          // Ties the store to the lease
);

// Append a runtime event
await runtimeStore.appendRuntimeEvent('session-1', 'run-a', {
  id: 'evt-001',
  sessionId: 'session-1',
  runId: 'run-a',
  ts: Date.now(),
  payload: { /* … */ },
});

For read operations with memory protection:

// Read immutable runtime events with a budget (protects against OOM)
import { readRuntimeEventsBounded } from '@maka/storage';

const result = await runtimeStore.readRuntimeEventsBounded('session-1', 'run-a', {
  maxRecords: 1000,
  maxPartialRecords: 200,
  maxRecordBytes: 64 * 1024,
  maxPartialBytes: 1 * 1024 * 1024,
});

if (result.status === 'complete') {
  console.log('Fetched', result.records.length, 'events');
}

Summary

• Maka’s packages/storage separates directory authority (storage-root control plane) from data persistence (SQLite control plane) to create a robust, deterministic storage layer.
• The storage-root authority in src/root-authority.ts manages filesystem identity through .maka-storage-root.json markers and enforces exclusive access via StorageRootLease objects and POSIX advisory locks.
• SQLite stores in src/sqlite-runtime-store.ts provide transactional artifact persistence with automatic schema migrations, strict PRAGMA-based access controls, and support for both read-only and read-write modes.
• The two planes integrate through lease coupling, where SQLite stores reuse connections from active storage-root leases, ensuring filesystem locks remain held during database operations.
• Security mechanisms include root identity verification, private control directories (mode 0o700), TOCTOU-resistant locking, and deterministic repair of storage markers after filesystem changes.

Frequently Asked Questions

What is the purpose of the .maka-storage-root.json marker file?

The marker file stores the rootId and host-local filesystem identifiers (dev/ino) that allow Maka to verify the storage root's integrity before operations. If the underlying filesystem changes (e.g., remount or restore), functions like assertRootPathIdentity detect the mismatch, and replaceRootMarkerIdentity can repair the marker without losing authority.

How does Maka prevent concurrent access to the SQLite databases?

Maka uses a lease-based locking system where tryAcquireInteractiveRootOwner acquires an exclusive lock on owner.lock in the control directory. SQLite stores then receive this lease via the databaseLease option, coupling the database connection lifecycle to the filesystem lock. This ensures that only one process holds the SQLite database open for writes at any time.

What happens if the SQLite schema version doesn't match the expected version?

When opening a store, Maka checks the user_version PRAGMA against the embedded SQLITE_*_SCHEMA_VERSION constant. If the database version is older, the system automatically runs migration functions (e.g., migrateSqliteRuntimeDatabase) to upgrade the schema. Read-only mode enforces exact version matching, returning errors if the database is incompatible with the current code.

Why does Maka use a separate control directory outside the storage root?

The control directory (located in the user's cache directory, e.g., ~/.cache/maka/runtime-hosts/<rootId>) contains lock files and bootstrap artifacts with strict permissions (0o700). This separation allows Maka to enforce POSIX ownership and locking guarantees even when the storage root resides on shared or network filesystems that might not support reliable advisory locking or permission isolation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}