How the Session Persistence Layer Works with JSONL Storage in craft-agents-oss

The session persistence layer stores each conversation as a newline-delimited JSON (JSONL) file with a metadata header on line 1 and one message per subsequent line, using an atomic-write queue to prevent corruption while supporting fast metadata-only listing.

The session persistence layer in the craft-ai-agents/craft-agents-oss repository implements a crash-safe storage mechanism using JSONL (JSON Lines) format. Each session lives in its own directory under the workspace's sessions/ folder, with a single session.jsonl file containing both pre-computed metadata and message history. This architecture enables the UI to list hundreds of sessions quickly by reading only the first line of each file, while lazy-loading full conversation data only when needed.

Session Layout on Disk

When a session is created via createSession in packages/shared/src/sessions/storage.ts, the system creates a dedicated folder structure:

{workspaceRoot}/sessions/
└─ {sessionId}/
   ├─ session.jsonl          ← Header + messages
   ├─ attachments/
   ├─ plans/
   ├─ data/
   └─ downloads/

The session.jsonl file follows a strict two-part structure:

• Line 1: A SessionHeader object containing metadata and pre-computed fields for fast UI rendering
• Line 2+: One StoredMessage object per line

The header is generated by createSessionHeader in packages/shared/src/sessions/jsonl.ts and includes fields such as id, workspaceRootPath, name, labels, sessionStatus, and permissionMode. Critically, it also stores pre-computed values like messageCount, lastMessageRole, preview, tokenUsage, and lastFinalMessageId. These fields allow listSessions to display session summaries without parsing the entire message history.

Writing to Disk: The Persistence Queue

All write operations flow through the sessionPersistenceQueue in packages/shared/src/sessions/persistence-queue.ts. When saveSession is called, it invokes sessionPersistenceQueue.enqueue(session), which implements four key safety mechanisms:

1. Debouncing: A 500ms timer resets on each enqueue, coalescing rapid updates into a single write operation.

2. Per-session serialization: The writeInProgress flag ensures that concurrent flushes for the same session execute in order.

3. External change detection: The queue computes a metadata signature using getHeaderMetadataSignature before writing. If the on-disk header diverges from the last-written signature (indicating an external edit), the system merges changes back via mergeHeaderWithExternalMetadata to prevent clobbering file-watcher updates.

4. Atomic writes: The new content writes to session.jsonl.tmp, then moves into place via an atomic rename:

// From packages/shared/src/sessions/persistence-queue.ts
await writeFile(tmpFile, lines.join('\n') + '\n', 'utf-8');
await unlink(filePath);          // Windows requires explicit removal
await rename(tmpFile, filePath); // Atomic replace

If the process crashes mid-write, the original session.jsonl remains untouched. The signature is stored in lastWrittenHeaderSignature after the rename, allowing the file-system watcher to ignore self-generated events.

Reading: Fast Headers and Lazy Messages

The storage layer optimizes read performance through three distinct access patterns:

listSessions calls readSessionHeader on each session.jsonl file, reading only the first ~8 KB of data. This provides instant directory scanning without loading message histories.

loadSession uses readSessionJsonl to parse the entire file, returning both the header and all messages as a complete StoredSession object.

readSessionMessages skips the header line and parses only message lines, supporting lazy loading when a user selects a specific conversation in the UI.

Both reading utilities automatically expand portable path tokens via expandSessionPath, converting {{SESSION_PATH}} placeholders back to the absolute workspace path.

End-to-End Usage Example

import {
  createSession,
  loadSession,
  saveSession,
  listSessions,
} from './packages/shared/src/sessions/storage.ts';

// 1. Create a new session
const cfg = await createSession('/my/workspace', { name: 'Demo' });

// 2. Mutate the session
cfg.messages.push({
  id: 'msg-1',
  type: 'user',
  content: 'Hello, world!',
  createdAt: Date.now(),
});

// 3. Persist through the queue
await saveSession(cfg);

// 4. Load full session later
const full = loadSession('/my/workspace', cfg.id);
console.log(full?.messages[0].content); // → "Hello, world!"

// 5. List all sessions (fast metadata-only)
const all = listSessions('/my/workspace');
console.log(all.map(s => s.name));

Why JSONL Matters

The JSONL format provides specific advantages for session persistence:

• Line-oriented recovery: Each line is a valid JSON value, making it possible to recover partially corrupted files by truncating damaged lines.
• Header-first performance: The UI can list sessions by reading only the first line, avoiding the overhead of parsing entire conversation histories.
• Atomic consistency: Because the header and messages write together in a single atomic operation (via temp file and rename), the file is always in a coherent state—either the old version or the new version, never a mix.

Summary

• File Location: Each session resides in {workspaceRoot}/sessions/{sessionId}/session.jsonl as defined in packages/shared/src/sessions/storage.ts.
• Atomic Writes: The persistence queue in packages/shared/src/sessions/persistence-queue.ts writes to a temporary file and renames it, ensuring crash-safe operations.
• Conflict Detection: Metadata signatures prevent the queue from overwriting external changes made by file watchers.
• Performance Optimization: Pre-computed header fields enable listSessions to scan directories without loading full message histories.
• Portable Paths: The system stores relative path tokens ({{SESSION_PATH}}) and expands them on read via expandSessionPath in packages/shared/src/sessions/jsonl.ts.

Frequently Asked Questions

How does the persistence layer prevent data loss during crashes?

The system writes to a temporary file (session.jsonl.tmp) first, then performs an atomic rename to replace the original file. If the process crashes during the write operation, the original session.jsonl remains intact and uncorrupted. This pattern is implemented in packages/shared/src/sessions/persistence-queue.ts.

What is the purpose of the metadata signature in the persistence queue?

The metadata signature, generated by getHeaderMetadataSignature, creates a hash of UI-relevant header fields before writing. When the queue prepares to flush updates, it compares the current on-disk signature against lastWrittenHeaderSignature. If they differ, the system detects external modifications and merges them via mergeHeaderWithExternalMetadata rather than overwriting them.

How does the system handle external file modifications?

When the persistence queue detects a signature mismatch (indicating the file changed outside the application), it invokes mergeHeaderWithExternalMetadata to reconcile the external changes with the in-memory session state. This ensures that edits made by file watchers, git operations, or other processes are preserved rather than clobbered by the application's next write.

Why are pre-computed fields stored in the JSONL header?

Fields like messageCount, lastMessageRole, and tokenUsage are stored in the header to enable listSessions to render the session list without parsing every message in every file. By reading only the first line of each session.jsonl (approximately 8 KB), the UI achieves fast directory scans even with hundreds of sessions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}