# How the Session Persistence Layer Works with JSONL Storage in craft-agents-oss

> Discover how the session persistence layer uses JSONL storage in craft-agents-oss for efficient and safe conversation saving. Learn about atomic writes and fast metadata listing.

- Repository: [Craft Ai Agents/craft-agents-oss](https://github.com/craft-ai-agents/craft-agents-oss)
- Tags: internals
- Published: 2026-07-03

---

**The session persistence layer stores each conversation as a newline-delimited JSON (JSONL) file with a metadata header on line 1 and one message per subsequent line, using an atomic-write queue to prevent corruption while supporting fast metadata-only listing.**

The session persistence layer in the [craft-ai-agents/craft-agents-oss](https://github.com/craft-ai-agents/craft-agents-oss) repository implements a crash-safe storage mechanism using JSONL (JSON Lines) format. Each session lives in its own directory under the workspace's `sessions/` folder, with a single `session.jsonl` file containing both pre-computed metadata and message history. This architecture enables the UI to list hundreds of sessions quickly by reading only the first line of each file, while lazy-loading full conversation data only when needed.

## Session Layout on Disk

When a session is created via `createSession` in [`packages/shared/src/sessions/storage.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/sessions/storage.ts), the system creates a dedicated folder structure:

```

{workspaceRoot}/sessions/
└─ {sessionId}/
   ├─ session.jsonl          ← Header + messages
   ├─ attachments/
   ├─ plans/
   ├─ data/
   └─ downloads/

```

The `session.jsonl` file follows a strict two-part structure:

- **Line 1**: A `SessionHeader` object containing metadata and pre-computed fields for fast UI rendering
- **Line 2+**: One `StoredMessage` object per line

The header is generated by `createSessionHeader` in [`packages/shared/src/sessions/jsonl.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/sessions/jsonl.ts) and includes fields such as `id`, `workspaceRootPath`, `name`, `labels`, `sessionStatus`, and `permissionMode`. Critically, it also stores pre-computed values like `messageCount`, `lastMessageRole`, `preview`, `tokenUsage`, and `lastFinalMessageId`. These fields allow `listSessions` to display session summaries without parsing the entire message history.

## Writing to Disk: The Persistence Queue

All write operations flow through the `sessionPersistenceQueue` in [`packages/shared/src/sessions/persistence-queue.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/sessions/persistence-queue.ts). When `saveSession` is called, it invokes `sessionPersistenceQueue.enqueue(session)`, which implements four key safety mechanisms:

1. **Debouncing**: A 500ms timer resets on each enqueue, coalescing rapid updates into a single write operation.

2. **Per-session serialization**: The `writeInProgress` flag ensures that concurrent flushes for the same session execute in order.

3. **External change detection**: The queue computes a metadata signature using `getHeaderMetadataSignature` before writing. If the on-disk header diverges from the last-written signature (indicating an external edit), the system merges changes back via `mergeHeaderWithExternalMetadata` to prevent clobbering file-watcher updates.

4. **Atomic writes**: The new content writes to `session.jsonl.tmp`, then moves into place via an atomic rename:

```typescript
// From packages/shared/src/sessions/persistence-queue.ts
await writeFile(tmpFile, lines.join('\n') + '\n', 'utf-8');
await unlink(filePath);          // Windows requires explicit removal
await rename(tmpFile, filePath); // Atomic replace

```

If the process crashes mid-write, the original `session.jsonl` remains untouched. The signature is stored in `lastWrittenHeaderSignature` after the rename, allowing the file-system watcher to ignore self-generated events.

## Reading: Fast Headers and Lazy Messages

The storage layer optimizes read performance through three distinct access patterns:

**`listSessions`** calls `readSessionHeader` on each `session.jsonl` file, reading only the first ~8 KB of data. This provides instant directory scanning without loading message histories.

**`loadSession`** uses `readSessionJsonl` to parse the entire file, returning both the header and all messages as a complete `StoredSession` object.

**`readSessionMessages`** skips the header line and parses only message lines, supporting lazy loading when a user selects a specific conversation in the UI.

Both reading utilities automatically expand portable path tokens via `expandSessionPath`, converting `{{SESSION_PATH}}` placeholders back to the absolute workspace path.

## End-to-End Usage Example

```typescript
import {
  createSession,
  loadSession,
  saveSession,
  listSessions,
} from './packages/shared/src/sessions/storage.ts';

// 1. Create a new session
const cfg = await createSession('/my/workspace', { name: 'Demo' });

// 2. Mutate the session
cfg.messages.push({
  id: 'msg-1',
  type: 'user',
  content: 'Hello, world!',
  createdAt: Date.now(),
});

// 3. Persist through the queue
await saveSession(cfg);

// 4. Load full session later
const full = loadSession('/my/workspace', cfg.id);
console.log(full?.messages[0].content); // → "Hello, world!"

// 5. List all sessions (fast metadata-only)
const all = listSessions('/my/workspace');
console.log(all.map(s => s.name));

```

## Why JSONL Matters

The JSONL format provides specific advantages for session persistence:

- **Line-oriented recovery**: Each line is a valid JSON value, making it possible to recover partially corrupted files by truncating damaged lines.
- **Header-first performance**: The UI can list sessions by reading only the first line, avoiding the overhead of parsing entire conversation histories.
- **Atomic consistency**: Because the header and messages write together in a single atomic operation (via temp file and rename), the file is always in a coherent state—either the old version or the new version, never a mix.

## Summary

- **File Location**: Each session resides in `{workspaceRoot}/sessions/{sessionId}/session.jsonl` as defined in [`packages/shared/src/sessions/storage.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/sessions/storage.ts).
- **Atomic Writes**: The persistence queue in [`packages/shared/src/sessions/persistence-queue.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/sessions/persistence-queue.ts) writes to a temporary file and renames it, ensuring crash-safe operations.
- **Conflict Detection**: Metadata signatures prevent the queue from overwriting external changes made by file watchers.
- **Performance Optimization**: Pre-computed header fields enable `listSessions` to scan directories without loading full message histories.
- **Portable Paths**: The system stores relative path tokens (`{{SESSION_PATH}}`) and expands them on read via `expandSessionPath` in [`packages/shared/src/sessions/jsonl.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/sessions/jsonl.ts).

## Frequently Asked Questions

### How does the persistence layer prevent data loss during crashes?

The system writes to a temporary file (`session.jsonl.tmp`) first, then performs an atomic rename to replace the original file. If the process crashes during the write operation, the original `session.jsonl` remains intact and uncorrupted. This pattern is implemented in [`packages/shared/src/sessions/persistence-queue.ts`](https://github.com/craft-ai-agents/craft-agents-oss/blob/main/packages/shared/src/sessions/persistence-queue.ts).

### What is the purpose of the metadata signature in the persistence queue?

The metadata signature, generated by `getHeaderMetadataSignature`, creates a hash of UI-relevant header fields before writing. When the queue prepares to flush updates, it compares the current on-disk signature against `lastWrittenHeaderSignature`. If they differ, the system detects external modifications and merges them via `mergeHeaderWithExternalMetadata` rather than overwriting them.

### How does the system handle external file modifications?

When the persistence queue detects a signature mismatch (indicating the file changed outside the application), it invokes `mergeHeaderWithExternalMetadata` to reconcile the external changes with the in-memory session state. This ensures that edits made by file watchers, git operations, or other processes are preserved rather than clobbered by the application's next write.

### Why are pre-computed fields stored in the JSONL header?

Fields like `messageCount`, `lastMessageRole`, and `tokenUsage` are stored in the header to enable `listSessions` to render the session list without parsing every message in every file. By reading only the first line of each `session.jsonl` (approximately 8 KB), the UI achieves fast directory scans even with hundreds of sessions.