How Craft-Agents Persists Sessions and Stores Message History: A Technical Deep Dive

Craft-Agents persists session data and message history to the local file system using a JSON Lines format within workspace-specific directories, implementing atomic writes and portable paths to ensure data integrity across machine migrations.

Craft-Agents implements a robust, file-based persistence layer that stores every user session directly within the workspace directory. The system uses a specialized JSON Lines format for message history and implements atomic write operations to prevent data corruption during crashes.

Directory Layout and Session Storage Structure

The Sessions Folder Hierarchy

When a workspace initializes, Craft-Agents creates a sessions/ folder at the workspace root. The system generates a unique session identifier for each conversation and creates a dedicated subdirectory structure:

workspace/
└── sessions/
    └── <session-id>/
        ├── session.jsonl
        ├── attachments/
        ├── plans/
        ├── data/
        ├── long_responses/
        └── downloads/

Each subfolder serves a specific purpose. The session.jsonl file contains the session header and complete message history. The attachments/ directory stores binary files uploaded by users or generated by tools. Agent plans reside in plans/, while the data/ folder holds JSON outputs from the transform_data tool. Long responses that were truncated for UI display are preserved in long_responses/, and external downloads (PDFs, images, archives) go into downloads/.

Path Resolution Utilities

The helper functions in packages/shared/src/sessions/storage.ts manage these paths. The ensureSessionDir function creates the directory structure, while getSessionPath and getSessionFilePath generate absolute paths for session files. These utilities ensure consistent path resolution across different operating systems and workspace locations.

Message History Format and JSON Lines Implementation

The session.jsonl File Structure

Craft-Agents stores message history in a JSON Lines format within session.jsonl. This format provides several advantages: append-only writes, easy parsing of partial files, and resilience against corruption.

The file structure follows a strict line-based protocol:

{"id":"260411-bright-river","workspacePath":"/Users/me/my-workspace",...}  // Line 1: SessionHeader
{"id":"msg-1","type":"user","timestamp":1234567890,"content":"Hello"}       // Line 2+: StoredMessage
{"id":"msg-2","type":"assistant","timestamp":1234567891,"content":"Hi there"}

The first line contains the SessionHeader with metadata including the session ID, workspace path, creation timestamps, current status, and token usage statistics. Subsequent lines contain StoredMessage objects with fields for message type, unique identifier, content, and timestamps.

Reading and Writing JSON Lines

The packages/shared/src/sessions/jsonl.ts file implements the read/write logic. For performance optimization, the readSessionHeader function reads only the first 8KB of the file to extract metadata without parsing the entire message history. This enables fast session list rendering in the UI.

When loading complete sessions, readSessionJsonl expands portable path tokens, parses the header, and processes each message line using parseMessagesResilient. This resilient parsing tolerates partially written lines that might occur during system crashes, ensuring that valid messages are recovered even if the last write was interrupted.

Atomic Persistence and Portable Paths

The Atomic Write Pattern

Craft-Agents implements atomic file writes to prevent session corruption. The writeSessionJsonl function in packages/shared/src/sessions/jsonl.ts performs writes using a temporary file and rename operation:

1. The system writes the header and all messages to a temporary file with a .tmp extension
2. Once the write completes successfully, the function atomically renames the temporary file over the existing session.jsonl
3. This ensures that readers never see partially written data; they either see the complete old file or the complete new file

Portable Path Tokens

To enable session portability across different machines, Craft-Agents converts absolute paths to portable tokens before writing. The makeSessionPathPortable function replaces the session's absolute directory path with the token {{SESSION_PATH}}}.

When loading sessions, expandSessionPath resolves these tokens back to absolute paths based on the current workspace location. This mechanism allows users to copy session directories between computers or share them via version control without path breakage.

The Persistence Queue

The SessionPersistenceQueue singleton in packages/shared/src/sessions/persistence-queue.ts manages write operations to prevent race conditions and optimize disk I/O. The queue implements debouncing for rapid successive updates, coalescing multiple save requests into a single write operation.

The queue maintains a metadata signature to detect external modifications. If the session.jsonl file changes on disk between the queue's read and write operations (for example, through a configuration watcher), the system reconciles the changes to prevent data loss.

Practical Code Examples

Creating a New Session

import { createSession } from '@/packages/shared/src/sessions/storage.ts';

const newSession = await createSession('/Users/me/my-workspace', {
  name: 'Research on AI',
  workingDirectory: '/Users/me/my-workspace',
});

console.log('Session ID:', newSession.id);

Loading an Existing Session and Reading Message History

import { loadSession } from '@/packages/shared/src/sessions/storage.ts';

const stored = loadSession('/Users/me/my-workspace', '260411-bright-river');

if (stored) {
  console.log('Header preview:', stored.preview);
  console.log('Messages count:', stored.messages.length);
}

Appending a New Message and Persisting Changes

import { loadSession, saveSession } from '@/packages/shared/src/sessions/storage.ts';
import type { StoredMessage } from '@/packages/shared/src/sessions/types.ts';

const sess = loadSession('/Users/me/my-workspace', '260411-bright-river');

if (sess) {
  const msg: StoredMessage = {
    id: crypto.randomUUID(),
    type: 'user',
    timestamp: Date.now(),
    content: 'Can you summarise the latest article?',
  };
  
  sess.messages.push(msg);
  await saveSession(sess);   // goes through the persistence queue
}

Deleting a Session

import { deleteSession } from '@/packages/shared/src/sessions/storage.ts';

const ok = deleteSession('/Users/me/my-workspace', '260411-bright-river');
console.log('Deleted?', ok);

Summary

• Craft-Agents persists sessions to the local file system using a workspace-scoped sessions/ directory that contains individual subfolders for each conversation.
• Message history is stored in session.jsonl using a JSON Lines format where line 1 contains the session header and subsequent lines contain individual messages.
• Atomic writes prevent corruption by writing to temporary files and performing atomic renames, ensuring readers never see partially written data.
• Portable paths using the {{SESSION_PATH}} token enable sessions to be moved between machines without breaking file references.
• SessionPersistenceQueue debounces rapid saves and tracks metadata signatures to prevent race conditions and external modification conflicts.

Frequently Asked Questions

How does Craft-Agents prevent data loss if the application crashes during a save?

The system implements atomic file writes in packages/shared/src/sessions/jsonl.ts. When saving, it writes the entire session data to a temporary .tmp file first, then performs an atomic rename operation to replace the existing session.jsonl. This ensures that the file is never in a partially written state; readers either see the complete old version or the complete new version.

Can I move a Craft-Agents session to another computer?

Yes. Craft-Agents uses portable path tokens to ensure session portability. Before writing to disk, absolute paths are converted to the {{SESSION_PATH}} token via makeSessionPathPortable in packages/shared/src/sessions/jsonl.ts. When loading on a new machine, the system expands these tokens based on the current workspace location, allowing attachments and file references to resolve correctly regardless of the absolute path differences between machines.

What is the SessionPersistenceQueue and why is it necessary?

The SessionPersistenceQueue in packages/shared/src/sessions/persistence-queue.ts is a singleton that manages all disk write operations for sessions. It prevents race conditions by serializing write requests and optimizes performance by debouncing rapid successive updates into single atomic writes. The queue also tracks metadata signatures to detect when external processes (like file watchers) modify the session file, preventing accidental overwrites of external changes.

How does Craft-Agents handle large message histories efficiently?

For operations that only need session metadata (such as rendering a session list), Craft-Agents uses the readSessionHeader function in packages/shared/src/sessions/jsonl.ts to read only the first 8KB of the file, parsing just the header line without loading the entire message history. For full session loads, the parseMessagesResilient function tolerates partially written lines, ensuring that valid messages are recovered even if the last write operation was interrupted.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}