Session State Caching Mechanism and lastProcessedIndex in Claude Subconscious

The session state caching mechanism uses a durable JSON file to store lastProcessedIndex, a persistent cursor that tracks which transcript messages have been forwarded to the Letta backend to prevent duplicates across process restarts.

The letta-ai/claude-subconscious repository implements a robust session state caching mechanism to ensure exactly-once delivery of Claude Code transcripts to the Letta backend. At the core of this system is the lastProcessedIndex field, a zero-based integer cursor stored in the filesystem that survives process restarts and multiple hook invocations.

Where the Session State Cache Lives

The cache persists as a JSON file in the project's .letta/claude directory. The file path is generated by getSyncStateFile() in scripts/conversation_utils.ts:

// scripts/conversation_utils.ts#L35-L39
export function getSyncStateFile(cwd: string, sessionId: string): string {
  return path.join(cwd, '.letta', 'claude', `session-${sessionId}.json`);
}

The JSON object conforms to the SyncState interface, which defines the structure of the session state caching mechanism:

export interface SyncState {
  lastProcessedIndex: number;   // Cursor tracking processed messages
  sessionId: string;
  conversationId?: string;
  // …other optional fields
}

Initializing the Cursor on Session Start

When a new Claude Code session begins, scripts/session_start.ts initializes the cache with lastProcessedIndex set to -1, indicating no messages have been processed yet:

// scripts/session_start.ts#L152-L162
const state = {
  sessionId,
  conversationId,
  lastProcessedIndex: -1,
  startedAt: new Date().toISOString(),
};
fs.writeFileSync(getSyncStateFile(cwd, sessionId), JSON.stringify(state, null, 2));

This initialization ensures the session state caching mechanism starts with a clean slate, ready to track progress from the first transcript entry.

Loading and Using the Cursor in the Stop-Hook

Every time the stop-hook (send_messages_to_letta.ts) executes, it loads the cached cursor via loadSyncState():

// scripts/send_messages_to_letta.ts#L66-L71
const state = loadSyncState(cwd, hookInput.session_id, log);

The loadSyncState function in scripts/conversation_utils.ts (lines 54-68) reads the JSON file and returns the stored lastProcessedIndex, which serves as the starting point for filtering.

Filtering Messages by Index

The formatMessagesForLetta() function in scripts/transcript_utils.ts receives the full transcript and the cached startIndex. It iterates only over messages with indices greater than the cursor:

// scripts/transcript_utils.ts#L155-L159
for (let i = startIndex + 1; i < messages.length; i++) {
  // Process only new messages...
}

This logic ensures the session state caching mechanism prevents duplicate forwarding by excluding already-processed entries from the payload sent to Letta.

Updating the Cursor After Successful Delivery

After the Letta SDK successfully receives the messages, the background worker (send_worker_sdk.ts) updates the cursor. The worker writes the new lastProcessedIndex (set to the index of the last message sent) back to the JSON file:

// scripts/send_worker_sdk.ts#L133-L138
state.lastProcessedIndex = payload.newLastProcessedIndex;
fs.writeFileSync(payload.stateFile, JSON.stringify(state, null, 2));

The newLastProcessedIndex value is calculated in the main hook as messages.length - 1 (the newest transcript line index) before spawning the worker (lines 22-24 in send_messages_to_letta.ts).

Practical Code Examples

Reading the Cached State

To manually inspect the current cursor position:

import { loadSyncState } from './scripts/conversation_utils.js';

const cwd = process.cwd();
const sessionId = 'abc123';
const state = loadSyncState(cwd, sessionId);
console.log('Last processed message index:', state.lastProcessedIndex);

Implementing Cursor-Based Filtering

When processing transcripts, use the cursor to extract only new entries:

import { formatMessagesForLetta, loadSyncState, saveSyncState } from './scripts/conversation_utils.js';

const state = loadSyncState(cwd, sessionId);
const newMessages = formatMessagesForLetta(transcript, state.lastProcessedIndex);

if (newMessages.length > 0) {
  // Send to Letta backend...
  // After success:
  state.lastProcessedIndex = transcript.length - 1;
  saveSyncState(cwd, state);
}

Updating State After Batch Processing

To manually advance the cursor after sending messages:

import { loadSyncState, saveSyncState } from './scripts/conversation_utils.js';

const state = loadSyncState(cwd, sessionId);
// After successfully sending messages 0-24:
state.lastProcessedIndex = 24;
saveSyncState(cwd, state);

Summary

• The session state caching mechanism persists progress in .letta/claude/session-<id>.json, enabling durability across process restarts and machine migrations.
• lastProcessedIndex acts as a zero-based cursor initialized to -1, tracking the highest transcript index forwarded to Letta.
• The stop-hook (send_messages_to_letta.ts) loads the cursor and passes it to formatMessagesForLetta(), which filters out indices ≤ lastProcessedIndex.
• The SDK worker (send_worker_sdk.ts) atomically updates the JSON file after successful API delivery, setting lastProcessedIndex to messages.length - 1.
• Exactly-once delivery is guaranteed because each invocation only processes messages with indices greater than the stored cursor.

Frequently Asked Questions

What is the default value of lastProcessedIndex for new sessions?

New sessions initialize lastProcessedIndex to -1 in scripts/session_start.ts (lines 154-162). This sentinel value indicates that no transcript messages have been processed yet, ensuring the first message (index 0) is included in the initial payload.

How does the session state caching mechanism prevent duplicate messages?

The mechanism stores the highest processed transcript index in a durable JSON file. On each hook invocation, formatMessagesForLetta() in scripts/transcript_utils.ts iterates only over messages where i > startIndex (lines 155-159), effectively skipping entries that were already sent in previous runs.

Where is the session state file stored on disk?

The cache resides at <project-root>/.letta/claude/session-<sessionId>.json, generated by getSyncStateFile() in scripts/conversation_utils.ts (lines 35-39). This location ensures the state persists within the project directory and survives across Claude Code sessions.

Which component updates the lastProcessedIndex after sending?

The send_worker_sdk.ts background worker updates the cursor after successfully calling the Letta SDK. It writes the new index to the JSON file (lines 133-138), ensuring the session state caching mechanism reflects the latest delivered message before the next hook execution.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}