# Session State Caching Mechanism and lastProcessedIndex in Claude Subconscious

> Discover how Claude Subconscious uses session state caching with a durable JSON file to store lastProcessedIndex, preventing duplicate transcript messages across restarts.

- Repository: [Letta/claude-subconscious](https://github.com/letta-ai/claude-subconscious)
- Tags: internals
- Published: 2026-03-26

---

**The session state caching mechanism uses a durable JSON file to store `lastProcessedIndex`, a persistent cursor that tracks which transcript messages have been forwarded to the Letta backend to prevent duplicates across process restarts.**

The `letta-ai/claude-subconscious` repository implements a robust session state caching mechanism to ensure exactly-once delivery of Claude Code transcripts to the Letta backend. At the core of this system is the `lastProcessedIndex` field, a zero-based integer cursor stored in the filesystem that survives process restarts and multiple hook invocations.

## Where the Session State Cache Lives

The cache persists as a JSON file in the project's `.letta/claude` directory. The file path is generated by `getSyncStateFile()` in [`scripts/conversation_utils.ts`](https://github.com/letta-ai/claude-subconscious/blob/main/scripts/conversation_utils.ts):

```typescript
// scripts/conversation_utils.ts#L35-L39
export function getSyncStateFile(cwd: string, sessionId: string): string {
  return path.join(cwd, '.letta', 'claude', `session-${sessionId}.json`);
}

```

The JSON object conforms to the `SyncState` interface, which defines the structure of the session state caching mechanism:

```typescript
export interface SyncState {
  lastProcessedIndex: number;   // Cursor tracking processed messages
  sessionId: string;
  conversationId?: string;
  // …other optional fields
}

```

## Initializing the Cursor on Session Start

When a new Claude Code session begins, [`scripts/session_start.ts`](https://github.com/letta-ai/claude-subconscious/blob/main/scripts/session_start.ts) initializes the cache with `lastProcessedIndex` set to **-1**, indicating no messages have been processed yet:

```typescript
// scripts/session_start.ts#L152-L162
const state = {
  sessionId,
  conversationId,
  lastProcessedIndex: -1,
  startedAt: new Date().toISOString(),
};
fs.writeFileSync(getSyncStateFile(cwd, sessionId), JSON.stringify(state, null, 2));

```

This initialization ensures the session state caching mechanism starts with a clean slate, ready to track progress from the first transcript entry.

## Loading and Using the Cursor in the Stop-Hook

Every time the stop-hook ([`send_messages_to_letta.ts`](https://github.com/letta-ai/claude-subconscious/blob/main/send_messages_to_letta.ts)) executes, it loads the cached cursor via `loadSyncState()`:

```typescript
// scripts/send_messages_to_letta.ts#L66-L71
const state = loadSyncState(cwd, hookInput.session_id, log);

```

The `loadSyncState` function in [`scripts/conversation_utils.ts`](https://github.com/letta-ai/claude-subconscious/blob/main/scripts/conversation_utils.ts) (lines 54-68) reads the JSON file and returns the stored `lastProcessedIndex`, which serves as the starting point for filtering.

### Filtering Messages by Index

The `formatMessagesForLetta()` function in [`scripts/transcript_utils.ts`](https://github.com/letta-ai/claude-subconscious/blob/main/scripts/transcript_utils.ts) receives the full transcript and the cached `startIndex`. It iterates only over messages with indices greater than the cursor:

```typescript
// scripts/transcript_utils.ts#L155-L159
for (let i = startIndex + 1; i < messages.length; i++) {
  // Process only new messages...
}

```

This logic ensures the session state caching mechanism prevents duplicate forwarding by excluding already-processed entries from the payload sent to Letta.

## Updating the Cursor After Successful Delivery

After the Letta SDK successfully receives the messages, the background worker ([`send_worker_sdk.ts`](https://github.com/letta-ai/claude-subconscious/blob/main/send_worker_sdk.ts)) updates the cursor. The worker writes the new `lastProcessedIndex` (set to the index of the last message sent) back to the JSON file:

```typescript
// scripts/send_worker_sdk.ts#L133-L138
state.lastProcessedIndex = payload.newLastProcessedIndex;
fs.writeFileSync(payload.stateFile, JSON.stringify(state, null, 2));

```

The `newLastProcessedIndex` value is calculated in the main hook as `messages.length - 1` (the newest transcript line index) before spawning the worker (lines 22-24 in [`send_messages_to_letta.ts`](https://github.com/letta-ai/claude-subconscious/blob/main/send_messages_to_letta.ts)).

## Practical Code Examples

### Reading the Cached State

To manually inspect the current cursor position:

```typescript
import { loadSyncState } from './scripts/conversation_utils.js';

const cwd = process.cwd();
const sessionId = 'abc123';
const state = loadSyncState(cwd, sessionId);
console.log('Last processed message index:', state.lastProcessedIndex);

```

### Implementing Cursor-Based Filtering

When processing transcripts, use the cursor to extract only new entries:

```typescript
import { formatMessagesForLetta, loadSyncState, saveSyncState } from './scripts/conversation_utils.js';

const state = loadSyncState(cwd, sessionId);
const newMessages = formatMessagesForLetta(transcript, state.lastProcessedIndex);

if (newMessages.length > 0) {
  // Send to Letta backend...
  // After success:
  state.lastProcessedIndex = transcript.length - 1;
  saveSyncState(cwd, state);
}

```

### Updating State After Batch Processing

To manually advance the cursor after sending messages:

```typescript
import { loadSyncState, saveSyncState } from './scripts/conversation_utils.js';

const state = loadSyncState(cwd, sessionId);
// After successfully sending messages 0-24:
state.lastProcessedIndex = 24;
saveSyncState(cwd, state);

```

## Summary

- **The session state caching mechanism** persists progress in `.letta/claude/session-<id>.json`, enabling durability across process restarts and machine migrations.
- **`lastProcessedIndex`** acts as a zero-based cursor initialized to -1, tracking the highest transcript index forwarded to Letta.
- **The stop-hook** ([`send_messages_to_letta.ts`](https://github.com/letta-ai/claude-subconscious/blob/main/send_messages_to_letta.ts)) loads the cursor and passes it to `formatMessagesForLetta()`, which filters out indices ≤ `lastProcessedIndex`.
- **The SDK worker** ([`send_worker_sdk.ts`](https://github.com/letta-ai/claude-subconscious/blob/main/send_worker_sdk.ts)) atomically updates the JSON file after successful API delivery, setting `lastProcessedIndex` to `messages.length - 1`.
- **Exactly-once delivery** is guaranteed because each invocation only processes messages with indices greater than the stored cursor.

## Frequently Asked Questions

### What is the default value of lastProcessedIndex for new sessions?

New sessions initialize `lastProcessedIndex` to **-1** in [`scripts/session_start.ts`](https://github.com/letta-ai/claude-subconscious/blob/main/scripts/session_start.ts) (lines 154-162). This sentinel value indicates that no transcript messages have been processed yet, ensuring the first message (index 0) is included in the initial payload.

### How does the session state caching mechanism prevent duplicate messages?

The mechanism stores the highest processed transcript index in a durable JSON file. On each hook invocation, `formatMessagesForLetta()` in [`scripts/transcript_utils.ts`](https://github.com/letta-ai/claude-subconscious/blob/main/scripts/transcript_utils.ts) iterates only over messages where `i > startIndex` (lines 155-159), effectively skipping entries that were already sent in previous runs.

### Where is the session state file stored on disk?

The cache resides at `<project-root>/.letta/claude/session-<sessionId>.json`, generated by `getSyncStateFile()` in [`scripts/conversation_utils.ts`](https://github.com/letta-ai/claude-subconscious/blob/main/scripts/conversation_utils.ts) (lines 35-39). This location ensures the state persists within the project directory and survives across Claude Code sessions.

### Which component updates the lastProcessedIndex after sending?

The [`send_worker_sdk.ts`](https://github.com/letta-ai/claude-subconscious/blob/main/send_worker_sdk.ts) background worker updates the cursor after successfully calling the Letta SDK. It writes the new index to the JSON file (lines 133-138), ensuring the session state caching mechanism reflects the latest delivered message before the next hook execution.