How the SDK Worker Processes Transcripts in the Background in letta-ai/claude-subconscious

The SDK worker (send_worker_sdk.ts) is a detached Node.js process that reads a JSON payload from the main Claude-Code hook, sends transcript messages to a Letta agent via the Letta Code SDK, streams the response, and updates the sync state file—all without blocking the main process.

The letta-ai/claude-subconscious repository implements an asynchronous bridge between Claude-Code and Letta agents. The SDK worker located at scripts/send_worker_sdk.ts handles background transcript processing to ensure the main Claude-Code stop-hook remains non-blocking while maintaining accurate incremental sync with Letta agents.

Architecture Overview

The SDK worker operates as a detached child process spawned by the main stop-hook. This architecture decouples the Letta SDK communication from the Claude-Code lifecycle, preventing network latency or LLM processing time from delaying the user's editor experience. The worker receives its instructions through a temporary JSON payload file, executes the SDK operations asynchronously, and manages persistent state to enable incremental transcript synchronization.

Step-by-Step Processing Flow

The transcript processing follows a precise eight-step pipeline:

Payload Creation in the Main Hook

In scripts/send_messages_to_letta.ts, the hook constructs a sdkPayload object containing the agent ID, conversation ID, formatted Letta message XML, sync state file path, last processed index, working directory, and tool mode configuration (lines 16-25). This payload is serialized to a temporary JSON file that serves as the inter-process communication mechanism.

Worker Spawning

The hook invokes spawnSilentWorker() from scripts/conversation_utils.ts to launch send_worker_sdk.ts as a detached child process, passing the payload file path as the sole CLI argument (lines 29-31). The spawn configuration uses detached: true with stdio redirected to /dev/null, ensuring the worker survives independently of the parent process.

Payload Loading

Upon startup, the worker reads the JSON payload from the temporary file path provided in process.argv[2], logs the session identifier, and delegates to the sendViaSdk() function (lines 28-30 in send_worker_sdk.ts).

SDK Session Setup

Inside sendViaSdk(), the Letta Code SDK (@letta-ai/letta-code-sdk) is dynamically imported at line 44 to prevent parsing errors if the SDK is not installed. The session options configure tool permissions based on the sdkToolsMode parameter:

• off: Blocks all client-side tools, creating a read-only mode
• read-only: Whitelists only Read, Grep, Glob, web_search, and fetch_webpage tools (lines 47-65)
• full: Places no restrictions on available tools

Message Dispatch

The worker creates a Letta SDK session using resumeSession(conversationId, sessionOptions), then transmits the payload message via session.send(payload.message) (lines 73-78). This establishes the connection to the Letta agent with the appropriate conversation context.

Response Streaming

The worker iterates over session.stream() to capture real-time assistant output, logging each chunk as it arrives. The stream handling loop (lines 83-93) also captures any tool calls or errors that occur during Letta-side processing, providing complete observability into the agent's execution.

State File Update

After successful message delivery, the worker reads the persistent sync-state file specified in the payload, updates lastProcessedIndex to the newLastProcessedIndex value from the payload, and writes the updated state back to disk (lines 34-38). This guarantees that subsequent hook executions only process new transcript entries, preventing duplicate message sending.

Cleanup

The worker removes the temporary payload file using fs.unlinkSync(), closes the SDK session, and logs completion status (lines 41-45). This ensures no ephemeral files accumulate and resources are properly released.

Configuration and Tool Modes

The SDK tool mode (sdkToolsMode) provides granular control over agent capabilities during background processing. When set to read-only, the worker specifically restricts the Letta agent to information-retrieval tools only, preventing file system modifications during automated transcript analysis. The off mode provides maximum safety for audit scenarios, while full enables complete agent autonomy for complex refactoring tasks.

Code Implementation

Creating the SDK Payload

The main hook prepares the worker instructions:

const sdkPayload = {
  agentId,
  conversationId,
  sessionId: hookInput.session_id,
  message: userMessage,                     // XML with new transcript entries
  stateFile: getSyncStateFile(hookInput.cwd, hookInput.session_id),
  newLastProcessedIndex: messages.length - 1,
  cwd: hookInput.cwd,
  sdkToolsMode,                            // 'off' | 'read-only' | 'full'
};

fs.writeFileSync(payloadFile, JSON.stringify(sdkPayload), 'utf-8');

Spawning the Background Worker

The hook initiates the detached process:

const workerScript = path.join(__dirname, 'send_worker_sdk.ts');
const child = spawnSilentWorker(workerScript, payloadFile, hookInput.cwd);
log(`Spawned SDK worker (PID: ${child.pid})`);

Processing in the Worker

The worker executes the SDK communication and state management:

async function main(): Promise<void> {
  const payloadFile = process.argv[2];
  const payload: SdkPayload = JSON.parse(fs.readFileSync(payloadFile, 'utf-8'));
  const success = await sendViaSdk(payload);

  if (success) {
    // Update sync state so next hook run knows the last line processed
    const state = JSON.parse(fs.readFileSync(payload.stateFile, 'utf-8'));
    state.lastProcessedIndex = payload.newLastProcessedIndex;
    fs.writeFileSync(payload.stateFile, JSON.stringify(state, null, 2));
  }

  fs.unlinkSync(payloadFile);   // clean up temp file
}

Summary

• The detached process architecture prevents Letta SDK network calls from blocking the Claude-Code stop-hook execution.
• Dynamic SDK importing ensures the codebase remains parseable even when @letta-ai/letta-code-sdk is not installed.
• Three-tier tool permission system (off, read-only, full) controls agent capabilities during background processing.
• Incremental synchronization via lastProcessedIndex tracking prevents duplicate transcript processing and ensures message ordering.
• Streaming response handling captures real-time agent output and tool execution logs for comprehensive debugging.

Frequently Asked Questions

What happens if the Letta Code SDK is not installed?

The worker uses dynamic imports (import('@letta-ai/letta-code-sdk')) at line 44 of send_worker_sdk.ts to conditionally load the SDK only when needed. If the SDK is missing, the import fails gracefully, allowing the rest of the codebase to function for users who do not require Letta integration.

How does the worker handle transcript formatting?

The worker does not directly read the transcript file. Instead, scripts/transcript_utils.ts reads the JSONL transcript and converts entries into Letta-compatible XML via formatMessagesForLetta() before the worker spawns. The worker receives pre-formatted XML in the payload's message field.

What is the purpose of the sync state file?

The sync state file (managed by getSyncStateFile() in conversation_utils.ts) persists the lastProcessedIndex between hook executions. This index tracks which transcript lines have already been sent to Letta, enabling the system to send only new messages during incremental updates and preventing duplicate processing.

Why use a detached process instead of asynchronous calls in the main thread?

The detached process ensures survival independent of the Claude-Code lifecycle. If the main hook process exits, the worker continues processing the Letta communication to completion. Additionally, this isolation prevents any SDK errors or network timeouts from affecting the main editor's stop-hook performance.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}