SimStudio Agent Handler Architecture: Real-Time AI Workflow Execution

The SimStudio agent handler architecture uses a Socket.IO-based Realtime Engine to orchestrate AI workflows through a seven-phase pipeline that authenticates sessions, resolves variables, traverses execution graphs, streams LLM responses, and processes side effects.

The simstudioai/sim repository implements a sophisticated real-time orchestration layer for AI agent execution. At its core, the agent handler architecture manages multi-step workflows through a modular system of Socket.IO handlers that coordinate graph traversal, variable resolution, and streamed AI responses.

Architecture Overview

The SimStudio Realtime Engine operates as a bidirectional Socket.IO server that maintains persistent connections with clients while executing complex workflow graphs. The architecture centers on the WorkflowExecutionContext, an in-memory state container created per execution that tracks the block registry, variable stores, and execution metadata.

The system implements a layered handler pattern where specialized TypeScript modules manage distinct phases of the execution lifecycle. Each handler resides in apps/realtime/src/handlers/ and communicates through a shared context object, enabling real-time updates while maintaining clean separation of concerns between authentication, graph traversal, and AI inference.

The Seven-Phase Execution Pipeline

The agent handler architecture processes workflow execution through seven discrete phases, each handled by dedicated modules in the Realtime Engine.

1. Connection and Authentication (apps/realtime/src/auth.ts)

When a client initiates a Socket.IO connection to /socket.io, the auth.ts middleware intercepts the request and validates the session token. Upon successful authentication, the middleware injects the user and workspace context into the socket instance, ensuring all subsequent operations execute within the correct authorization boundary.

2. Workflow Initialization (apps/realtime/src/handlers/connection.ts)

The execution begins when the client emits a workflow:start event containing a serialized workflow definition. The connection.ts handler instantiates a WorkflowExecutionContext, generating a unique execution ID and initializing the block registry. This context object persists throughout the lifecycle of the workflow, serving as the central state repository for all subsequent operations.

3. Variable Resolution (apps/realtime/src/handlers/variables.ts)

Before any block executes, the variables.ts handler resolves input variables from multiple sources, including user-provided prompts and previous tool outputs. The system maintains a per-execution in-memory variable store that makes resolved values available to downstream blocks through a unified access interface.

4. Graph Traversal (apps/realtime/src/handlers/subblocks.ts)

Each workflow node represents a sub-block (starter, agent, function, or API call) within a directed acyclic graph (DAG). The subblocks.ts handler walks this graph, ensures all dependencies are satisfied before execution, and delegates each ready node to the appropriate executor. This phase implements the orchestration logic that determines execution order and parallelization opportunities.

5. Agent Block Execution (apps/realtime/src/handlers/workflow.ts)

When the traversal reaches a node of type agent, the workflow.ts handler invokes the runAgentBlock function. This critical phase loads the agent's configuration—including the LLM model, system prompt, and tool catalog—and initiates a streaming request via the requestJson client from the SDK. The handler respects selectedOutputs and stream flags, emitting incremental agent:output events to the client for real-time display of partial responses.

6. Post-Execution Operations (apps/realtime/src/handlers/operations.ts)

After the LLM returns a response, the operations.ts handler processes declared actions such as external API calls, database writes, or tool invocations. This phase updates the variable store with operation results and may trigger follow-up sub-blocks, enabling complex multi-step agent workflows with side effects.

7. State Synchronization and Cleanup (apps/realtime/src/handlers/presence.ts)

Throughout execution, presence.ts maintains real-time synchronization of the "running" state indicator visible to clients. When the workflow completes successfully or encounters an error, connection.ts tears down the WorkflowExecutionContext and emits a final workflow:finished event, releasing memory resources and closing the execution lifecycle.

Implementation Details and Code Examples

The following examples demonstrate the client-server interaction patterns and the core agent execution logic.

Client-side initiation:

// Initiate workflow execution
socket.emit('workflow:start', {
  workflow: serializedWorkflow,   // JSON from the block editor
  variables: { prompt: 'Summarize the document' }
});

// Listen for streamed agent outputs
socket.on('agent:output', ({ blockId, chunk }) => {
  console.log(`Agent ${blockId} response:`, chunk);
});

Server-side agent execution:

// Simplified runAgentBlock implementation from workflow.ts
async function runAgentBlock(ctx: WorkflowExecutionContext, block: AgentBlock) {
  const { model, systemPrompt, tools } = block.config;
  
  const response = await requestJson(
    llmContract,                     // Defined in SDK
    { 
      body: { 
        model, 
        systemPrompt, 
        messages: ctx.vars.input 
      } 
    },
    { signal: ctx.abortSignal, stream: true }
  );
  
  // Stream chunks back to client
  for await (const chunk of response) {
    ctx.socket.emit('agent:output', { 
      blockId: block.id, 
      chunk 
    });
  }
  
  return response.finalResult;
}

Summary

• The agent handler architecture implements a seven-phase pipeline through the SimStudio Realtime Engine, using Socket.IO for bidirectional communication.
• Core execution logic resides in apps/realtime/src/handlers/workflow.ts, specifically within the runAgentBlock function that streams LLM responses via requestJson.
• Variable resolution occurs through apps/realtime/src/handlers/variables.ts, maintaining per-execution state in an isolated in-memory store.
• The DAG traversal engine in apps/realtime/src/handlers/subblocks.ts coordinates execution order and dependency resolution across heterogeneous block types.
• Real-time streaming is achieved through incremental agent:output events emitted during the LLM inference phase.

Frequently Asked Questions

How does SimStudio authenticate agent execution requests?

SimStudio validates session tokens through the apps/realtime/src/auth.ts middleware before allowing Socket.IO connections. This injects user and workspace context into the execution environment, ensuring all agent operations occur within authorized boundaries.

What component triggers AI agent block execution?

The workflow.ts handler triggers agent execution when the DAG traversal identifies a block with type agent. The runAgentBlock function then orchestrates the LLM call using configuration from the block's metadata, including model selection and system prompts.

How does the architecture support real-time response streaming?

The runAgentBlock function in apps/realtime/src/handlers/workflow.ts utilizes the stream: true flag when calling requestJson. It iterates through the async response generator, emitting agent:output events for each chunk received from the LLM provider, enabling clients to display partial responses immediately.

Where are workflow variables stored during execution?

Variables are stored in a per-execution in-memory store managed by apps/realtime/src/handlers/variables.ts. This store persists within the WorkflowExecutionContext for the duration of the workflow lifecycle, providing isolated state management that prevents cross-contamination between concurrent executions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}