How OpenWork Handles Real-Time Updates: Voice Streaming and WebSocket Architecture

OpenWork handles real-time updates through a dual-architecture system that combines OpenAI Realtime API voice streaming with WebSocket-based UI state broadcasting managed by a lightweight mock MCP server.

OpenWork is an open-source workspace automation platform that requires instantaneous synchronization between server-side workflows and the user interface. According to the different-ai/openwork source code, the platform achieves this through two complementary mechanisms: streaming audio via OpenAI's Realtime API for voice interactions, and pushing JSON state updates through persistent WebSocket connections managed in packages/enterprise-mcp-mock-server/src/runtime/mock-server.ts.

OpenAI Realtime Voice Streaming

The voice streaming layer enables low-latency audio transcription and tool execution by establishing direct WebSocket connections between the client and OpenAI's Realtime service.

Requesting a Voice Session

The process begins when the front-end calls the /voice/realtime endpoint defined in apps/server/src/routes/core.ts. The server handler invokes createOpenAiRealtimeVoiceSession from apps/server/src/server.ts, which mints a temporary client-secret by contacting OpenAI's Realtime API.

// apps/server/src/server.ts
async function createOpenAiRealtimeVoiceSession(env: EnvService, input: unknown) {
  const fallbackKey = await resolveOpenAiRealtimeApiKey(env);
  const response = await fetch(
    "https://api.openai.com/v1/realtime/client_secret",
    {
      method: "POST",
      headers: {
        Authorization: `Bearer ${fallbackKey}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ /* payload */ })
    }
  );

  if (!response.ok) {
    throw new ApiError(
      response.status,
      "openai_realtime_failed",
      "Failed to create OpenAI Realtime session"
    );
  }
  const { client_secret } = await response.json();
  return { client_secret };
}

Establishing the WebSocket Connection

Once the client receives the secret, it opens a WebSocket using the provided URL. The low-level socket management resides in packages/cdp/src/cdp.ts, which handles the binary audio frame transmission and JSON response parsing.

// packages/cdp/src/cdp.ts
const socket = new WebSocket(webSocketDebuggerUrl);
socket.addEventListener("open", () => {
  // start sending audio frames
});
socket.addEventListener("message", (event) => {
  const data = JSON.parse(event.data);
  // handle transcribed text or tool responses
});

Handling Audio Streams

Audio chunks flow bidirectionally through this socket: the client streams raw PCM audio to OpenAI, and the service returns transcribed text or tool-generated JSON payloads. The front-end injects these transcripts directly into the UI state without page reloads, closing the session when the user stops speaking.

WebSocket-Based UI State Management

While voice streaming handles audio-specific real-time needs, general UI updates—such as workflow progress, MCP activity, and collaborative edits—flow through a separate WebSocket infrastructure embedded in the enterprise MCP mock server.

The Mock MCP Server Architecture

Located in packages/enterprise-mcp-mock-server/src/runtime/mock-server.ts, the server maintains a Set<Socket> to track active connections. Each new client connection is added to this set, with automatic cleanup on disconnect.

// packages/enterprise-mcp-mock-server/src/runtime/mock-server.ts
private sockets = new Set<Socket>();
server.on("connection", (socket) => {
  this.sockets.add(socket);
  socket.once("close", () => this.sockets.delete(socket));
});

Broadcasting Updates to Connected Clients

When workflow steps complete or worker heartbeats arrive from apps/server/src/worker-activity-heartbeat.ts, the server serializes the state change and writes it to every socket in the managed set.

// packages/enterprise-mcp-mock-server/src/runtime/mock-server.ts
function broadcastUpdate(payload: any) {
  const message = JSON.stringify(payload);
  for (const socket of this.sockets) {
    socket.write(message);
  }
}

This broadcast mechanism pushes data immediately rather than requiring clients to poll for updates.

Client-Side Integration with TanStack Query

The Electron front-end and web UI subscribe to these pushes using TanStack Query. By configuring queries with refetchInterval: 0, the client relies entirely on WebSocket-driven cache invalidation.

// apps/app/src/app/lib/openwork-server.ts
const { data } = useQuery(['workspaceState', workspaceId], fetchWorkspaceState, {
  refetchInterval: 0, // updates come via WebSocket push
});

When a message arrives from the mock MCP server, the client parses the JSON, invalidates the relevant query keys, and React re-renders the components instantly.

End-to-End Data Flow in Practice

A typical voice command scenario illustrates how these two systems coordinate:

1. Voice Initiation: The user clicks the microphone button, triggering a request to /voice/realtime and obtaining a client-secret from createOpenAiRealtimeVoiceSession.
2. Audio Processing: The client opens a WebSocket via packages/cdp/src/cdp.ts, streams audio to OpenAI, and receives a transcription like "run the monthly report."
3. Workflow Trigger: The server interprets the transcription, initiates a background workflow, and emits a status update through the mock MCP server's broadcast mechanism.
4. UI Synchronization: The front-end receives the JSON payload via its persistent socket, TanStack Query updates its cache, and the interface instantly displays the new workflow state.

The same socket infrastructure in packages/enterprise-mcp-mock-server/src/runtime/mock-server.ts powers collaborative workspace edits and plugin-generated events, ensuring all clients remain synchronized regardless of the action source.

Summary

• Dual-mechanism architecture: OpenWork separates real-time concerns into OpenAI Realtime for voice and WebSocket broadcasts for UI state.
• Server-side session management: The createOpenAiRealtimeVoiceSession function in apps/server/src/server.ts handles authentication with OpenAI's Realtime API.
• Persistent socket management: The mock MCP server maintains a Set<Socket> to broadcast JSON updates to all connected clients without polling overhead.
• React integration: TanStack Query subscriptions in apps/app/src/app/lib/openwork-server.ts enable instantaneous UI updates when server-side state changes.

Frequently Asked Questions

What is the difference between OpenAI Realtime and the WebSocket UI updates?

OpenAI Realtime handles bidirectional audio streaming and transcription through a dedicated WebSocket to OpenAI's servers, managed by the createOpenAiRealtimeVoiceSession flow in apps/server/src/server.ts. WebSocket UI updates use a separate socket connection managed by packages/enterprise-mcp-mock-server/src/runtime/mock-server.ts to push JSON state changes—like workflow progress or heartbeats—to the React front-end. The former is specialized for voice, while the latter is general-purpose for application state.

How does OpenWork prevent memory leaks with WebSocket connections?

The mock MCP server in packages/enterprise-mcp-mock-server/src/runtime/mock-server.ts stores sockets in a Set<Socket> and attaches a once("close") listener that immediately removes disconnected sockets from the set. This ensures the broadcastUpdate function only iterates over active connections, preventing accumulation of dead socket references.

Can the WebSocket infrastructure handle multiple concurrent workspaces?

Yes. The broadcast mechanism writes to all sockets in the managed set, and the client-side TanStack Query subscriptions include workspace-specific query keys like ['workspaceState', workspaceId]. When a message arrives, the client can filter or route updates based on the workspace ID contained in the JSON payload, allowing a single socket connection to multiplex updates for multiple contexts.

Where does the fallback OpenAI API key resolution happen?

The resolveOpenAiRealtimeApiKey function in apps/server/src/server.ts provides fallback key resolution when the primary authentication method fails. This ensures voice sessions can still be created even if the initial credential check encounters issues, though the specific fallback logic depends on the environment configuration managed by the EnvService parameter.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}