# Agent Runtime in OpenMAIC's Pro Workbench: Core Execution Engine Explained

> Discover the agent runtime, OpenMAIC Pro workbench's core execution engine. Understand its role in AI capabilities, session orchestration, LLM invocation, and state synchronization.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: internals
- Published: 2026-09-06

---

**The agent runtime is the central server-side execution engine that powers all AI-driven capabilities in OpenMAIC's Pro workbench, handling session orchestration, tool validation, LLM invocation, and state synchronization.**

The **Pro workbench** in OpenMAIC represents the advanced authoring environment where users leverage intelligent agents for content generation. At its foundation lies the **agent runtime**—a dedicated server-side subsystem that bridges interactive UI components with backend AI services. This article examines how the runtime operates, its architectural responsibilities, and how developers can interact with it.

## What the Agent Runtime Does in OpenMAIC Pro

The agent runtime lives in `lib/server/agent-runtime/*` and serves four critical functions for the Pro workbench:

- **Session lifecycle management** – creates, persists, and streams conversation state
- **Tool registry and validation** – exposes callable tools with contract enforcement
- **AI service orchestration** – invokes LLMs, image/video generators, and voice cloning
- **UI state synchronization** – broadcasts runtime status and diagnostics to the client

When a user accesses Pro features—such as slide generation, voice-over addition, or skill execution—the workbench queries the runtime to determine availability and dispatch operations.

## Session Orchestration and State Management

The runtime's entry point, **[`runner.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/runner.ts)**, handles all session-level operations. It registers new sessions, loads historical context, and establishes event streams for real-time UI updates.

```typescript
// lib/server/agent-runtime/runner.ts
// Core session orchestration logic

```

Each session receives a stable `sessionId` that the Pro workbench uses to track ongoing operations. The runner coordinates between:

1. **Incoming HTTP requests** from [`owner-session-client.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/owner-session-client.ts)
2. **Tool call dispatch** to specialized modules like [`generation-tools.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/generation-tools.ts)
3. **Event streaming** back to the UI via [`event-notify-bus.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/event-notify-bus.ts)

The **owner-session-client** ([`lib/workbench/owner-session-client.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/workbench/owner-session-client.ts)) provides the client-side API surface that the workbench uses to fetch active sessions, check status, and retrieve runtime-driven URLs.

## Tool Registry and Call Validation

Before any AI operation executes, the runtime validates tool calls through **[`tool-call-integrity.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tool-call-integrity.ts)**. This ensures that:

- Tool names match registered contracts
- Arguments conform to expected schemas
- Execution context carries proper authorization

```typescript
// Example: Invoking a generation tool from a Pro session
import { GENERATION_TOOL_NAMES } from '@/lib/server/agent-runtime/generation-tools';
import { callAgentTool } from '@/lib/server/agent-runtime/runner';

async function generateSlide(deckId: string) {
  const result = await callAgentTool({
    tool: GENERATION_TOOL_NAMES.SLIDE_GENERATOR,
    args: { deckId, prompt: 'Explain photosynthesis' },
  });
  // result contains generated slide content, ready for UI display
  return result;
}

```

Available tools include:

- **SLIDE_GENERATOR** – AI-powered presentation creation
- **IMAGE_GENERATOR** – Visual asset synthesis via [`generate-image.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/generate-image.ts)
- **VIDEO_GENERATOR** – Video composition via [`generate-video.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/generate-video.ts)
- **VOICE_CLONE** – Audio synthesis and voice replication
- **MATERIAL_TOOLS** – Asset upload and management utilities

## Runtime Configuration and Feature Flags

The runtime's behavior is controlled by **[`config.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/config.ts)**, which parses environment variables and produces a `RuntimeConfig` object:

```typescript
// lib/server/agent-runtime/config.ts
// Environment-driven configuration

```

Key configuration sources:

- **`OPENMAIC_AGENT_RUNTIME_ENABLED`** – master toggle for runtime availability
- **`DATABASE_URL`** – persistence layer connection for session storage

```tsx
// Example: Checking if the runtime is enabled before showing a Pro‑only button
import { getRuntimeConfig } from '@/lib/server/agent-runtime/config';

async function ProButton() {
  const cfg = await getRuntimeConfig(); // → { enabled: boolean, ... }
  if (!cfg.enabled) return null; // hide button if runtime is off
  return <button onClick={startProFeature}>Run Pro Feature</button>;
}

```

When disabled or misconfigured, the Pro workbench gracefully degrades—UI elements show disabled states while the server logs diagnostic warnings (validated in [`server/config-validation.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/server/config-validation.test.ts)).

## AI Generation and Media Synthesis

The runtime delegates actual AI operations to specialized modules:

| Module | Responsibility |
|--------|---------------|
| [`generation-ai-call.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/generation-ai-call.ts) | LLM invocation and response handling |
| [`generate-image.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/generate-image.ts) | Image synthesis pipeline |
| [`generate-video.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/generate-video.ts) | Video generation and composition |
| [`session-material.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/session-material.ts) | Asset metadata and upload status |

These modules abstract provider-specific implementations, presenting a unified interface to the runner regardless of underlying AI service.

## State Synchronization and Diagnostics

Runtime state propagates to the UI through several channels:

- **[`session-material.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/session-material.ts)** and **[`session-urls.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/session-urls.ts)** – expose asset URLs and upload progress
- **[`event-notify-bus.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/event-notify-bus.ts)** – streams real-time status updates
- **Video export integration** – embeds runtime diagnostics into final outputs

```typescript
// Example: Adding a runtime‑diagnostic to a video manifest (used internally)
import { addRuntimeDiagnostic } from '@/lib/video-export/emit';

function handleRuntimeError(err: Error) {
  addRuntimeDiagnostic({
    code: 'interactive-runtime-failure',
    message: err.message,
  });
}

```

This design ensures that failures are visible in exported content without breaking the user-facing output.

## Pro Workbench Integration Points

The runtime connects to the Pro workbench UI through several established patterns:

**[`use-workbench-pro-edit.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/use-workbench-pro-edit.ts)** – React hook that adapts editor layout based on runtime state, enabling dynamic tool palettes and context-sensitive controls.

**[`pro-swap.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/pro-swap.ts)** – Handles seamless transitions between classic and Pro workbench modes, preserving session continuity and runtime badges.

**[`tool-presentation.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tool-presentation.test.ts)** – Test suite verifying that every registered runtime tool surfaces correctly in the Pro UI, preventing drift between backend capabilities and frontend exposure.

## Summary

The **agent runtime in OpenMAIC's Pro workbench** serves as the authoritative execution layer for all AI-driven authoring features:

- Lives server-side in `lib/server/agent-runtime/*` with [`runner.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/runner.ts) as the primary entry point
- Validates and dispatches tool calls through [`tool-call-integrity.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tool-call-integrity.ts) and specialized tool modules
- Configures via environment variables parsed by [`config.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/config.ts)
- Synchronizes state to the Pro workbench UI through [`owner-session-client.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/owner-session-client.ts) and material/session modules
- Gracefully degrades when disabled, ensuring consistent user experience

## Frequently Asked Questions

### What happens if the agent runtime is disabled in OpenMAIC?

The Pro workbench detects the disabled state via `getRuntimeConfig()` and hides Pro-only features. UI buttons and menus do not render, and the server logs configuration warnings without throwing errors. Users retain access to classic workbench functionality.

### How does the agent runtime validate tool calls?

Validation occurs in [`tool-call-integrity.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tool-call-integrity.ts), which checks tool names against registered contracts and verifies argument schemas before execution. Invalid calls are rejected with diagnostic codes that propagate through [`event-notify-bus.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/event-notify-bus.ts) to the UI.

### Can I add custom tools to the OpenMAIC agent runtime?

Yes. Register new tool names in a dedicated `*-tools.ts` module following the pattern of [`generation-tools.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/generation-tools.ts), implement the execution logic, and export through [`runner.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/runner.ts). The test suite in [`tool-presentation.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tool-presentation.test.ts) ensures automatic UI exposure.

### Where does the agent runtime store session data?

Session persistence uses the database connection specified by `DATABASE_URL`. The [`runner.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/runner.ts) module loads historical context at session start and commits state changes through the standard ORM layer, with [`session-material.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/session-material.ts) handling asset-specific metadata.