# How @openmaic/generation Orchestrates AI Model Calls: Inside the OpenMAIC Engine

> Discover how OpenMAIC generation orchestrates AI model calls via a seven-stage pipeline. Learn about prompt templating, AI function calls, JSON repair, and output normalization.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: internals
- Published: 2026-09-10

---

**The @openmaic/generation package orchestrates AI model calls through a seven-stage pipeline that constructs templated prompts, invokes a configurable `AICallFn`, repairs malformed JSON responses, and normalizes outputs into strongly typed educational content objects.**

The `@openmaic/generation` package serves as the core orchestration layer within the **THU-MAIC/OpenMAIC** repository, transforming high-level instructional requirements into concrete slides, quizzes, and interactive widgets. It abstracts the complexity of large language model interactions by encapsulating the entire LLM workflow—from prompt construction to domain-specific post-processing—into a modular, TypeScript-based engine.

## The Seven-Stage Orchestration Pipeline

The package implements a robust workflow that handles every aspect of AI content generation. Each stage is designed to be interchangeable, allowing developers to swap underlying models or extend support for new content types without disrupting the core architecture.

### Stage 1: Prompt Template Loading and Variable Interpolation

The orchestration begins with **prompt construction** in `packages/@openmaic/generation/src/prompts/loader.js`. The `buildPrompt` function loads templated system and user prompts, then interpolates required variables—such as title, outline, image lists, and language directives—into the template structure. These templates are registered via `PROMPT_IDS` in `packages/@openmaic/generation/src/prompts/index.ts`, enabling type-safe selection of the appropriate prompt for each generation task.

### Stage 2: Model Invocation via the AICallFn Interface

The package decouples itself from specific LLM providers through the **`AICallFn`** interface defined in `packages/@openmaic/generation/src/pipeline-types.ts`. This generic function signature accepts `(system, user, images?)` and returns `Promise<string>`, allowing the host application to inject any compatible AI client. The orchestrator invokes this function via `aiCall` directly within generation functions like `generateSlideContent` and `generateSceneOutlinesFromRequirements` in `packages/@openmaic/generation/src/scene-generator.ts` and `packages/@openmaic/generation/src/outline-generator.ts`, optionally passing vision images for multimodal models.

### Stage 3: Response Parsing and JSON Repair

Raw LLM responses often contain malformed JSON or extraneous markdown formatting. The `parseJsonResponse` utility in `packages/@openmaic/generation/src/json-repair.ts` handles robust parsing with automatic error recovery. For interactive widget generation, the `extractHtml` function (also in [`scene-generator.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/scene-generator.ts)) isolates HTML content from model responses, ensuring downstream processors receive clean, structured data regardless of the LLM's output format.

### Stage 4: Content Normalization and Asset Resolution

After parsing, the package executes domain-specific **post-processing** to normalize generated content. Key functions in `packages/@openmaic/generation/src/scene-generator.ts` include:
- **`fixElementDefaults`** – Fills missing required fields with sensible defaults.
- **`processLatexElements`** – Renders mathematical expressions via KaTeX.
- **`resolveImageIds`** – Maps abstract image IDs to concrete URLs using the provided `imageMapping`.
- **`normalizeGeneratedVideoRefs`** – Standardizes video reference formats across different scene types.

### Stage 5: Resilient Execution with Retry Logic

Transient failures such as rate limits or network timeouts are handled by `callWithRetry` in `packages/@openmaic/generation/src/generation-retry.ts`. This utility detects retry-able errors using a regex-based classifier and implements exponential backoff. When retries are exhausted, the orchestrator wraps failures into typed error objects like `SceneContentFailure` or `PBLGenerationError`, optionally invoking fallback planners for project-based learning (PBL) content.

### Stage 6: Dynamic Scene Type Dispatch

The `generateSceneContent` function acts as a central router (lines 27–78 in [`scene-generator.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/scene-generator.ts)), dispatching to specialized generators based on the scene type:
- **`generateSlideContent`** for presentation slides.
- **`generateQuizContent`** for assessment items.
- **`generateWidgetContent`** for ultra-mode interactive elements (`simulation`, `diagram`, `code`, `game`, `visualization3d`, `procedural-skill`).
- **`generatePBLSceneContent`** for project-based learning modules.

### Stage 7: Type-Safe Output Generation

Each generator returns a strongly typed object defined in `packages/@openmaic/generation/src/scene-types.ts`. These include `GeneratedSlideContent`, `GeneratedQuizContent`, `GeneratedInteractiveContent`, and `GeneratedPBLContent`, ensuring that downstream rendering engines receive predictably structured data with full TypeScript intellisense support.

## Practical Implementation Examples

### Generating Slide Content from an Outline

```typescript
import { generateSceneContent } from '@openmaic/generation';
import { myAICall } from './my-ai-client';

// `outline` is a SceneOutline produced by the outline generator
const slide = await generateSceneContent(outline, myAICall, {
  imageMapping,         // map of image IDs → URLs
  visionEnabled: true,   // pass vision images if the model supports them
  logger: consoleLogger,
});

```

This call internally builds the prompt using `PROMPT_IDS.SLIDE_CONTENT`, invokes `myAICall`, parses the JSON response, fixes defaults, resolves image IDs, renders LaTeX, and returns a `GeneratedSlideContent` object.

### Implementing Resilient AI Calls with Retry Logic

```typescript
import { callWithRetry } from '@openmaic/generation/src/generation-retry';

const response = await callWithRetry(
  async () => myAICall(systemPrompt, userPrompt, visionImages),
  { maxAttempts: 3, logger: consoleLogger }
);

```

The `callWithRetry` wrapper detects transient errors via the regex defined in [`generation-retry.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/generation-retry.ts) and automatically retries the operation up to the configured limit before throwing a terminal error.

### Creating Interactive Ultra-Mode Widgets

```typescript
import { generateWidgetContent } from '@openmaic/generation';

const widget = await generateWidgetContent(outline, myAICall, undefined, {
  allowProceduralSkill: true,
  logger: consoleLogger,
});

console.log(widget.html);          // HTML returned by the LLM
console.log(widget.widgetConfig); // Optional JSON config embedded in the HTML

```

This function selects the appropriate `PROMPT_IDS` (e.g., `DIAGRAM_CONTENT`), constructs the prompt, calls the model, extracts the HTML using `extractHtml`, and returns the widget payload with configuration metadata.

## Core Files and Architecture

The orchestration logic is distributed across specialized modules for maintainability:

- **`packages/@openmaic/generation/src/scene-generator.ts`** – Central dispatcher containing `generateSceneContent` and scene-specific generators (lines 27–78), plus post-processing utilities like `fixElementDefaults` and `extractHtml`.
- **`packages/@openmaic/generation/src/outline-generator.ts`** – Generates scene outlines from user requirements, serving as the entry point for the orchestration pipeline.
- **`packages/@openmaic/generation/src/prompts/loader.js`** – Implements `buildPrompt` for template loading and variable interpolation.
- **`packages/@openmaic/generation/src/prompts/index.ts`** – Defines `PROMPT_IDS` and exports prompt-building utilities.
- **`packages/@openmaic/generation/src/json-repair.ts`** – Contains `parseJsonResponse` for tolerant JSON parsing.
- **`packages/@openmaic/generation/src/generation-retry.ts`** – Implements `callWithRetry` with regex-based error classification for transient failures.
- **`packages/@openmaic/generation/src/scene-types.ts`** – Type definitions for all generated content variants.
- **`packages/@openmaic/generation/src/pipeline-types.ts`** – Defines the `AICallFn` contract for model agnosticism.
- **`packages/@openmaic/generation/src/prompt-formatters.ts`** – Helper functions for assembling context (teacher personas, agent configurations).
- **`packages/@openmaic/generation/src/interactive-post-processor.ts`** – Sanitizes and validates HTML for interactive widgets.

## Summary

- The **@openmaic/generation** package provides a complete orchestration framework for AI-driven educational content, handling everything from prompt construction to error recovery.
- It uses a **generic `AICallFn` interface** to remain agnostic of specific LLM providers while supporting multimodal inputs including vision images.
- **Robust parsing utilities** like `parseJsonResponse` and `extractHtml` ensure reliable handling of imperfect LLM outputs.
- The architecture supports **six distinct scene types** through a dynamic dispatch system in [`scene-generator.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/scene-generator.ts), delivering strongly typed results for each category.
- **Automatic retry logic** with exponential backoff mitigates transient failures, improving reliability in production environments.

## Frequently Asked Questions

### How does @openmaic/generation handle malformed JSON responses from AI models?

The package uses the `parseJsonResponse` utility in `packages/@openmaic/generation/src/json-repair.ts` to tolerate malformed JSON by attempting repairs before parsing. For HTML extraction in widget generation, the `extractHtml` function isolates markup from surrounding text, ensuring downstream processors receive valid content even when the LLM includes explanatory markdown or code fences.

### What types of educational content can the orchestration engine generate?

The system supports four primary content categories defined in [`scene-types.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/scene-types.ts): **slides** (`GeneratedSlideContent`), **quizzes** (`GeneratedQuizContent`), **interactive widgets** (`GeneratedInteractiveContent` including simulations, diagrams, and games), and **project-based learning scenes** (`GeneratedPBLContent`). The `generateSceneContent` router automatically dispatches to the appropriate generator based on the scene type specified in the outline.

### Can developers swap the underlying LLM provider without modifying the core logic?

Yes. The package accepts an **`AICallFn`** implementation from the host application, defined in [`pipeline-types.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/pipeline-types.ts) as `(system: string, user: string, images?: string[]) => Promise<string>`. This dependency injection pattern allows seamless swapping between OpenAI, Anthropic, Gemini, or self-hosted models without changing the orchestration code in [`scene-generator.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/scene-generator.ts).

### How does the retry mechanism determine which errors are transient?

The `callWithRetry` function in `packages/@openmaic/generation/src/generation-retry.ts` uses a regex-based classifier to identify retry-able conditions such as rate limits, network timeouts, and temporary service unavailability. When detected, it implements exponential backoff with configurable `maxAttempts`, wrapping terminal failures into typed errors like `SceneContentFailure` for upstream handling.