# Agent Tool Interfaces in OpenMAIC: How `tool-schemas.ts` Shapes LLM Action Selection

> Explore agent tool interfaces in OpenMAIC's tool-schemas.ts. Learn how getEffectiveActions and getActionDescriptions filter and describe LLM actions for better system prompts.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: api-reference
- Published: 2026-09-02

---

**The [`lib/orchestration/tool-schemas.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/orchestration/tool-schemas.ts) module defines two primary agent tool interfaces—`getEffectiveActions` and `getActionDescriptions`—that filter available actions by context and generate human-readable descriptions for system prompts.**

The OpenMAIC repository provides a multimodal agent orchestration framework where language models control presentation tools through structured JSON outputs. The [`lib/orchestration/tool-schemas.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/orchestration/tool-schemas.ts) file implements the **agent tool interfaces** that determine which actions an LLM can invoke and how those capabilities are communicated to the model. These interfaces bridge the gap between raw action definitions and context-aware prompt engineering.

## The Two Core Agent Tool Interfaces

The module exports two complementary functions that work together to shape the agent's tool set at runtime:

### `getEffectiveActions`: Context-Aware Action Filtering

This function filters a list of allowed action identifiers based on the **current scene type** (slide versus non-slide contexts). It is defined starting at line 16 in [`lib/orchestration/tool-schemas.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/orchestration/tool-schemas.ts).

**Implementation logic:**
- Accepts an array of action names and an optional `sceneType` parameter
- If the scene is not a slide, removes actions belonging to the `SLIDE_ONLY_ACTIONS` set (spotlight, laser, and similar slide-specific tools)
- Returns the filtered array of context-appropriate actions

```typescript
import { getEffectiveActions } from '@/lib/orchestration/tool-schemas';

const allActions = [
  'spotlight',      // Slide-only
  'laser',          // Slide-only  
  'wb_open',        // Universal
  'wb_draw_text',   // Universal
  'play_video',     // Universal
];

// Non-slide context filters out slide-only actions
const usable = getEffectiveActions(allActions, 'whiteboard');
// Result: ['wb_open', 'wb_draw_text', 'play_video']

```

The `SLIDE_ONLY_ACTIONS` constant imported from [`lib/types/action.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/types/action.ts) drives this filtering logic. This ensures agents receive only actions that are meaningful for the active context, preventing the model from attempting unavailable operations.

### `getActionDescriptions`: Prompt-Ready Action Documentation

This function generates **human-readable descriptions** of each permitted action for inclusion in system prompts. It is implemented across lines 29-72 in [`lib/orchestration/tool-schemas.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/orchestration/tool-schemas.ts).

**Implementation structure:**
- A `descriptions` map enumerates every supported action with explanatory text
- Builds a bullet-point list of descriptions for actions in the supplied `allowedActions` array
- Falls back to a generic speech-only message when no actions are permitted

**Supported actions in the descriptions map include:**
- `spotlight` – Presentation spotlight control
- `laser` – Laser pointer activation
- `wb_open` – Open whiteboard for hand-drawn explanations
- `wb_draw_text` – Add text annotations
- `wb_draw_shape` – Insert geometric shapes
- `wb_draw_chart` – Create data visualizations
- `wb_draw_latex` – Render mathematical expressions
- `wb_draw_table` – Insert tabular content
- `wb_draw_line` – Freeform line drawing
- `wb_draw_code` – Display formatted code blocks
- `wb_edit_code` – Modify existing code elements
- `wb_clear`, `wb_delete`, `wb_close` – Whiteboard management
- `play_video` – Video playback control

```typescript
import { getActionDescriptions } from '@/lib/orchestration/tool-schemas';

const allowedActions = ['wb_open', 'wb_draw_text', 'play_video'];

const promptFragment = getActionDescriptions(allowedActions);
// Returns formatted string:
// - wb_open: Open the whiteboard for hand-drawn explanations...
// - wb_draw_text: Add text to the whiteboard...
// - play_video: Start playback of a video element on the current slide...

```

## Integration with the Orchestration Layer

The [`lib/orchestration/orchestrator.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/orchestration/orchestrator.ts) file demonstrates how these agent tool interfaces are consumed in practice. The orchestration layer uses both functions to:

1. **Determine contextually valid actions** via `getEffectiveActions`
2. **Format action documentation** via `getActionDescriptions` 
3. **Inject structured descriptions** into LLM system prompts

This enables the model to emit **JSON-encoded action calls** that the frontend can execute, with full awareness of which capabilities are available in the current scene.

## Complete Usage Example

```typescript
import { 
  getEffectiveActions, 
  getActionDescriptions 
} from '@/lib/orchestration/tool-schemas';

// Define the full action palette
const actionPool = [
  'spotlight',
  'laser', 
  'wb_open',
  'wb_draw_text',
  'wb_draw_chart',
  'play_video',
];

// Step 1: Filter for current scene context
const sceneType = 'whiteboard'; // vs. 'slide'
const effective = getEffectiveActions(actionPool, sceneType);

// Step 2: Generate prompt-ready descriptions
const actionDocs = getActionDescriptions(effective);

// Step 3: Include in system prompt sent to LLM
const systemPrompt = `
You are a multimodal presentation assistant.

Available tools:
${actionDocs}

Respond with JSON action calls to control the presentation.
`;

```

## Summary

- **`getEffectiveActions`** filters action lists by scene type, excluding `SLIDE_ONLY_ACTIONS` (spotlight, laser, etc.) in non-slide contexts
- **`getActionDescriptions`** maps allowed actions to human-readable explanations for system prompt injection
- Together these functions constitute the **agent tool interface** contract in OpenMAIC
- The implementation in [`lib/orchestration/tool-schemas.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/orchestration/tool-schemas.ts) supports 15+ distinct actions spanning whiteboard operations and media controls
- Downstream consumption in [`lib/orchestration/orchestrator.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/orchestration/orchestrator.ts) enables structured LLM output generation

## Frequently Asked Questions

### What actions are considered slide-only in OpenMAIC?

The `SLIDE_ONLY_ACTIONS` set defined in [`lib/types/action.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/types/action.ts) includes spotlight and laser pointer controls. These require an active slide context and are filtered out by `getEffectiveActions` when the scene type is not `'slide'`.

### How does the orchestrator use these tool interfaces?

The [`lib/orchestration/orchestrator.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/orchestration/orchestrator.ts) module calls `getEffectiveActions` to determine valid capabilities for the current session, then passes the filtered list to `getActionDescriptions` to generate documentation strings. These descriptions are embedded in system prompts so the LLM knows which JSON action schemas it may emit.

### Can custom actions be added to the tool-schemas system?

Yes. Adding a custom action requires: (1) defining the action identifier in your action pool, (2) adding a corresponding entry to the `descriptions` map in [`lib/orchestration/tool-schemas.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/orchestration/tool-schemas.ts), and (3) optionally updating `SLIDE_ONLY_ACTIONS` if the action is slide-specific. The existing filtering and documentation generation logic will handle the new action automatically.