Agent Tool Interfaces in OpenMAIC: How `tool-schemas.ts` Shapes LLM Action Selection

The lib/orchestration/tool-schemas.ts module defines two primary agent tool interfaces—getEffectiveActions and getActionDescriptions—that filter available actions by context and generate human-readable descriptions for system prompts.

The OpenMAIC repository provides a multimodal agent orchestration framework where language models control presentation tools through structured JSON outputs. The lib/orchestration/tool-schemas.ts file implements the agent tool interfaces that determine which actions an LLM can invoke and how those capabilities are communicated to the model. These interfaces bridge the gap between raw action definitions and context-aware prompt engineering.

The Two Core Agent Tool Interfaces

The module exports two complementary functions that work together to shape the agent's tool set at runtime:

getEffectiveActions: Context-Aware Action Filtering

This function filters a list of allowed action identifiers based on the current scene type (slide versus non-slide contexts). It is defined starting at line 16 in lib/orchestration/tool-schemas.ts.

Implementation logic:

• Accepts an array of action names and an optional sceneType parameter
• If the scene is not a slide, removes actions belonging to the SLIDE_ONLY_ACTIONS set (spotlight, laser, and similar slide-specific tools)
• Returns the filtered array of context-appropriate actions

import { getEffectiveActions } from '@/lib/orchestration/tool-schemas';

const allActions = [
  'spotlight',      // Slide-only
  'laser',          // Slide-only  
  'wb_open',        // Universal
  'wb_draw_text',   // Universal
  'play_video',     // Universal
];

// Non-slide context filters out slide-only actions
const usable = getEffectiveActions(allActions, 'whiteboard');
// Result: ['wb_open', 'wb_draw_text', 'play_video']

The SLIDE_ONLY_ACTIONS constant imported from lib/types/action.ts drives this filtering logic. This ensures agents receive only actions that are meaningful for the active context, preventing the model from attempting unavailable operations.

getActionDescriptions: Prompt-Ready Action Documentation

This function generates human-readable descriptions of each permitted action for inclusion in system prompts. It is implemented across lines 29-72 in lib/orchestration/tool-schemas.ts.

Implementation structure:

• A descriptions map enumerates every supported action with explanatory text
• Builds a bullet-point list of descriptions for actions in the supplied allowedActions array
• Falls back to a generic speech-only message when no actions are permitted

Supported actions in the descriptions map include:

• spotlight – Presentation spotlight control
• laser – Laser pointer activation
• wb_open – Open whiteboard for hand-drawn explanations
• wb_draw_text – Add text annotations
• wb_draw_shape – Insert geometric shapes
• wb_draw_chart – Create data visualizations
• wb_draw_latex – Render mathematical expressions
• wb_draw_table – Insert tabular content
• wb_draw_line – Freeform line drawing
• wb_draw_code – Display formatted code blocks
• wb_edit_code – Modify existing code elements
• wb_clear, wb_delete, wb_close – Whiteboard management
• play_video – Video playback control

import { getActionDescriptions } from '@/lib/orchestration/tool-schemas';

const allowedActions = ['wb_open', 'wb_draw_text', 'play_video'];

const promptFragment = getActionDescriptions(allowedActions);
// Returns formatted string:
// - wb_open: Open the whiteboard for hand-drawn explanations...
// - wb_draw_text: Add text to the whiteboard...
// - play_video: Start playback of a video element on the current slide...

Integration with the Orchestration Layer

The lib/orchestration/orchestrator.ts file demonstrates how these agent tool interfaces are consumed in practice. The orchestration layer uses both functions to:

1. Determine contextually valid actions via getEffectiveActions
2. Format action documentation via getActionDescriptions
3. Inject structured descriptions into LLM system prompts

This enables the model to emit JSON-encoded action calls that the frontend can execute, with full awareness of which capabilities are available in the current scene.

Complete Usage Example

import { 
  getEffectiveActions, 
  getActionDescriptions 
} from '@/lib/orchestration/tool-schemas';

// Define the full action palette
const actionPool = [
  'spotlight',
  'laser', 
  'wb_open',
  'wb_draw_text',
  'wb_draw_chart',
  'play_video',
];

// Step 1: Filter for current scene context
const sceneType = 'whiteboard'; // vs. 'slide'
const effective = getEffectiveActions(actionPool, sceneType);

// Step 2: Generate prompt-ready descriptions
const actionDocs = getActionDescriptions(effective);

// Step 3: Include in system prompt sent to LLM
const systemPrompt = `
You are a multimodal presentation assistant.

Available tools:
${actionDocs}

Respond with JSON action calls to control the presentation.
`;

Summary

• getEffectiveActions filters action lists by scene type, excluding SLIDE_ONLY_ACTIONS (spotlight, laser, etc.) in non-slide contexts
• getActionDescriptions maps allowed actions to human-readable explanations for system prompt injection
• Together these functions constitute the agent tool interface contract in OpenMAIC
• The implementation in lib/orchestration/tool-schemas.ts supports 15+ distinct actions spanning whiteboard operations and media controls
• Downstream consumption in lib/orchestration/orchestrator.ts enables structured LLM output generation

Frequently Asked Questions

What actions are considered slide-only in OpenMAIC?

The SLIDE_ONLY_ACTIONS set defined in lib/types/action.ts includes spotlight and laser pointer controls. These require an active slide context and are filtered out by getEffectiveActions when the scene type is not 'slide'.

How does the orchestrator use these tool interfaces?

The lib/orchestration/orchestrator.ts module calls getEffectiveActions to determine valid capabilities for the current session, then passes the filtered list to getActionDescriptions to generate documentation strings. These descriptions are embedded in system prompts so the LLM knows which JSON action schemas it may emit.

Can custom actions be added to the tool-schemas system?

Yes. Adding a custom action requires: (1) defining the action identifier in your action pool, (2) adding a corresponding entry to the descriptions map in lib/orchestration/tool-schemas.ts, and (3) optionally updating SLIDE_ONLY_ACTIONS if the action is slide-specific. The existing filtering and documentation generation logic will handle the new action automatically.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}