How to Handle Multi-Modal Inputs (Images, Audio) in Ax Applications

Ax applications natively support images, audio, files, and URLs through the AxChatRequest interface, automatically processing unsupported media types via configurable fallback strategies including degradation to text.

The ax-llm/ax library provides a robust architecture for building AI applications that process diverse media types. Whether you're building a chatbot that analyzes screenshots or a voice-enabled assistant, understanding how to handle multi-modal inputs in Ax applications ensures your code gracefully adapts to provider capabilities.

Understanding Ax's Multi-Modal Architecture

Ax centralizes media handling through type-safe interfaces that define how content flows through the system.

The AxChatRequest Interface

The foundation of multi-modal support resides in src/ax/ai/types.ts, where AxChatRequest defines the content property as a union type supporting text and media objects. This interface accepts an array of content items where each item specifies its type, allowing a single request to mix text, images, audio, files, and URLs.

Supported Content Types

According to the source code in src/ax/ai/types.ts (lines 311-376), Ax recognizes four primary media content types:

• image – Supports base64-encoded image data with MIME type specification and optional quality optimization settings
• audio – Accepts audio data with format identifiers (e.g., wav, mp3) and optional transcription text
• file – Handles document data (PDFs, text files) with filename and MIME type metadata
• url – References external resources with title attributes for context

Each type supports both inline data (base64) and cloud-URI references where applicable.

Processing Multi-Modal Content with axProcessContentForProvider

When a provider cannot natively handle a specific media type, Ax invokes the content processor defined in src/ax/ai/processor.ts.

Fallback Strategies

The axProcessContentForProvider function examines the provider's declared capabilities via provider.getFeatures() and applies one of three configurable behaviors when encountering unsupported media:

• error – Immediately throws AxMediaNotSupportedError, halting execution
• skip – Silently omits the unsupported content item from the request
• degrade – Attempts conversion to plain text using available conversion services or fallback metadata like altText or transcription

Custom Conversion Services

The ProcessingOptions interface (lines 10-21 in src/ax/ai/processor.ts) enables developers to inject custom conversion logic:

{
  fallbackBehavior: 'degrade',
  imageToText: async (img) => await visionService.describe(img),
  audioToText: async (aud, fmt) => await speechService.transcribe(aud, fmt),
  fileToText: async (data, mime) => await pdfExtractor.extract(data),
  urlToText: async (url) => await webScraper.fetch(url)
}

The main processing loop (lines 96-150) iterates through each content item, checks features.media.<type>.supported, and either passes the item through unchanged, extracts existing text metadata, or invokes the appropriate conversion service.

Routing and Provider Selection

Ax provides intelligent routing capabilities to ensure multi-modal requests reach compatible providers.

Capability Detection with axGetCompatibilityReport

Before sending requests, Ax can analyze registered AI services using axGetCompatibilityReport (utilized by the router in src/ax/ai/router.ts). This function generates a compatibility report indicating whether images, audio, files, or URLs are required by the request and which providers can handle them.

Selecting the Optimal Provider

The router selects the best provider or constructs a fallback sequence based on the compatibility report. When no single provider supports all required media types, the system can optionally perform content conversion on-the-fly using the processor before routing to a text-only provider.

Complete Implementation Example

The repository includes a comprehensive demonstration in src/examples/multi-modal-abstraction.ts that illustrates the full workflow:

import type { AxChatRequest } from '@ax-llm/ax';
import { axSelectOptimalProvider, axProcessContentForProvider } from '@ax-llm/ax';

// 1. Build a multi-modal request
const request: AxChatRequest = {
  chatPrompt: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'Analyze the following data:' },
        {
          type: 'image',
          mimeType: 'image/jpeg',
          image: 'base64-encoded-image-data...',
          altText: 'Bar chart of quarterly sales',
          optimize: 'quality',
        },
        {
          type: 'audio',
          data: 'base64-audio-data...',
          format: 'wav',
          transcription: 'Speaker says revenue grew 15%',
        },
        {
          type: 'file',
          data: 'base64-pdf-data...',
          filename: 'report.pdf',
          mimeType: 'application/pdf',
        },
        { type: 'url', url: 'https://example.com/market', title: 'Market report' },
      ],
    },
  ],
  capabilities: { requiresImages: true, requiresAudio: true },
};

// 2. Select provider and process content
const provider = axSelectOptimalProvider(request, availableProviders);

const processed = await axProcessContentForProvider(
  request.chatPrompt[0].content,
  provider,
  {
    fallbackBehavior: 'degrade',
    imageToText: async (img) => await visionService.describe(img),
    audioToText: async (aud, fmt) => await speechService.transcribe(aud, fmt),
    fileToText: async (data, mime) => await pdfExtractor.extract(data),
    urlToText: async (url) => await webScraper.fetch(url),
  },
);

// 3. Send the transformed request
const response = await provider.chat({
  ...request,
  chatPrompt: [{ role: 'user', content: processed }],
});

Running the example in src/examples/multi-modal-abstraction.ts produces a step-by-step walkthrough of capability analysis, provider selection, content processing, and routing recommendations.

Summary

• AxChatRequest in src/ax/ai/types.ts natively supports images, audio, files, and URLs alongside text content
• axProcessContentForProvider in src/ax/ai/processor.ts automatically handles unsupported media through configurable fallback strategies (error, skip, degrade)
• ProcessingOptions allows injection of custom conversion services for vision-to-text, speech-to-text, document extraction, and web scraping
• axSelectOptimalProvider and compatibility reporting in src/ax/ai/router.ts ensure requests reach AI services capable of handling their media requirements
• The complete workflow is demonstrated in src/examples/multi-modal-abstraction.ts

Frequently Asked Questions

What content types does Ax support for multi-modal inputs?

Ax supports text, image, audio, file, and url content types within a single AxChatRequest. Each type accepts either inline base64 data or cloud URIs, with specific metadata fields like mimeType, altText, and transcription to support graceful degradation when providers cannot process binary media directly.

How does Ax handle providers that don't support images or audio?

When a provider lacks support for specific media types, Ax invokes axProcessContentForProvider from src/ax/ai/processor.ts to apply a fallback strategy. The default behavior can be configured as error (throw AxMediaNotSupportedError), skip (remove the content), or degrade (convert to text using altText, transcription, or custom conversion services).

Can I customize how images or audio are converted to text?

Yes. The ProcessingOptions interface in src/ax/ai/processor.ts accepts optional conversion callbacks: imageToText, audioToText, fileToText, and urlToText. These functions receive the raw media data and format information, allowing you to integrate external vision APIs, speech recognition services, or document parsers to generate text representations when degrading content for text-only providers.

Where can I find a complete working example of multi-modal handling?

The repository includes a comprehensive demonstration in src/examples/multi-modal-abstraction.ts. This example shows the complete workflow: constructing a multi-modal AxChatRequest, running compatibility analysis with axGetCompatibilityReport, selecting the optimal provider, processing content with custom conversion services, and sending the transformed request to the AI service.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}