# How to Handle Multi-Modal Inputs (Images, Audio) in Ax Applications

> Learn to handle multi-modal inputs like images and audio in Ax applications. Discover how AxChatRequest processes various media or uses fallback strategies for seamless integration.

- Repository: [Ax/ax](https://github.com/ax-llm/ax)
- Tags: how-to-guide
- Published: 2026-02-25

---

**Ax applications natively support images, audio, files, and URLs through the `AxChatRequest` interface, automatically processing unsupported media types via configurable fallback strategies including degradation to text.**

The `ax-llm/ax` library provides a robust architecture for building AI applications that process diverse media types. Whether you're building a chatbot that analyzes screenshots or a voice-enabled assistant, understanding how to handle multi-modal inputs in Ax applications ensures your code gracefully adapts to provider capabilities.

## Understanding Ax's Multi-Modal Architecture

Ax centralizes media handling through type-safe interfaces that define how content flows through the system.

### The AxChatRequest Interface

The foundation of multi-modal support resides in [`src/ax/ai/types.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/ai/types.ts), where `AxChatRequest` defines the `content` property as a union type supporting text and media objects. This interface accepts an array of content items where each item specifies its type, allowing a single request to mix text, images, audio, files, and URLs.

### Supported Content Types

According to the source code in [`src/ax/ai/types.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/ai/types.ts) (lines 311-376), Ax recognizes four primary media content types:

- **`image`** – Supports base64-encoded image data with MIME type specification and optional quality optimization settings
- **`audio`** – Accepts audio data with format identifiers (e.g., `wav`, `mp3`) and optional transcription text
- **`file`** – Handles document data (PDFs, text files) with filename and MIME type metadata
- **`url`** – References external resources with title attributes for context

Each type supports both inline data (base64) and cloud-URI references where applicable.

## Processing Multi-Modal Content with axProcessContentForProvider

When a provider cannot natively handle a specific media type, Ax invokes the content processor defined in [`src/ax/ai/processor.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/ai/processor.ts).

### Fallback Strategies

The `axProcessContentForProvider` function examines the provider's declared capabilities via `provider.getFeatures()` and applies one of three configurable behaviors when encountering unsupported media:

- **`error`** – Immediately throws `AxMediaNotSupportedError`, halting execution
- **`skip`** – Silently omits the unsupported content item from the request
- **`degrade`** – Attempts conversion to plain text using available conversion services or fallback metadata like `altText` or `transcription`

### Custom Conversion Services

The `ProcessingOptions` interface (lines 10-21 in [`src/ax/ai/processor.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/ai/processor.ts)) enables developers to inject custom conversion logic:

```typescript
{
  fallbackBehavior: 'degrade',
  imageToText: async (img) => await visionService.describe(img),
  audioToText: async (aud, fmt) => await speechService.transcribe(aud, fmt),
  fileToText: async (data, mime) => await pdfExtractor.extract(data),
  urlToText: async (url) => await webScraper.fetch(url)
}

```

The main processing loop (lines 96-150) iterates through each content item, checks `features.media.<type>.supported`, and either passes the item through unchanged, extracts existing text metadata, or invokes the appropriate conversion service.

## Routing and Provider Selection

Ax provides intelligent routing capabilities to ensure multi-modal requests reach compatible providers.

### Capability Detection with axGetCompatibilityReport

Before sending requests, Ax can analyze registered AI services using `axGetCompatibilityReport` (utilized by the router in [`src/ax/ai/router.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/ai/router.ts)). This function generates a compatibility report indicating whether images, audio, files, or URLs are required by the request and which providers can handle them.

### Selecting the Optimal Provider

The router selects the best provider or constructs a fallback sequence based on the compatibility report. When no single provider supports all required media types, the system can optionally perform content conversion on-the-fly using the processor before routing to a text-only provider.

## Complete Implementation Example

The repository includes a comprehensive demonstration in [`src/examples/multi-modal-abstraction.ts`](https://github.com/ax-llm/ax/blob/main/src/examples/multi-modal-abstraction.ts) that illustrates the full workflow:

```typescript
import type { AxChatRequest } from '@ax-llm/ax';
import { axSelectOptimalProvider, axProcessContentForProvider } from '@ax-llm/ax';

// 1. Build a multi-modal request
const request: AxChatRequest = {
  chatPrompt: [
    {
      role: 'user',
      content: [
        { type: 'text', text: 'Analyze the following data:' },
        {
          type: 'image',
          mimeType: 'image/jpeg',
          image: 'base64-encoded-image-data...',
          altText: 'Bar chart of quarterly sales',
          optimize: 'quality',
        },
        {
          type: 'audio',
          data: 'base64-audio-data...',
          format: 'wav',
          transcription: 'Speaker says revenue grew 15%',
        },
        {
          type: 'file',
          data: 'base64-pdf-data...',
          filename: 'report.pdf',
          mimeType: 'application/pdf',
        },
        { type: 'url', url: 'https://example.com/market', title: 'Market report' },
      ],
    },
  ],
  capabilities: { requiresImages: true, requiresAudio: true },
};

// 2. Select provider and process content
const provider = axSelectOptimalProvider(request, availableProviders);

const processed = await axProcessContentForProvider(
  request.chatPrompt[0].content,
  provider,
  {
    fallbackBehavior: 'degrade',
    imageToText: async (img) => await visionService.describe(img),
    audioToText: async (aud, fmt) => await speechService.transcribe(aud, fmt),
    fileToText: async (data, mime) => await pdfExtractor.extract(data),
    urlToText: async (url) => await webScraper.fetch(url),
  },
);

// 3. Send the transformed request
const response = await provider.chat({
  ...request,
  chatPrompt: [{ role: 'user', content: processed }],
});

```

Running the example in [`src/examples/multi-modal-abstraction.ts`](https://github.com/ax-llm/ax/blob/main/src/examples/multi-modal-abstraction.ts) produces a step-by-step walkthrough of capability analysis, provider selection, content processing, and routing recommendations.

## Summary

- **AxChatRequest** in [`src/ax/ai/types.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/ai/types.ts) natively supports images, audio, files, and URLs alongside text content
- **axProcessContentForProvider** in [`src/ax/ai/processor.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/ai/processor.ts) automatically handles unsupported media through configurable fallback strategies (`error`, `skip`, `degrade`)
- **ProcessingOptions** allows injection of custom conversion services for vision-to-text, speech-to-text, document extraction, and web scraping
- **axSelectOptimalProvider** and compatibility reporting in [`src/ax/ai/router.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/ai/router.ts) ensure requests reach AI services capable of handling their media requirements
- The complete workflow is demonstrated in [`src/examples/multi-modal-abstraction.ts`](https://github.com/ax-llm/ax/blob/main/src/examples/multi-modal-abstraction.ts)

## Frequently Asked Questions

### What content types does Ax support for multi-modal inputs?

Ax supports **text**, **image**, **audio**, **file**, and **url** content types within a single `AxChatRequest`. Each type accepts either inline base64 data or cloud URIs, with specific metadata fields like `mimeType`, `altText`, and `transcription` to support graceful degradation when providers cannot process binary media directly.

### How does Ax handle providers that don't support images or audio?

When a provider lacks support for specific media types, Ax invokes `axProcessContentForProvider` from [`src/ax/ai/processor.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/ai/processor.ts) to apply a **fallback strategy**. The default behavior can be configured as `error` (throw `AxMediaNotSupportedError`), `skip` (remove the content), or `degrade` (convert to text using `altText`, `transcription`, or custom conversion services).

### Can I customize how images or audio are converted to text?

Yes. The `ProcessingOptions` interface in [`src/ax/ai/processor.ts`](https://github.com/ax-llm/ax/blob/main/src/ax/ai/processor.ts) accepts optional conversion callbacks: `imageToText`, `audioToText`, `fileToText`, and `urlToText`. These functions receive the raw media data and format information, allowing you to integrate external vision APIs, speech recognition services, or document parsers to generate text representations when degrading content for text-only providers.

### Where can I find a complete working example of multi-modal handling?

The repository includes a comprehensive demonstration in [`src/examples/multi-modal-abstraction.ts`](https://github.com/ax-llm/ax/blob/main/src/examples/multi-modal-abstraction.ts). This example shows the complete workflow: constructing a multi-modal `AxChatRequest`, running compatibility analysis with `axGetCompatibilityReport`, selecting the optimal provider, processing content with custom conversion services, and sending the transformed request to the AI service.