# Supported Inference Types in Calliope: Complete Guide to Multimodal AI Integration

> Explore Calliope's supported inference types for multimodal AI integration including text to image audio to text and more Abstracting providers like OpenAI and Hugging Face

- Repository: [chrisimmel/calliope](https://github.com/chrisimmel/calliope)
- Tags: deep-dive
- Published: 2026-02-27

---

**Calliope supports seven distinct inference directions—text-to-text, text-to-image, image analysis, OCR, audio-to-text, messages-to-object conversion, and image-plus-text-to-video—through a unified inference package abstracting providers like OpenAI, Hugging Face, Stability AI, Azure, and Runway.**

Calliope is an open-source AI orchestration framework by chrisimmel that standardizes interactions with diverse model-as-a-service backends. Understanding the supported inference types in Calliope is essential for building applications that leverage large language models, image generators, and multimodal AI services through a consistent Python API.

## Overview of the Calliope Inference Package

The `calliope.inference` package provides a unified abstraction layer over various AI providers. All callable inference helpers are exported from [`calliope/inference/__init__.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/__init__.py) and implemented in dedicated modules for each modality. This architecture allows developers to switch between providers—such as OpenAI, Hugging Face, Stability AI, Replicate, Runway, and Azure—without changing application logic.

The public API imports shown in [`calliope/inference/__init__.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/__init__.py) include:

```python
from .image_analysis import image_analysis_inference
from .messages_to_object import messages_to_object_inference
from .text_to_image import text_to_image_file_inference
from .text_to_text import text_to_text_inference
from .text_to_video import image_and_text_to_video_file_inference

```

## Complete List of Supported Inference Types

Calliope currently implements seven primary inference directions, each optimized for specific multimodal AI tasks.

### Text-to-Text Generation

The `text_to_text_inference` function in [`calliope/inference/text_to_text.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_text.py) sends prompts to large language models and returns generated text. This helper supports providers including OpenAI, Hugging Face, and Replicate, allowing seamless switching between GPT models and open-source alternatives.

### Text-to-Image Generation

Use `text_to_image_file_inference` from [`calliope/inference/text_to_image.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_image.py) to generate images from textual descriptions. This function supports Stability AI, OpenAI DALL-E 2, and Replicate backends, writing the resulting PNG or JPEG to a specified file path.

### Image Analysis (Image-to-Text)

The `image_analysis_inference` function in [`calliope/inference/image_analysis.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/image_analysis.py) produces rich descriptions of visual content using multimodal LLMs or Azure Computer Vision. It returns comprehensive metadata including captions, tags, detected objects, and OCR text for complete image understanding.

### OCR Text Extraction

For dedicated text extraction from images, `image_ocr_inference`—also in [`calliope/inference/image_analysis.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/image_analysis.py)—calls Azure Vision OCR to extract raw textual content. This provides a lightweight alternative to full image analysis when only text recognition is required.

### Audio-to-Text Transcription

The `audio_to_text_inference` helper in [`calliope/inference/audio_to_text.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/audio_to_text.py) interfaces with OpenAI Whisper to transcribe audio files. This enables speech-to-text capabilities within the unified Calliope framework.

### Messages-to-Object Structured Output

Use `messages_to_object_inference` from [`calliope/inference/messages_to_object.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/messages_to_object.py) to convert conversation histories into domain-specific Python objects. This function leverages language models to parse chat message lists and extract structured data according to your schema.

### Image-and-Text-to-Video Generation

The `image_and_text_to_video_file_inference` function in [`calliope/inference/text_to_video.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_video.py) currently wraps Runway's Gen-4 text-to-video model. It accepts both a prompt image file and textual prompt to generate video content, extending Calliope's capabilities into generative video.

## Model Provider Configuration

Calliope determines which backend to use through a configuration registry defined in [`calliope/models/inference_model_config.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/models/inference_model_config.py). This file defines the `InferenceModelProvider` enum with values including `HuggingFace`, `OpenAI`, `Azure`, `Replicate`, `Runway`, and `Stability`.

Default configurations map inference directions to specific model IDs—for example, `"huggingface_gpt_neo_2.7B"` for text-to-text or `"stability_stable_diffusion_1.5"` for text-to-image generation. You can override these defaults by passing custom `model_config` objects to any inference function.

## Practical Usage Example

All inference functions follow an async pattern requiring an HTTPX client. Here is a complete example generating an image from text:

```python
from calliope.inference import text_to_image_file_inference
import httpx

async def generate():
    async with httpx.AsyncClient() as client:
        img_path = await text_to_image_file_inference(
            httpx_client=client,
            prompt="A futuristic city at sunset",
            output_image_filename="city.png",
            model_config=your_model_config,   # e.g. from InferenceModelConfigsModel

            keys=your_keys_model,
        )
        print("Image saved to:", img_path)

```

Each helper accepts parameters specific to its modality while maintaining consistent patterns for authentication, error handling, and output formatting across the library.

## Summary

- **Seven inference directions** are supported: text-to-text, text-to-image, image analysis, OCR, audio-to-text, messages-to-object, and image-plus-text-to-video.
- **Unified entry point**: Import all helpers from [`calliope/inference/__init__.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/__init__.py).
- **Provider abstraction**: Switch between OpenAI, Hugging Face, Stability AI, Azure, Replicate, and Runway without code changes.
- **Configuration-driven**: Model selection is handled via `InferenceModelProvider` enums in [`calliope/models/inference_model_config.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/models/inference_model_config.py).
- **Async-first API**: All functions use `httpx.AsyncClient` for non-blocking I/O.

## Frequently Asked Questions

### How do I import the inference functions in Calliope?

All public inference functions are exported from [`calliope/inference/__init__.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/__init__.py). You can import specific helpers like `text_to_text_inference` or `image_analysis_inference` directly from the `calliope.inference` namespace without referencing individual submodules.

### What providers are supported for text-to-image generation?

According to the source in [`calliope/inference/text_to_image.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_image.py), the `text_to_image_file_inference` function supports Stability AI, OpenAI DALL-E 2, and Replicate backends. The specific provider is determined by the `model_config` parameter you pass to the function.

### Can I use Azure Computer Vision with Calliope?

Yes. The `image_analysis_inference` and `image_ocr_inference` functions in [`calliope/inference/image_analysis.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/image_analysis.py) specifically support Azure Computer Vision for generating rich image descriptions and extracting OCR text, alongside multimodal LLM alternatives.

### Is audio transcription supported natively?

Yes. Calliope provides `audio_to_text_inference` in [`calliope/inference/audio_to_text.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/audio_to_text.py), which wraps OpenAI Whisper to transcribe audio files. This is the dedicated helper for speech-to-text operations within the framework.