Supported Inference Types in Calliope: Complete Guide to Multimodal AI Integration

Calliope supports seven distinct inference directions—text-to-text, text-to-image, image analysis, OCR, audio-to-text, messages-to-object conversion, and image-plus-text-to-video—through a unified inference package abstracting providers like OpenAI, Hugging Face, Stability AI, Azure, and Runway.

Calliope is an open-source AI orchestration framework by chrisimmel that standardizes interactions with diverse model-as-a-service backends. Understanding the supported inference types in Calliope is essential for building applications that leverage large language models, image generators, and multimodal AI services through a consistent Python API.

Overview of the Calliope Inference Package

The calliope.inference package provides a unified abstraction layer over various AI providers. All callable inference helpers are exported from calliope/inference/__init__.py and implemented in dedicated modules for each modality. This architecture allows developers to switch between providers—such as OpenAI, Hugging Face, Stability AI, Replicate, Runway, and Azure—without changing application logic.

The public API imports shown in calliope/inference/__init__.py include:

from .image_analysis import image_analysis_inference
from .messages_to_object import messages_to_object_inference
from .text_to_image import text_to_image_file_inference
from .text_to_text import text_to_text_inference
from .text_to_video import image_and_text_to_video_file_inference

Complete List of Supported Inference Types

Calliope currently implements seven primary inference directions, each optimized for specific multimodal AI tasks.

Text-to-Text Generation

The text_to_text_inference function in calliope/inference/text_to_text.py sends prompts to large language models and returns generated text. This helper supports providers including OpenAI, Hugging Face, and Replicate, allowing seamless switching between GPT models and open-source alternatives.

Text-to-Image Generation

Use text_to_image_file_inference from calliope/inference/text_to_image.py to generate images from textual descriptions. This function supports Stability AI, OpenAI DALL-E 2, and Replicate backends, writing the resulting PNG or JPEG to a specified file path.

Image Analysis (Image-to-Text)

The image_analysis_inference function in calliope/inference/image_analysis.py produces rich descriptions of visual content using multimodal LLMs or Azure Computer Vision. It returns comprehensive metadata including captions, tags, detected objects, and OCR text for complete image understanding.

OCR Text Extraction

For dedicated text extraction from images, image_ocr_inference—also in calliope/inference/image_analysis.py—calls Azure Vision OCR to extract raw textual content. This provides a lightweight alternative to full image analysis when only text recognition is required.

Audio-to-Text Transcription

The audio_to_text_inference helper in calliope/inference/audio_to_text.py interfaces with OpenAI Whisper to transcribe audio files. This enables speech-to-text capabilities within the unified Calliope framework.

Messages-to-Object Structured Output

Use messages_to_object_inference from calliope/inference/messages_to_object.py to convert conversation histories into domain-specific Python objects. This function leverages language models to parse chat message lists and extract structured data according to your schema.

Image-and-Text-to-Video Generation

The image_and_text_to_video_file_inference function in calliope/inference/text_to_video.py currently wraps Runway's Gen-4 text-to-video model. It accepts both a prompt image file and textual prompt to generate video content, extending Calliope's capabilities into generative video.

Model Provider Configuration

Calliope determines which backend to use through a configuration registry defined in calliope/models/inference_model_config.py. This file defines the InferenceModelProvider enum with values including HuggingFace, OpenAI, Azure, Replicate, Runway, and Stability.

Default configurations map inference directions to specific model IDs—for example, "huggingface_gpt_neo_2.7B" for text-to-text or "stability_stable_diffusion_1.5" for text-to-image generation. You can override these defaults by passing custom model_config objects to any inference function.

Practical Usage Example

All inference functions follow an async pattern requiring an HTTPX client. Here is a complete example generating an image from text:

from calliope.inference import text_to_image_file_inference
import httpx

async def generate():
    async with httpx.AsyncClient() as client:
        img_path = await text_to_image_file_inference(
            httpx_client=client,
            prompt="A futuristic city at sunset",
            output_image_filename="city.png",
            model_config=your_model_config,   # e.g. from InferenceModelConfigsModel

            keys=your_keys_model,
        )
        print("Image saved to:", img_path)

Each helper accepts parameters specific to its modality while maintaining consistent patterns for authentication, error handling, and output formatting across the library.

Summary

• Seven inference directions are supported: text-to-text, text-to-image, image analysis, OCR, audio-to-text, messages-to-object, and image-plus-text-to-video.
• Unified entry point: Import all helpers from calliope/inference/__init__.py.
• Provider abstraction: Switch between OpenAI, Hugging Face, Stability AI, Azure, Replicate, and Runway without code changes.
• Configuration-driven: Model selection is handled via InferenceModelProvider enums in calliope/models/inference_model_config.py.
• Async-first API: All functions use httpx.AsyncClient for non-blocking I/O.

Frequently Asked Questions

How do I import the inference functions in Calliope?

All public inference functions are exported from calliope/inference/__init__.py. You can import specific helpers like text_to_text_inference or image_analysis_inference directly from the calliope.inference namespace without referencing individual submodules.

What providers are supported for text-to-image generation?

According to the source in calliope/inference/text_to_image.py, the text_to_image_file_inference function supports Stability AI, OpenAI DALL-E 2, and Replicate backends. The specific provider is determined by the model_config parameter you pass to the function.

Can I use Azure Computer Vision with Calliope?

Yes. The image_analysis_inference and image_ocr_inference functions in calliope/inference/image_analysis.py specifically support Azure Computer Vision for generating rich image descriptions and extracting OCR text, alongside multimodal LLM alternatives.

Is audio transcription supported natively?

Yes. Calliope provides audio_to_text_inference in calliope/inference/audio_to_text.py, which wraps OpenAI Whisper to transcribe audio files. This is the dedicated helper for speech-to-text operations within the framework.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}