# What AI Tools Does Calliope Use for Creating Interactive Art?

> Discover the AI tools Calliope uses for interactive art creation. Explore its multi-provider stack featuring Stability AI, OpenAI, Runway, and more.

- Repository: [chrisimmel/calliope](https://github.com/chrisimmel/calliope)
- Tags: deep-dive
- Published: 2026-02-27

---

**Calliope utilizes a modular multi-provider AI stack—including Stability AI, OpenAI, Replicate, Hugging Face, Runway, and Azure—to generate images, analyze visual content, produce text, and create videos for interactive art experiences.**

The open-source Calliope project (`chrisimmel/calliope`) is a creative engine designed for interactive storytelling and generative art. At its core, the system dynamically selects the appropriate AI provider at runtime based on the `InferenceModelProvider` stored in the database, enabling flexible, plug-and-play inference across multiple state-of-the-art models.

## Core AI Providers and Capabilities

### Text-to-Image Generation

Calliope supports four major providers for converting text prompts into images:

- **Stability AI** – Implements Stable Diffusion via the REST API in [[`calliope/inference/engines/stability_image.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/stability_image.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/stability_image.py).
- **OpenAI** – Supports DALL-E 2, DALL-E 3, and the newer `gpt-image-1` model through [[`calliope/inference/engines/openai_image.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_image.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_image.py).
- **Replicate** – Hosts community models including Flux and various Stable Diffusion variants in [[`calliope/inference/engines/replicate.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/replicate.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/replicate.py).
- **Hugging Face** – Provides access to Stable Diffusion, DreamStudio, and other diffusion models via [[`calliope/inference/engines/hugging_face.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/hugging_face.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/hugging_face.py).

### Image Analysis and Vision

For analyzing visual content and extracting structured descriptions, Calliope integrates:

- **OpenAI GPT-4o Vision** – Processes images and returns JSON-structured descriptions of people, objects, and text fragments. Implemented in [[`calliope/inference/engines/openai_image.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_image.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_image.py).
- **Azure Computer Vision** – Supports API versions 3 and 4 for image captioning and analysis via [[`calliope/inference/engines/azure_vision.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/azure_vision.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/azure_vision.py).
- **Replicate Multimodal Models** – Hosts MiniGPT-4 and LLaVA-13B for vision-language tasks in [[`calliope/inference/engines/replicate.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/replicate.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/replicate.py).

### Text-to-Video Generation

Calliope currently routes video generation exclusively to **Runway** for models including Gen-4 turbo. The implementation in [[`calliope/inference/engines/runway.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/runway.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/runway.py) accepts both text prompts and reference images to generate short video clips.

### Large Language Models for Text

For narrative generation, dialogue, and text completion, Calliope supports:

- **OpenAI** – GPT-4o, GPT-3.5-Turbo, and legacy models via [[`calliope/inference/engines/openai_text.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_text.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_text.py).
- **Hugging Face** – Open-source LLMs including Llama-2 through [[`calliope/inference/engines/hugging_face.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/hugging_face.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/hugging_face.py).
- **Replicate** – Hosted LLMs on the Replicate platform via [[`calliope/inference/engines/replicate.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/replicate.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/replicate.py).

## Architecture and Provider Dispatch System

Calliope’s architecture decouples model configuration from execution through a dynamic dispatch system.

### Model Configuration

The `ModelConfig` table (defined in [`calliope/tables/model_config.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/tables/model_config.py)) stores:
- The selected `InferenceModelProvider` enum value
- The concrete `provider_model_name` (e.g., `"gpt-4o"`, `"stability-sdxl"`)
- Provider-specific parameters as JSON blobs

### Provider Dispatch Logic

High-level wrapper functions read the `model.provider` field and delegate to the appropriate engine module:

- **Image Generation** – `text_to_image_file_inference` in [[`calliope/inference/text_to_image.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_image.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_image.py) (lines 56-86) routes to Stability, OpenAI, Replicate, or Hugging Face based on the provider enum.
- **Text Generation** – `text_to_text_inference` in [[`calliope/inference/text_to_text.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_text.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_text.py) (lines 34-52) selects the appropriate LLM engine.
- **Video Generation** – `image_and_text_to_video_file_inference` in [[`calliope/inference/text_to_video.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_video.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_video.py) (lines 39-52) currently routes exclusively to Runway.

### Error Handling and Content Safety

The system implements robust error handling and content filtering. If a provider rejects a prompt due to safety filters, `text_to_image_file_inference` catches the exception, invokes an internal "cleaner" LLM via `censor_text` to sanitize the prompt, and retries up to three times (see lines 119-133 in [`calliope/inference/text_to_image.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_image.py)).

### Utility Modules

- [[`calliope/utils/file.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/utils/file.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/utils/file.py) – Handles base64 encoding/decoding of binary assets for API transmission.
- [[`calliope/utils/piccolo.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/utils/piccolo.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/utils/piccolo.py) – Loads JSON-encoded parameter blobs from Piccolo ORM models.

## Implementation Examples

### Generating Images from Text Prompts

The following example demonstrates how Calliope dispatches to any configured image provider:

```python
import httpx
from calliope.models import KeysModel
from calliope.tables import ModelConfig
from calliope.inference.text_to_image import text_to_image_file_inference

async def create_image():
    async with httpx.AsyncClient() as client:
        # Load API keys from the database

        keys = await KeysModel.objects().first()
        
        # Load model configuration (e.g., Stability SDXL)

        model_cfg = await ModelConfig.objects().where(
            ModelConfig.slug == "stability-sdxl"
        ).first()

        output_file = "output.png"
        prompt = "A futuristic cityscape at sunset, cinematic lighting"
        
        image_path = await text_to_image_file_inference(
            httpx_client=client,
            text=prompt,
            output_image_filename=output_file,
            model_config=model_cfg,
            keys=keys,
        )
        print(f"Image saved to {image_path}")

```

The dispatcher automatically selects the correct engine based on `model_cfg.model.provider`.

### Analyzing Images with GPT-4o Vision

To extract structured descriptions from images:

```python
from calliope.inference.engines.openai_image import openai_vision_inference_ext

async def describe_image():
    async with httpx.AsyncClient() as client:
        keys = await KeysModel.objects().first()
        model_cfg = await ModelConfig.objects().where(
            ModelConfig.slug == "gpt-4o-vision"
        ).first()

        description = await openai_vision_inference_ext(
            httpx_client=client,
            image_file="output.png",
            b64_encoded_image=None,
            model_config=model_cfg,
            keys=keys,
        )
        print(description)  # JSON with people, objects, text fragments

```

### Generating Video from Images and Text

For creating short video clips using Runway:

```python
from calliope.inference.text_to_video import image_and_text_to_video_file_inference

async def make_video():
    async with httpx.AsyncClient() as client:
        keys = await KeysModel.objects().first()
        model_cfg = await ModelConfig.objects().where(
            ModelConfig.slug == "runway-gen4-turbo"
        ).first()

        video_path = await image_and_text_to_video_file_inference(
            httpx_client=client,
            prompt_image_file="output.png",
            prompt_text="A soaring dragon over a misty mountain range, sunrise",
            output_video_filename="dragon.mp4",
            model_config=model_cfg,
            keys=keys,
        )
        print(f"Video saved to {video_path}")

```

## Key Source Files

| File | Role |
|------|------|
| [[`calliope/inference/text_to_image.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_image.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_image.py) | Dispatcher for image generation across providers (lines 56-86) |
| [[`calliope/inference/text_to_text.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_text.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_text.py) | Dispatcher for LLM-based text generation (lines 34-52) |
| [[`calliope/inference/text_to_video.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_video.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_video.py) | Dispatcher for video generation, currently routing to Runway (lines 39-52) |
| [[`calliope/inference/engines/stability_image.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/stability_image.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/stability_image.py) | Stable Diffusion via Stability REST API |
| [[`calliope/inference/engines/openai_image.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_image.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_image.py) | DALL-E 3, gpt-image-1, and GPT-4o Vision |
| [[`calliope/inference/engines/replicate.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/replicate.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/replicate.py) | Community models including Flux and LLaVA |
| [[`calliope/inference/engines/runway.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/runway.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/runway.py) | RunwayML Gen-4 text-to-video |
| [[`calliope/inference/engines/azure_vision.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/azure_vision.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/azure_vision.py) | Azure Computer Vision API v3/v4 |
| [[`calliope/inference/engines/openai_text.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_text.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_text.py) | GPT-4o and GPT-3.5-Turbo for text generation |
| [[`calliope/utils/file.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/utils/file.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/utils/file.py) | Base64 encoding/decoding for binary assets |
| [[`calliope/utils/piccolo.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/utils/piccolo.py)](https://github.com/chrisimmel/calliope/blob/main/calliope/utils/piccolo.py) | JSON parameter loading from Piccolo ORM |

## Summary

- Calliope employs a **pluggable inference architecture** that dynamically routes requests to Stability AI, OpenAI, Replicate, Hugging Face, Runway, or Azure based on the configured `InferenceModelProvider`.
- The system supports **four primary media modalities**: text-to-image, image-to-text (vision), text-to-text (LLM), and text-to-video.
- **Provider dispatch logic** resides in [`calliope/inference/text_to_image.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_image.py), [`text_to_text.py`](https://github.com/chrisimmel/calliope/blob/main/text_to_text.py), and [`text_to_video.py`](https://github.com/chrisimmel/calliope/blob/main/text_to_video.py), enabling seamless switching between engines without changing client code.
- Built-in **safety mechanisms** automatically censor and retry prompts up to three times when providers reject content due to safety filters.

## Frequently Asked Questions

### What AI providers does Calliope support for image generation?

Calliope supports four primary providers for text-to-image generation: **Stability AI** (Stable Diffusion), **OpenAI** (DALL-E 2, DALL-E 3, and gpt-image-1), **Replicate** (Flux and community models), and **Hugging Face** (Stable Diffusion and DreamStudio). The specific provider is determined at runtime by the `InferenceModelProvider` enum stored in the `ModelConfig` table.

### How does Calliope handle AI provider failures or content filtering?

If a provider rejects a prompt due to safety filters or other errors, the `text_to_image_file_inference` function in [`calliope/inference/text_to_image.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_image.py) catches the exception and invokes an internal `censor_text` function to sanitize the prompt. The system automatically retries the request up to three times with the cleaned prompt before failing permanently.

### Can Calliope generate video content?

Yes, Calliope supports text-to-video generation through **RunwayML**, specifically using models like Gen-4 turbo. The `image_and_text_to_video_file_inference` function in [`calliope/inference/text_to_video.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_video.py) handles dispatching to the Runway engine, accepting both text prompts and reference images to generate short video clips.

### What file handles the dispatch logic between different AI providers?

Provider dispatch logic is centralized in three main files: [`calliope/inference/text_to_image.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_image.py) (lines 56-86) for image generation, [`calliope/inference/text_to_text.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_text.py) (lines 34-52) for language models, and [`calliope/inference/text_to_video.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_video.py) (lines 39-52) for video generation. These wrappers read the `model.provider` field from the database and route requests to the appropriate engine module.