What AI Tools Does Calliope Use for Creating Interactive Art?

Calliope utilizes a modular multi-provider AI stack—including Stability AI, OpenAI, Replicate, Hugging Face, Runway, and Azure—to generate images, analyze visual content, produce text, and create videos for interactive art experiences.

The open-source Calliope project (chrisimmel/calliope) is a creative engine designed for interactive storytelling and generative art. At its core, the system dynamically selects the appropriate AI provider at runtime based on the InferenceModelProvider stored in the database, enabling flexible, plug-and-play inference across multiple state-of-the-art models.

Core AI Providers and Capabilities

Text-to-Image Generation

Calliope supports four major providers for converting text prompts into images:

• Stability AI – Implements Stable Diffusion via the REST API in [calliope/inference/engines/stability_image.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/stability_image.py).
• OpenAI – Supports DALL-E 2, DALL-E 3, and the newer gpt-image-1 model through [calliope/inference/engines/openai_image.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_image.py).
• Replicate – Hosts community models including Flux and various Stable Diffusion variants in [calliope/inference/engines/replicate.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/replicate.py).
• Hugging Face – Provides access to Stable Diffusion, DreamStudio, and other diffusion models via [calliope/inference/engines/hugging_face.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/hugging_face.py).

Image Analysis and Vision

For analyzing visual content and extracting structured descriptions, Calliope integrates:

• OpenAI GPT-4o Vision – Processes images and returns JSON-structured descriptions of people, objects, and text fragments. Implemented in [calliope/inference/engines/openai_image.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_image.py).
• Azure Computer Vision – Supports API versions 3 and 4 for image captioning and analysis via [calliope/inference/engines/azure_vision.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/azure_vision.py).
• Replicate Multimodal Models – Hosts MiniGPT-4 and LLaVA-13B for vision-language tasks in [calliope/inference/engines/replicate.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/replicate.py).

Text-to-Video Generation

Calliope currently routes video generation exclusively to Runway for models including Gen-4 turbo. The implementation in [calliope/inference/engines/runway.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/runway.py) accepts both text prompts and reference images to generate short video clips.

Large Language Models for Text

For narrative generation, dialogue, and text completion, Calliope supports:

• OpenAI – GPT-4o, GPT-3.5-Turbo, and legacy models via [calliope/inference/engines/openai_text.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_text.py).
• Hugging Face – Open-source LLMs including Llama-2 through [calliope/inference/engines/hugging_face.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/hugging_face.py).
• Replicate – Hosted LLMs on the Replicate platform via [calliope/inference/engines/replicate.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/replicate.py).

Architecture and Provider Dispatch System

Calliope’s architecture decouples model configuration from execution through a dynamic dispatch system.

Model Configuration

The ModelConfig table (defined in calliope/tables/model_config.py) stores:

• The selected InferenceModelProvider enum value
• The concrete provider_model_name (e.g., "gpt-4o", "stability-sdxl")
• Provider-specific parameters as JSON blobs

Provider Dispatch Logic

High-level wrapper functions read the model.provider field and delegate to the appropriate engine module:

• Image Generation – text_to_image_file_inference in [calliope/inference/text_to_image.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_image.py) (lines 56-86) routes to Stability, OpenAI, Replicate, or Hugging Face based on the provider enum.
• Text Generation – text_to_text_inference in [calliope/inference/text_to_text.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_text.py) (lines 34-52) selects the appropriate LLM engine.
• Video Generation – image_and_text_to_video_file_inference in [calliope/inference/text_to_video.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_video.py) (lines 39-52) currently routes exclusively to Runway.

Error Handling and Content Safety

The system implements robust error handling and content filtering. If a provider rejects a prompt due to safety filters, text_to_image_file_inference catches the exception, invokes an internal "cleaner" LLM via censor_text to sanitize the prompt, and retries up to three times (see lines 119-133 in calliope/inference/text_to_image.py).

Utility Modules

• [calliope/utils/file.py](https://github.com/chrisimmel/calliope/blob/main/calliope/utils/file.py) – Handles base64 encoding/decoding of binary assets for API transmission.
• [calliope/utils/piccolo.py](https://github.com/chrisimmel/calliope/blob/main/calliope/utils/piccolo.py) – Loads JSON-encoded parameter blobs from Piccolo ORM models.

Implementation Examples

Generating Images from Text Prompts

The following example demonstrates how Calliope dispatches to any configured image provider:

import httpx
from calliope.models import KeysModel
from calliope.tables import ModelConfig
from calliope.inference.text_to_image import text_to_image_file_inference

async def create_image():
    async with httpx.AsyncClient() as client:
        # Load API keys from the database

        keys = await KeysModel.objects().first()
        
        # Load model configuration (e.g., Stability SDXL)

        model_cfg = await ModelConfig.objects().where(
            ModelConfig.slug == "stability-sdxl"
        ).first()

        output_file = "output.png"
        prompt = "A futuristic cityscape at sunset, cinematic lighting"
        
        image_path = await text_to_image_file_inference(
            httpx_client=client,
            text=prompt,
            output_image_filename=output_file,
            model_config=model_cfg,
            keys=keys,
        )
        print(f"Image saved to {image_path}")

The dispatcher automatically selects the correct engine based on model_cfg.model.provider.

Analyzing Images with GPT-4o Vision

To extract structured descriptions from images:

from calliope.inference.engines.openai_image import openai_vision_inference_ext

async def describe_image():
    async with httpx.AsyncClient() as client:
        keys = await KeysModel.objects().first()
        model_cfg = await ModelConfig.objects().where(
            ModelConfig.slug == "gpt-4o-vision"
        ).first()

        description = await openai_vision_inference_ext(
            httpx_client=client,
            image_file="output.png",
            b64_encoded_image=None,
            model_config=model_cfg,
            keys=keys,
        )
        print(description)  # JSON with people, objects, text fragments

Generating Video from Images and Text

For creating short video clips using Runway:

from calliope.inference.text_to_video import image_and_text_to_video_file_inference

async def make_video():
    async with httpx.AsyncClient() as client:
        keys = await KeysModel.objects().first()
        model_cfg = await ModelConfig.objects().where(
            ModelConfig.slug == "runway-gen4-turbo"
        ).first()

        video_path = await image_and_text_to_video_file_inference(
            httpx_client=client,
            prompt_image_file="output.png",
            prompt_text="A soaring dragon over a misty mountain range, sunrise",
            output_video_filename="dragon.mp4",
            model_config=model_cfg,
            keys=keys,
        )
        print(f"Video saved to {video_path}")

Key Source Files

File	Role
[calliope/inference/text_to_image.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_image.py)	Dispatcher for image generation across providers (lines 56-86)
[calliope/inference/text_to_text.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_text.py)	Dispatcher for LLM-based text generation (lines 34-52)
[calliope/inference/text_to_video.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/text_to_video.py)	Dispatcher for video generation, currently routing to Runway (lines 39-52)
[calliope/inference/engines/stability_image.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/stability_image.py)	Stable Diffusion via Stability REST API
[calliope/inference/engines/openai_image.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_image.py)	DALL-E 3, gpt-image-1, and GPT-4o Vision
[calliope/inference/engines/replicate.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/replicate.py)	Community models including Flux and LLaVA
[calliope/inference/engines/runway.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/runway.py)	RunwayML Gen-4 text-to-video
[calliope/inference/engines/azure_vision.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/azure_vision.py)	Azure Computer Vision API v3/v4
[calliope/inference/engines/openai_text.py](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_text.py)	GPT-4o and GPT-3.5-Turbo for text generation
[calliope/utils/file.py](https://github.com/chrisimmel/calliope/blob/main/calliope/utils/file.py)	Base64 encoding/decoding for binary assets
[calliope/utils/piccolo.py](https://github.com/chrisimmel/calliope/blob/main/calliope/utils/piccolo.py)	JSON parameter loading from Piccolo ORM

Summary

• Calliope employs a pluggable inference architecture that dynamically routes requests to Stability AI, OpenAI, Replicate, Hugging Face, Runway, or Azure based on the configured InferenceModelProvider.
• The system supports four primary media modalities: text-to-image, image-to-text (vision), text-to-text (LLM), and text-to-video.
• Provider dispatch logic resides in calliope/inference/text_to_image.py, text_to_text.py, and text_to_video.py, enabling seamless switching between engines without changing client code.
• Built-in safety mechanisms automatically censor and retry prompts up to three times when providers reject content due to safety filters.

Frequently Asked Questions

What AI providers does Calliope support for image generation?

Calliope supports four primary providers for text-to-image generation: Stability AI (Stable Diffusion), OpenAI (DALL-E 2, DALL-E 3, and gpt-image-1), Replicate (Flux and community models), and Hugging Face (Stable Diffusion and DreamStudio). The specific provider is determined at runtime by the InferenceModelProvider enum stored in the ModelConfig table.

How does Calliope handle AI provider failures or content filtering?

If a provider rejects a prompt due to safety filters or other errors, the text_to_image_file_inference function in calliope/inference/text_to_image.py catches the exception and invokes an internal censor_text function to sanitize the prompt. The system automatically retries the request up to three times with the cleaned prompt before failing permanently.

Can Calliope generate video content?

Yes, Calliope supports text-to-video generation through RunwayML, specifically using models like Gen-4 turbo. The image_and_text_to_video_file_inference function in calliope/inference/text_to_video.py handles dispatching to the Runway engine, accepting both text prompts and reference images to generate short video clips.

What file handles the dispatch logic between different AI providers?

Provider dispatch logic is centralized in three main files: calliope/inference/text_to_image.py (lines 56-86) for image generation, calliope/inference/text_to_text.py (lines 34-52) for language models, and calliope/inference/text_to_video.py (lines 39-52) for video generation. These wrappers read the model.provider field from the database and route requests to the appropriate engine module.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}