What Inputs Can Calliope Process for Interactive Artworks?

Calliope processes four canonical input types—image, audio, text, and video—to power its AI storytelling engine, accepting them as base64-encoded snippets via the V2 API.

The open-source Calliope repository (chrisimmel/calliope) provides a multimodal storytelling framework that transforms diverse media into generative art. Whether you are building browser-based interfaces, IoT devices like the ESP32-Sparrow, or custom HTTP clients, understanding what inputs Calliope can process for interactive artworks is essential for crafting responsive, AI-driven narratives.

The Four Multimodal Input Types

Calliope's V2 API revolves around the Snippet model defined in calliope/routes/v2/models.py. Each snippet specifies one of four input types through the SnippetType enumeration, allowing clients to mix media in a single request.

Image Inputs

Image snippets accept base64-encoded pictures or URLs referencing Calliope's media store. When ingested, images route through image_analysis_inference, which employs multimodal LLMs and Azure Computer Vision (implemented in calliope/inference/engines/azure_vision.py) to generate descriptions and tags. These visual analyses then guide downstream text or image generation strategies.

Audio Inputs

Audio snippets handle base64-encoded audio clips, typically short voice recordings. Calliope processes these through audio_to_text_inference using OpenAI Whisper, as implemented in calliope/inference/engines/openai_whisper.py. The resulting transcript feeds into the story pipeline as textual guidance, enabling voice-controlled interactive artworks.

Text Inputs

Text snippets provide plain-text strings that serve direct prompts for story continuation. These route immediately to text_to_text_inference (found in calliope/inference/engines/openai_chat.py), bypassing transcription or analysis steps. Text inputs can standalone or complement other modalities to provide narrative context.

Video Inputs

Video snippets (experimental) accept base64-encoded video content or media store references. Currently, Calliope processes video through image_and_text_to_video_file_inference using the Runway engine (calliope/inference/engines/runway.py), enabling video-generation workflows within story strategies like fern.

How Calliope Transforms Inputs Into Art

The input processing pipeline follows four distinct stages:

1. Snippet Deserialization – The API parses incoming requests into AddFrameRequest objects containing lists of Snippet instances (calliope/routes/v2/models.py).

2. Inference Routing – Each snippet type maps to a specific inference function:

   • Image → image_analysis_inference (Azure Vision + multimodal LLM)
   • Audio → audio_to_text_inference (Whisper transcription)
   • Text → text_to_text_inference (direct LLM completion)
   • Video → image_and_text_to_video_file_inference (Runway generation)

3. Strategy Execution – Story strategies (located under calliope/strategies/, such as fern.py or lavender.py) consume processed snippets to orchestrate AI calls, generating new story frames that blend the input modalities.

4. Persistence and Real-Time Updates – Generated frames save to the database while Firebase pushes status updates to clients, enabling immediate UI reactions without polling.

Implementing Multimodal Inputs

Sending Mixed Snippets via Python

The following example demonstrates submitting image, audio, and text inputs simultaneously using httpx:

import httpx
import base64

# Encode local media files

with open("photo.jpg", "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode()

with open("voice.wav", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode()

payload = {
    "title": "Midnight Forest",
    "strategy": "fern",
    "snippets": [
        {"snippet_type": "image", "content": img_b64, "metadata": {}},
        {"snippet_type": "audio", "content": audio_b64, "metadata": {"duration": 2.8}},
        {"snippet_type": "text", "content": "A rustling wind through pine trees"}
    ]
}

async def create_story():
    async with httpx.AsyncClient() as client:
        resp = await client.post(
            "http://localhost:8008/v2/stories/?client_id=browser_123",
            json=payload,
            headers={"X-Api-Key": "YOUR_API_KEY"}
        )
        return resp.json()

Reference: Request schemas reside in AddFrameRequest and CreateStoryRequest (calliope/routes/v2/models.py), while endpoint logic lives in calliope/routes/v2/stories.py.

Adding Frames with curl

For lightweight integrations, add individual snippets to existing stories:

curl -X POST "http://localhost:8008/v2/stories/ck1234567890/frames/?client_id=browser_123" \
  -H "X-Api-Key: your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
        "snippets": [
          {
            "snippet_type": "text",
            "content": "The lantern flickers as the rain intensifies."
          }
        ]
      }'

Extending Requests with Custom Fields

Calliope supports experimental parameters through the extra_fields mechanism. The @root_validator in AddFrameRequest.build_extra_fields automatically captures non-standard JSON keys, making them available to custom story strategies without breaking schema validation:

{
  "snippets": [
    { "snippet_type": "text", "content": "A quiet sunrise." }
  ],
  "my_custom_flag": true
}

Summary

• Calliope accepts four input types for interactive artworks: image, audio, text, and video, defined in calliope/routes/v2/models.py.
• Image inputs analyze visual content via Azure Computer Vision; audio converts to text via OpenAI Whisper; text processes directly through LLM chat; video generates content via Runway.
• The Snippet model enables multimodal requests, allowing base64-encoded media mixing in single API calls to /v2/stories/ or /v2/stories/{id}/frames/.
• Story strategies (e.g., fern, lavender) consume processed inputs to generate AI art frames, with real-time status updates pushed via Firebase.
• The extra_fields system allows developers to pass custom parameters without modifying the core Pydantic schemas.

Frequently Asked Questions

Can Calliope process multiple input types in a single request?

Yes. The V2 API accepts arrays of Snippet objects containing any combination of image, audio, text, and video types. The create_story and request_new_frame endpoints in calliope/routes/v2/stories.py deserialize these arrays and route each snippet to its appropriate inference engine before passing processed outputs to the selected story strategy.

How does Calliope handle audio inputs for interactive artworks?

Calliope transcribes audio snippets using OpenAI Whisper through the audio_to_text_inference function in calliope/inference/engines/openai_whisper.py. The resulting text feeds into the story pipeline as if it were a text snippet, enabling voice commands or ambient audio to influence narrative generation without requiring manual transcription by the client.

What file formats does Calliope support for image inputs?

The system accepts base64-encoded image data or URLs referencing Calliope's internal media store. While the API schema in calliope/routes/v2/models.py treats content as opaque strings, the image_analysis_inference pipeline (utilizing calliope/inference/engines/azure_vision.py) typically processes standard web formats including JPEG and PNG through the Azure Computer Vision service.

Is video input fully supported in Calliope's production API?

Video input is currently experimental. The schema supports "video" as a SnippetType, and the Runway engine implementation in calliope/inference/engines/runway.py provides image_and_text_to_video_file_inference capabilities. However, video generation workflows are primarily utilized within specific story strategies like fern and may require additional configuration compared to stable image, audio, and text processing pipelines.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}