# What Inputs Can Calliope Process for Interactive Artworks?

> Discover what inputs Calliope accepts for interactive art. Calliope processes image, audio, text, and video snippets via its V2 API to fuel AI storytelling.

- Repository: [chrisimmel/calliope](https://github.com/chrisimmel/calliope)
- Tags: how-to-guide
- Published: 2026-02-27

---

**Calliope processes four canonical input types—image, audio, text, and video—to power its AI storytelling engine, accepting them as base64-encoded snippets via the V2 API.**

The open-source Calliope repository (`chrisimmel/calliope`) provides a multimodal storytelling framework that transforms diverse media into generative art. Whether you are building browser-based interfaces, IoT devices like the ESP32-Sparrow, or custom HTTP clients, understanding what inputs Calliope can process for interactive artworks is essential for crafting responsive, AI-driven narratives.

## The Four Multimodal Input Types

Calliope's V2 API revolves around the `Snippet` model defined in [`calliope/routes/v2/models.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/routes/v2/models.py). Each snippet specifies one of four input types through the `SnippetType` enumeration, allowing clients to mix media in a single request.

### Image Inputs

**Image snippets** accept base64-encoded pictures or URLs referencing Calliope's media store. When ingested, images route through `image_analysis_inference`, which employs multimodal LLMs and Azure Computer Vision (implemented in [`calliope/inference/engines/azure_vision.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/azure_vision.py)) to generate descriptions and tags. These visual analyses then guide downstream text or image generation strategies.

### Audio Inputs

**Audio snippets** handle base64-encoded audio clips, typically short voice recordings. Calliope processes these through `audio_to_text_inference` using OpenAI Whisper, as implemented in [`calliope/inference/engines/openai_whisper.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_whisper.py). The resulting transcript feeds into the story pipeline as textual guidance, enabling voice-controlled interactive artworks.

### Text Inputs

**Text snippets** provide plain-text strings that serve direct prompts for story continuation. These route immediately to `text_to_text_inference` (found in [`calliope/inference/engines/openai_chat.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_chat.py)), bypassing transcription or analysis steps. Text inputs can standalone or complement other modalities to provide narrative context.

### Video Inputs

**Video snippets** (experimental) accept base64-encoded video content or media store references. Currently, Calliope processes video through `image_and_text_to_video_file_inference` using the Runway engine ([`calliope/inference/engines/runway.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/runway.py)), enabling video-generation workflows within story strategies like **fern**.

## How Calliope Transforms Inputs Into Art

The input processing pipeline follows four distinct stages:

1. **Snippet Deserialization** – The API parses incoming requests into `AddFrameRequest` objects containing lists of `Snippet` instances ([`calliope/routes/v2/models.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/routes/v2/models.py)).

2. **Inference Routing** – Each snippet type maps to a specific inference function:
   - **Image** → `image_analysis_inference` (Azure Vision + multimodal LLM)
   - **Audio** → `audio_to_text_inference` (Whisper transcription)
   - **Text** → `text_to_text_inference` (direct LLM completion)
   - **Video** → `image_and_text_to_video_file_inference` (Runway generation)

3. **Strategy Execution** – Story strategies (located under `calliope/strategies/`, such as [`fern.py`](https://github.com/chrisimmel/calliope/blob/main/fern.py) or [`lavender.py`](https://github.com/chrisimmel/calliope/blob/main/lavender.py)) consume processed snippets to orchestrate AI calls, generating new story frames that blend the input modalities.

4. **Persistence and Real-Time Updates** – Generated frames save to the database while Firebase pushes status updates to clients, enabling immediate UI reactions without polling.

## Implementing Multimodal Inputs

### Sending Mixed Snippets via Python

The following example demonstrates submitting image, audio, and text inputs simultaneously using `httpx`:

```python
import httpx
import base64

# Encode local media files

with open("photo.jpg", "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode()

with open("voice.wav", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode()

payload = {
    "title": "Midnight Forest",
    "strategy": "fern",
    "snippets": [
        {"snippet_type": "image", "content": img_b64, "metadata": {}},
        {"snippet_type": "audio", "content": audio_b64, "metadata": {"duration": 2.8}},
        {"snippet_type": "text", "content": "A rustling wind through pine trees"}
    ]
}

async def create_story():
    async with httpx.AsyncClient() as client:
        resp = await client.post(
            "http://localhost:8008/v2/stories/?client_id=browser_123",
            json=payload,
            headers={"X-Api-Key": "YOUR_API_KEY"}
        )
        return resp.json()

```

*Reference:* Request schemas reside in `AddFrameRequest` and `CreateStoryRequest` ([`calliope/routes/v2/models.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/routes/v2/models.py)), while endpoint logic lives in [`calliope/routes/v2/stories.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/routes/v2/stories.py).

### Adding Frames with curl

For lightweight integrations, add individual snippets to existing stories:

```bash
curl -X POST "http://localhost:8008/v2/stories/ck1234567890/frames/?client_id=browser_123" \
  -H "X-Api-Key: your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
        "snippets": [
          {
            "snippet_type": "text",
            "content": "The lantern flickers as the rain intensifies."
          }
        ]
      }'

```

### Extending Requests with Custom Fields

Calliope supports experimental parameters through the `extra_fields` mechanism. The `@root_validator` in `AddFrameRequest.build_extra_fields` automatically captures non-standard JSON keys, making them available to custom story strategies without breaking schema validation:

```json
{
  "snippets": [
    { "snippet_type": "text", "content": "A quiet sunrise." }
  ],
  "my_custom_flag": true
}

```

## Summary

- Calliope accepts **four input types** for interactive artworks: image, audio, text, and video, defined in [`calliope/routes/v2/models.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/routes/v2/models.py).
- **Image** inputs analyze visual content via Azure Computer Vision; **audio** converts to text via OpenAI Whisper; **text** processes directly through LLM chat; **video** generates content via Runway.
- The **Snippet** model enables multimodal requests, allowing base64-encoded media mixing in single API calls to `/v2/stories/` or `/v2/stories/{id}/frames/`.
- **Story strategies** (e.g., `fern`, `lavender`) consume processed inputs to generate AI art frames, with real-time status updates pushed via Firebase.
- The `extra_fields` system allows developers to pass custom parameters without modifying the core Pydantic schemas.

## Frequently Asked Questions

### Can Calliope process multiple input types in a single request?

Yes. The V2 API accepts arrays of `Snippet` objects containing any combination of image, audio, text, and video types. The `create_story` and `request_new_frame` endpoints in [`calliope/routes/v2/stories.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/routes/v2/stories.py) deserialize these arrays and route each snippet to its appropriate inference engine before passing processed outputs to the selected story strategy.

### How does Calliope handle audio inputs for interactive artworks?

Calliope transcribes audio snippets using OpenAI Whisper through the `audio_to_text_inference` function in [`calliope/inference/engines/openai_whisper.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/openai_whisper.py). The resulting text feeds into the story pipeline as if it were a text snippet, enabling voice commands or ambient audio to influence narrative generation without requiring manual transcription by the client.

### What file formats does Calliope support for image inputs?

The system accepts base64-encoded image data or URLs referencing Calliope's internal media store. While the API schema in [`calliope/routes/v2/models.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/routes/v2/models.py) treats content as opaque strings, the `image_analysis_inference` pipeline (utilizing [`calliope/inference/engines/azure_vision.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/azure_vision.py)) typically processes standard web formats including JPEG and PNG through the Azure Computer Vision service.

### Is video input fully supported in Calliope's production API?

Video input is currently experimental. The schema supports `"video"` as a `SnippetType`, and the Runway engine implementation in [`calliope/inference/engines/runway.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/inference/engines/runway.py) provides `image_and_text_to_video_file_inference` capabilities. However, video generation workflows are primarily utilized within specific story strategies like **fern** and may require additional configuration compared to stable image, audio, and text processing pipelines.