How the Calliope Inference Engine Architecture Works: A 4-Layer Provider-Agnostic Pipeline

Calliope implements a provider-agnostic inference pipeline that routes multimodal requests through a unified Python API by combining a runtime model registry, persistent database configuration, strategy-based dispatch, and provider-specific engines.

The Calliope inference engine architecture powers the chrisimmel/calliope repository's multimodal story generation capabilities. This design abstracts away provider-specific implementation details, allowing developers to switch between OpenAI, Stability AI, Runway, and other providers without changing application code.

The Four Layers of the Calliope Inference Engine Architecture

1. Model Registry

The foundation of the architecture is the model registry defined in calliope/models/inference_model_config.py. This module declares the InferenceModelProvider enum (covering HuggingFace, Stability, OpenAI, Azure, Replicate, and Runway) and the InferenceModelProviderVariant enum for API flavors.

The registry itself is the _model_configs_by_name dictionary, which maps logical model names to InferenceModelConfigModel instances. Each entry specifies:

• provider – The hosting service.
• provider_variant – Optional API flavor (e.g., OpenAI chat vs. completion).
• provider_model_name – The exact identifier the provider expects.
• parameters – Default request parameters (temperature, steps, etc.).

# Example entry from inference_model_config.py

"stability_stable_diffusion_1.5": InferenceModelConfigModel(
    provider=InferenceModelProvider.STABILITY,
    provider_model_name="stable-diffusion-v1-5",
    parameters={"steps": 30, "seed": 0, "cfg_scale": 7.0},
)

The helper function load_model_configs() builds an InferenceModelConfigsModel that the application injects at startup.

2. Persistent Model Configuration

While the registry defines what can be used, calliope/tables/model_config.py persists what is being used. This layer uses Piccolo ORM tables to store per-deployment selections:

• InferenceModel – The database representation of a registry entry.
• ModelConfig – Links an InferenceModel to a specific prompt template and parameter overrides.
• StrategyConfig – Selects which ModelConfig to use for each inference modality (text-to-text, text-to-image, text-to-video, etc.).

class ModelConfig(Table):
    slug = Varchar(length=80, unique=True, index=True)
    model = ForeignKey(references=InferenceModel)
    prompt_template = ForeignKey(references=PromptTemplate, null=True)
    model_parameters = JSONB(null=True)  # Runtime overrides

Administrators can switch models by updating database rows rather than deploying new code.

3. Strategy and Dispatch Layer

The public API surface lives in calliope/inference/__init__.py, which re-exports concrete functions:

from .text_to_image import text_to_image_file_inference
from .text_to_text import text_to_text_inference
from .text_to_video import image_and_text_to_video_file_inference

When a client calls text_to_image_file_inference, the dispatcher:

1. Resolves the supplied ModelConfig.
2. Retrieves the linked InferenceModel to read the provider.
3. Forwards the request to the correct provider engine based on model.provider.
4. Handles retries, content censoring, and error propagation.

The dispatcher logic in calliope/inference/text_to_image.py illustrates this routing:

if model.provider == InferenceModelProvider.REPLICATE:
    return await text_to_image_file_inference_replicate(...)
elif model.provider == InferenceModelProvider.STABILITY:
    return await text_to_image_file_inference_stability(...)
elif model.provider == InferenceModelProvider.OPENAI:
    return await text_to_image_file_inference_openai(...)
elif model.provider == InferenceModelProvider.HUGGINGFACE:
    return await text_to_image_file_inference_hugging_face(...)
else:
    raise ValueError(f"Unsupported provider: {model.provider}")

4. Provider Engines

The final layer consists of concrete implementations in calliope/inference/engines/*.py. Each engine translates generic requests into provider-specific HTTP or SDK calls:

Engine	Provider	Key Function
runway.py	Runway Gen-4	runway_image_and_text_to_video_inference
stability_image.py	Stability AI	stability_image_to_image_inference
openai_image.py	OpenAI DALL-E	text_to_image_file_inference_openai
openai_text.py	OpenAI GPT	openai_text_to_text_inference
azure_vision.py	Azure Computer Vision	analyze_image, ocr_image
replicate.py	Replicate	replicate_text_to_image_inference

All engines respect a three-tier parameter override hierarchy:

1. Registry defaults from InferenceModel in inference_model_config.py.
2. Configuration overrides from the selected ModelConfig row.
3. Runtime arguments passed directly to the inference function.

The Runway engine in calliope/inference/engines/runway.py demonstrates this merging:

parameters = {
    **(load_json_if_necessary(model.model_parameters) or {}),
    **(load_json_if_necessary(model_config.model_parameters) or {}),
}
parameters["prompt_image"] = f"data:image/png;base64,{prompt_image}"
parameters["prompt_text"] = prompt_text

How Request Routing Works in Practice

When a client initiates a generation request, the Calliope inference engine architecture executes the following flow:

1. Resolution: The dispatcher loads the ModelConfig from the database using the provided slug.
2. Provider Identification: It reads the provider field from the linked InferenceModel registry entry.
3. Engine Selection: The if/elif chain routes to the specific provider engine (e.g., text_to_image_file_inference_stability for Stability AI).
4. Execution: The engine constructs the provider-specific payload, handles authentication via KeysModel, executes the HTTP request or SDK call, and manages polling for asynchronous providers like Runway or Replicate.
5. Response Handling: The engine writes output files (images, videos) or returns text content, while the dispatcher handles retries and error normalization.

Configuring and Extending the Architecture

Text-to-Image with Stability AI

import httpx
from calliope.models import KeysModel
from calliope.tables import ModelConfig
from calliope.inference import text_to_image_file_inference

async def generate_image():
    async with httpx.AsyncClient() as client:
        model_cfg = await ModelConfig.objects().where(
            ModelConfig.slug == "stable-diffusion-default"
        ).first()

        keys = KeysModel(stability_api_key="YOUR_STABILITY_API_KEY")

        filename = await text_to_image_file_inference(
            httpx_client=client,
            text="A futuristic city at sunset",
            output_image_filename="city.png",
            model_config=model_cfg,
            keys=keys,
            width=512,
            height=512,
        )
        print(f"Image saved to {filename}")

Text-to-Video with Runway

import httpx
from calliope.models import KeysModel
from calliope.tables import ModelConfig
from calliope.inference import image_and_text_to_video_file_inference

async def generate_video():
    async with httpx.AsyncClient() as client:
        cfg = await ModelConfig.objects().where(
            ModelConfig.slug == "runway-gen4-default"
        ).first()
        
        keys = KeysModel(runway_api_key="YOUR_RUNWAY_API_KEY")

        video_path = await image_and_text_to_video_file_inference(
            httpx_client=client,
            prompt_image_file="scene.png",
            prompt_text="A dragon soaring over a mountain range",
            output_video_filename="dragon.mp4",
            model_config=cfg,
            keys=keys,
        )
        print(f"Video saved at {video_path}")

Text-to-Text with OpenAI

import httpx
from calliope.models import KeysModel
from calliope.tables import ModelConfig
from calliope.inference import text_to_text_inference

async def extend_story():
    async with httpx.AsyncClient() as client:
        cfg = await ModelConfig.objects().where(
            ModelConfig.slug == "gpt-4-chat"
        ).first()
        
        keys = KeysModel(openai_api_key="YOUR_OPENAI_API_KEY")

        response = await text_to_text_inference(
            httpx_client=client,
            text="Continue the adventure of a pirate ship lost in a storm.",
            model_config=cfg,
            keys=keys,
        )
        print("LLM reply:", response)

Summary

• Four-layer abstraction: The Calliope inference engine architecture separates concerns into a runtime model registry, persistent database configuration, strategy-based dispatch, and provider-specific engines.
• Provider-agnostic dispatch: The dispatcher in calliope/inference/text_to_image.py and sibling modules routes requests to the correct engine (OpenAI, Stability, Runway, etc.) based on the provider field stored in the database.
• Hierarchical configuration: Parameters merge in three tiers—registry defaults from inference_model_config.py, database overrides from ModelConfig, and runtime arguments passed to inference functions.
• Extensible design: Adding a new provider requires only extending the _model_configs_by_name registry and implementing a matching engine module under calliope/inference/engines/.

Frequently Asked Questions

What makes Calliope's inference engine provider-agnostic?

The architecture abstracts provider details behind a unified interface. Client code interacts with high-level functions like text_to_image_file_inference in calliope/inference/__init__.py, while the dispatcher handles provider-specific routing. The InferenceModelProvider enum in calliope/models/inference_model_config.py defines supported providers, and the engine layer encapsulates all SDK and HTTP implementation details.

How does Calliope handle model parameter overrides?

The system implements a three-tier override hierarchy. First, default parameters from the InferenceModel registry entry in inference_model_config.py provide baseline values. Second, the model_parameters JSONB column in the ModelConfig table applies deployment-specific overrides. Third, runtime arguments passed directly to inference functions (like width and height in text_to_image_file_inference) take final precedence.

Where are the API keys managed in the Calliope architecture?

API keys flow through the KeysModel class, which aggregates credentials for all supported providers (OpenAI, Stability, Runway, etc.). Client code instantiates KeysModel with the relevant API keys and passes it to inference functions. The provider engines in calliope/inference/engines/*.py extract the specific key they need from this model to authenticate their respective SDK or HTTP requests.

How do I add a new provider to Calliope?

Extending the architecture requires two steps. First, add a new entry to the _model_configs_by_name dictionary in calliope/models/inference_model_config.py, specifying the provider enum value, model name, and default parameters. Second, create a new engine module under calliope/inference/engines/ (e.g., new_provider.py) implementing the provider's SDK or HTTP contract with the standard function signature accepting httpx_client, prompt data, model, model_config, and keys. The dispatcher will automatically route requests to your new engine when the corresponding provider is selected in the database configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}