Calliope Storage Backends: A Complete Guide to the Four-Tier Architecture

Calliope uses four distinct storage backends—PostgreSQL for relational data, local filesystem for ephemeral files, Google Cloud Storage for persistent media, and Pinecone for vector search—each optimized for specific data durability and access patterns.

The open-source Calliope project by chrisimmel implements a sophisticated multi-tier storage architecture to handle diverse data requirements ranging from transactional application state to AI-powered semantic search. Understanding these storage backends is essential for deploying, scaling, and customizing Calliope for production environments.

Overview of Calliope Storage Backends

Calliope's storage strategy separates data by access patterns, durability requirements, and query semantics. Rather than forcing all data into a single database, the architecture leverages specialized technologies:

• Relational transactions for structured application data
• Local filesystem for temporary processing artifacts
• Object storage for long-term media assets
• Vector database for semantic similarity search

This separation optimizes cost, performance, and scalability across different deployment environments, from local development to Google Cloud Platform (GCP) production deployments.

The Four Storage Backends Explained

1. PostgreSQL Relational Database

The primary storage backend for all core application state is PostgreSQL, accessed through the Piccolo ORM. This includes stories, story frames, configuration settings, user accounts, and client state.

The database schema is defined in calliope/tables/story.py, where model classes like Story and StoryFrame map directly to PostgreSQL tables. Connection parameters are configured via environment variables prefixed with POSTGRESQL_ in calliope/settings.py:

# From calliope/settings.py

POSTGRESQL_HOST = os.environ.get("POSTGRESQL_HOST", "localhost")
POSTGRESQL_DATABASE = os.environ.get("POSTGRESQL_DATABASE", "calliope")
POSTGRESQL_USER = os.environ.get("POSTGRESQL_USER", "calliope")
POSTGRESQL_PASSWORD = os.environ.get("POSTGRESQL_PASSWORD", "")

2. Ephemeral File Storage

For temporary files created during request processing—such as uploaded input images, audio clips, and intermediate generated images—Calliope uses the local filesystem as an ephemeral storage backend. These files are transient and do not require long-term durability.

The ephemeral storage locations include:

• input/ directory for user uploads
• media/ directory for intermediate processing artifacts

As documented in docs/storage.md, these paths are temporary by design. Files here are either processed into persistent storage or discarded after the request completes.

3. Persistent Media Storage

Final generated images that appear in stories require durable, long-term storage backends. Calliope supports two configurations:

• Google Cloud Storage (GCS) for cloud deployments
• Local filesystem (media/ folder) for development

The storage destination is determined by the CLOUD_ENV setting in calliope/settings.py. When CLOUD_ENV is set to "gcp", media URLs point to Cloud Storage buckets (gs://{bucket}/media/...); otherwise, they resolve to local filesystem paths.

The logic for determining media paths is implemented in utility functions that check settings.CLOUD_ENV and settings.CALLIOPE_BUCKET_NAME:

from pathlib import Path
from calliope.settings import settings

def get_media_path(uuid: str, ext: str) -> str:
    """
    Returns the final storage location for a generated image.
    In GCP it returns a GCS URL, otherwise a local path.
    """
    if settings.CLOUD_ENV == "gcp":
        return f"gs://{settings.CALLIOPE_BUCKET_NAME}/media/{uuid}.{ext}"
    return str(Path(settings.MEDIA_FOLDER) / f"{uuid}.{ext}")

4. Pinecone Vector Database

For semantic search capabilities, Calliope uses Pinecone as a specialized vector storage backend. This indexes story text and image metadata to enable fast similarity searches across the corpus.

The integration is implemented in calliope/storage/vector_manager.py, which handles:

• Embedding text using OpenAI embeddings
• Upserting vectors to Pinecone
• Querying with environment-specific filters

Configuration is managed through PINECONE_API_KEY and SEMANTIC_SEARCH_INDEX variables in calliope/settings.py.

To index frames for search:

from calliope.storage.vector_manager import index_frames

# Index up to 500 new frames

await index_frames(max_frames=500)

To perform semantic search:

from calliope.storage.vector_manager import semantic_search

results = semantic_search(
    query="a sunrise over mountains",
    max_results=10,
)

for doc, score in results:
    print(f"{doc.metadata['story_cuid']} – score {score:.2f}")

The semantic_search function automatically filters by the current cloud environment (env metadata field) to ensure local and GCP deployments do not return mixed results.

How the Storage Backends Work Together

Calliope's storage backends operate as a coordinated pipeline during story creation:

1. PostgreSQL maintains the canonical state—story metadata, frame sequences, and configuration—providing ACID compliance for transactional operations.

2. When users upload content, ephemeral storage (input/) receives the raw files temporarily. The application processes these files, potentially generating intermediate artifacts in media/.

3. Upon finalizing a frame, the system moves the generated image to persistent media storage—either GCS or local media/—and updates the PostgreSQL record with the storage URL.

4. Simultaneously, the Pinecone vector database indexes the frame's textual content and metadata, embedding the text via OpenAI and storing the vector with references (story_cuid, frame_id, env) back to the PostgreSQL records.

This architecture ensures that relational queries, media serving, and semantic search each use the optimal storage backend for their specific access patterns, while maintaining referential integrity through the PostgreSQL primary keys stored in vector metadata.

Summary

Calliope implements a multi-tier storage backend architecture optimized for different data types and query patterns:

• PostgreSQL serves as the primary relational database for all application state, accessed via Piccolo ORM with tables defined in calliope/tables/story.py.
• Ephemeral file storage handles temporary uploads and intermediate processing files locally in input/ and media/ directories.
• Persistent media storage uses Google Cloud Storage in production or local filesystem in development, controlled by CLOUD_ENV settings.
• Pinecone vector database enables semantic search through OpenAI embeddings, with environment-aware filtering to separate local and cloud indices.

Frequently Asked Questions

What database does Calliope use for primary data storage?

Calliope uses PostgreSQL as its primary relational database for all core application state including stories, frames, user accounts, and configuration. The application accesses PostgreSQL through the Piccolo ORM, with database tables defined in calliope/tables/story.py and connection settings configured via POSTGRESQL_* environment variables in calliope/settings.py.

How does Calliope handle file uploads and temporary storage?

Calliope uses the local filesystem as an ephemeral storage backend for temporary files during request processing. Uploaded images and audio are stored in the input/ directory, while intermediate generated images reside in media/ during processing. These files are transient—either promoted to persistent storage or discarded after processing completes, as documented in docs/storage.md.

Can Calliope run without Google Cloud Storage?

Yes, Calliope can operate entirely without Google Cloud Storage by running in local development mode. When the CLOUD_ENV setting (defined in calliope/settings.py) is not set to "gcp", the application uses the local filesystem in the media/ folder for persistent image storage. The get_media_path() function automatically returns local filesystem paths instead of gs:// URLs when running outside GCP.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}