# Calliope Storage Backends: A Complete Guide to the Four-Tier Architecture

> Explore Calliope storage backends: PostgreSQL, GCS, Pinecone, and local filesystem. Understand each tier's role in its four-tier architecture for optimal data management.

- Repository: [chrisimmel/calliope](https://github.com/chrisimmel/calliope)
- Tags: how-to-guide
- Published: 2026-02-27

---

**Calliope uses four distinct storage backends—PostgreSQL for relational data, local filesystem for ephemeral files, Google Cloud Storage for persistent media, and Pinecone for vector search—each optimized for specific data durability and access patterns.**

The open-source Calliope project by chrisimmel implements a sophisticated multi-tier storage architecture to handle diverse data requirements ranging from transactional application state to AI-powered semantic search. Understanding these storage backends is essential for deploying, scaling, and customizing Calliope for production environments.

## Overview of Calliope Storage Backends

Calliope's storage strategy separates data by access patterns, durability requirements, and query semantics. Rather than forcing all data into a single database, the architecture leverages specialized technologies:

- **Relational transactions** for structured application data
- **Local filesystem** for temporary processing artifacts
- **Object storage** for long-term media assets
- **Vector database** for semantic similarity search

This separation optimizes cost, performance, and scalability across different deployment environments, from local development to Google Cloud Platform (GCP) production deployments.

## The Four Storage Backends Explained

### 1. PostgreSQL Relational Database

The primary **storage backend** for all core application state is PostgreSQL, accessed through the Piccolo ORM. This includes stories, story frames, configuration settings, user accounts, and client state.

The database schema is defined in [`calliope/tables/story.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/tables/story.py), where model classes like `Story` and `StoryFrame` map directly to PostgreSQL tables. Connection parameters are configured via environment variables prefixed with `POSTGRESQL_` in [`calliope/settings.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/settings.py):

```python

# From calliope/settings.py

POSTGRESQL_HOST = os.environ.get("POSTGRESQL_HOST", "localhost")
POSTGRESQL_DATABASE = os.environ.get("POSTGRESQL_DATABASE", "calliope")
POSTGRESQL_USER = os.environ.get("POSTGRESQL_USER", "calliope")
POSTGRESQL_PASSWORD = os.environ.get("POSTGRESQL_PASSWORD", "")

```

### 2. Ephemeral File Storage

For temporary files created during request processing—such as uploaded input images, audio clips, and intermediate generated images—Calliope uses the local filesystem as an **ephemeral storage backend**. These files are transient and do not require long-term durability.

The ephemeral storage locations include:
- `input/` directory for user uploads
- `media/` directory for intermediate processing artifacts

As documented in [`docs/storage.md`](https://github.com/chrisimmel/calliope/blob/main/docs/storage.md), these paths are temporary by design. Files here are either processed into persistent storage or discarded after the request completes.

### 3. Persistent Media Storage

Final generated images that appear in stories require durable, long-term **storage backends**. Calliope supports two configurations:
- **Google Cloud Storage (GCS)** for cloud deployments
- **Local filesystem** (`media/` folder) for development

The storage destination is determined by the `CLOUD_ENV` setting in [`calliope/settings.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/settings.py). When `CLOUD_ENV` is set to `"gcp"`, media URLs point to Cloud Storage buckets (`gs://{bucket}/media/...`); otherwise, they resolve to local filesystem paths.

The logic for determining media paths is implemented in utility functions that check `settings.CLOUD_ENV` and `settings.CALLIOPE_BUCKET_NAME`:

```python
from pathlib import Path
from calliope.settings import settings

def get_media_path(uuid: str, ext: str) -> str:
    """
    Returns the final storage location for a generated image.
    In GCP it returns a GCS URL, otherwise a local path.
    """
    if settings.CLOUD_ENV == "gcp":
        return f"gs://{settings.CALLIOPE_BUCKET_NAME}/media/{uuid}.{ext}"
    return str(Path(settings.MEDIA_FOLDER) / f"{uuid}.{ext}")

```

### 4. Pinecone Vector Database

For **semantic search capabilities**, Calliope uses Pinecone as a specialized vector **storage backend**. This indexes story text and image metadata to enable fast similarity searches across the corpus.

The integration is implemented in [`calliope/storage/vector_manager.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/storage/vector_manager.py), which handles:
- Embedding text using OpenAI embeddings
- Upserting vectors to Pinecone
- Querying with environment-specific filters

Configuration is managed through `PINECONE_API_KEY` and `SEMANTIC_SEARCH_INDEX` variables in [`calliope/settings.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/settings.py).

To index frames for search:

```python
from calliope.storage.vector_manager import index_frames

# Index up to 500 new frames

await index_frames(max_frames=500)

```

To perform semantic search:

```python
from calliope.storage.vector_manager import semantic_search

results = semantic_search(
    query="a sunrise over mountains",
    max_results=10,
)

for doc, score in results:
    print(f"{doc.metadata['story_cuid']} – score {score:.2f}")

```

The `semantic_search` function automatically filters by the current cloud environment (`env` metadata field) to ensure local and GCP deployments do not return mixed results.

## How the Storage Backends Work Together

Calliope's **storage backends** operate as a coordinated pipeline during story creation:

1. **PostgreSQL** maintains the canonical state—story metadata, frame sequences, and configuration—providing ACID compliance for transactional operations.

2. When users upload content, **ephemeral storage** (`input/`) receives the raw files temporarily. The application processes these files, potentially generating intermediate artifacts in `media/`.

3. Upon finalizing a frame, the system moves the generated image to **persistent media storage**—either GCS or local `media/`—and updates the PostgreSQL record with the storage URL.

4. Simultaneously, the **Pinecone vector database** indexes the frame's textual content and metadata, embedding the text via OpenAI and storing the vector with references (`story_cuid`, `frame_id`, `env`) back to the PostgreSQL records.

This architecture ensures that relational queries, media serving, and semantic search each use the optimal **storage backend** for their specific access patterns, while maintaining referential integrity through the PostgreSQL primary keys stored in vector metadata.

## Summary

Calliope implements a multi-tier **storage backend** architecture optimized for different data types and query patterns:

- **PostgreSQL** serves as the primary relational database for all application state, accessed via Piccolo ORM with tables defined in [`calliope/tables/story.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/tables/story.py).
- **Ephemeral file storage** handles temporary uploads and intermediate processing files locally in `input/` and `media/` directories.
- **Persistent media storage** uses Google Cloud Storage in production or local filesystem in development, controlled by `CLOUD_ENV` settings.
- **Pinecone vector database** enables semantic search through OpenAI embeddings, with environment-aware filtering to separate local and cloud indices.

## Frequently Asked Questions

### What database does Calliope use for primary data storage?

Calliope uses **PostgreSQL** as its primary relational database for all core application state including stories, frames, user accounts, and configuration. The application accesses PostgreSQL through the **Piccolo ORM**, with database tables defined in [`calliope/tables/story.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/tables/story.py) and connection settings configured via `POSTGRESQL_*` environment variables in [`calliope/settings.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/settings.py).

### How does Calliope handle file uploads and temporary storage?

Calliope uses the **local filesystem** as an ephemeral storage backend for temporary files during request processing. Uploaded images and audio are stored in the `input/` directory, while intermediate generated images reside in `media/` during processing. These files are transient—either promoted to persistent storage or discarded after processing completes, as documented in [`docs/storage.md`](https://github.com/chrisimmel/calliope/blob/main/docs/storage.md).

### Can Calliope run without Google Cloud Storage?

Yes, Calliope can operate entirely without Google Cloud Storage by running in **local development mode**. When the `CLOUD_ENV` setting (defined in [`calliope/settings.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/settings.py)) is not set to `"gcp"`, the application uses the local filesystem in the `media/` folder for persistent image storage. The `get_media_path()` function automatically returns local filesystem paths instead of `gs://` URLs when running outside GCP.