# How Calliope Implements Semantic Search: A Technical Deep Dive

> Discover how Calliope implements semantic search using OpenAI embeddings and Pinecone. Learn to filter results by cloud environment for isolated dev and production data.

- Repository: [chrisimmel/calliope](https://github.com/chrisimmel/calliope)
- Tags: deep-dive
- Published: 2026-02-27

---

**Calliope implements semantic search by converting story frames into dense vector embeddings using OpenAI's models, storing them in Pinecone via LangChain, and filtering results by cloud environment to isolate development and production data.**

The open-source Calliope project (`chrisimmel/calliope`) adds intelligent **semantic search** capabilities to its narrative engine without storing raw text in the primary database. Instead, the system leverages a vector database architecture that transforms story content into high-dimensional embeddings, enabling similarity-based retrieval across large story collections.

## The Semantic Search Architecture

Calliope's semantic search system consists of three tightly integrated components: an embedding generation layer, a vector storage backend, and an environment-aware filtering mechanism.

### Embedding Generation with OpenAI

At the core of the pipeline, Calliope uses **OpenAI's embedding models** accessed through LangChain's `OpenAIEmbeddings` class (`langchain_community.embeddings`). The system reads the API key from `settings.OPENAI_API_KEY` and initializes the embedding model during both indexing and search operations.

In [`calliope/storage/vector_manager.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/storage/vector_manager.py), the embedding model is instantiated as:

```python
embeddings = OpenAIEmbeddings(openai_api_key=keys.openai_api_key)

```

This model converts raw story text into 1536-dimensional vectors (or equivalent, depending on the OpenAI model version) that capture semantic meaning rather than simple keyword matches.

### Vector Storage in Pinecone

Calliope persists embeddings in **Pinecone** using LangChain's `PineconeVectorStore` (`langchain_pinecone.vectorstores`). The vector store connects to an index named according to `settings.SEMANTIC_SEARCH_INDEX` (defaulting to `story-semantic-search`).

The implementation handles both write and read operations:

- **Indexing**: Uses `PineconeVectorStore.afrom_texts()` to asynchronously upload document chunks with metadata
- **Querying**: Uses `PineconeVectorStore.from_existing_index()` to connect to the index for similarity searches

### Environment-Aware Metadata Filtering

To support multiple deployment stages while sharing a single Pinecone index (particularly useful for free-tier constraints), Calliope implements **cloud-environment isolation**. Each embedded document carries metadata including an `"env"` field set via `get_cloud_environment()` from [`calliope/utils/google.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/utils/google.py).

During queries, the system applies a metadata filter to ensure only vectors from the current environment are returned:

```python
filter = {"env": {"$eq": cloud_env}}
docsearch.similarity_search_with_score(query, k=max_results, filter=filter)

```

This filtering occurs in [`calliope/storage/vector_manager.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/storage/vector_manager.py) (lines 74-76) and ensures that development, staging, and production data remain logically separated without requiring separate Pinecone indexes.

## Indexing Pipeline: From Story Frames to Vectors

Calliope provides two primary pathways for populating the vector database, both implemented in [`calliope/storage/vector_manager.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/storage/vector_manager.py).

### Batch Processing with index_frames

The `index_frames` function handles incremental updates when new story frames are added to the system. It queries the relational database (via [`calliope/tables/story.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/tables/story.py)) for frames where `indexed_for_search = False`, processes them in batches, and marks them as indexed upon successful vectorization.

The function uses `RecursiveCharacterTextSplitter` to divide long frame text into overlapping chunks before embedding, ensuring that semantic context is preserved across chunk boundaries while respecting token limits.

Key implementation details (lines 98-112 in [`vector_manager.py`](https://github.com/chrisimmel/calliope/blob/main/vector_manager.py)):
- Retrieves unindexed frames from the StoryFrame table
- Splits text into chunks with configurable overlap
- Creates LangChain Document objects with metadata (story CUID, frame number, environment)
- Asynchronously uploads to Pinecone via `afrom_texts`
- Updates the `indexed_for_search` flag in the relational database

### Full Re-indexing with send_all_stories_to_pinecone

For initial setup or complete data refreshes, Calliope provides `send_all_stories_to_pinecone`. This function processes every story in the database regardless of the indexed flag, making it suitable for migrating existing content into the vector store or recovering from index corruption.

Both indexing functions respect the cloud environment constraints, ensuring that vectors are tagged with the correct environment metadata during upload.

## Querying the Vector Store

### The semantic_search Function

The primary interface for retrieving semantically similar content is the `semantic_search` function (lines 44-82 in [`calliope/storage/vector_manager.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/storage/vector_manager.py)). This synchronous function provides a simple API that handles embedding generation, vector store connection, and environment filtering automatically.

The function signature accepts optional API keys for Pinecone and OpenAI, falling back to the configured settings if not provided:

```python
def semantic_search(
    query: str,
    pinecone_api_key: Optional[str] = None,
    openai_api_key: Optional[str] = None,
    max_results: int = 20
) -> List[Tuple[Document, float]]:

```

Execution flow:
1. Initializes `OpenAIEmbeddings` with the provided or configured API key
2. Connects to the existing Pinecone index using `PineconeVectorStore.from_existing_index`
3. Determines the current cloud environment via `get_cloud_environment()`
4. Constructs a metadata filter `{"env": {"$eq": cloud_env}}`
5. Executes `similarity_search_with_score` with the query, result limit, and environment filter
6. Returns scored documents containing story metadata and text snippets

The function includes logging for observability, tracking when the search initiates and completes.

## Key Implementation Files

The semantic search capability is distributed across several modules in the Calliope codebase:

- **[`calliope/storage/vector_manager.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/storage/vector_manager.py)** – Core implementation containing `semantic_search`, `index_frames`, `send_all_stories_to_pinecone`, and the embedding/Pinecone integration logic (lines 44-82, 98-112).

- **[`calliope/settings.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/settings.py)** – Configuration defaults for `SEMANTIC_SEARCH_INDEX`, `PINECONE_API_KEY`, and `OPENAI_API_KEY`.

- **[`calliope/tables/story.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/tables/story.py)** – ORM definitions for `Story` and `StoryFrame` models, including the `indexed_for_search` boolean flag used by the indexing pipeline.

- **[`calliope/utils/google.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/utils/google.py)** – Provides `get_cloud_environment()` for cloud-environment detection and metadata tagging.

- **[`calliope/routes/thoth.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/routes/thoth.py)** – FastAPI route handlers that expose semantic search via HTTP endpoints by wrapping `semantic_search`.

- **[`calliope/commands/index_stories.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/commands/index_stories.py)** – CLI command (`calliope index-stories`) that triggers `index_frames` and performs "heartbeat" searches to maintain Pinecone connection warmth.

## Summary

Calliope implements **semantic search** through a modular pipeline that keeps raw story data in PostgreSQL while storing semantic representations in Pinecone:

- **OpenAI embeddings** (via LangChain) convert story frames into dense vectors that capture semantic meaning beyond keywords.
- **Pinecone vector store** provides scalable similarity search with metadata filtering capabilities.
- **Environment isolation** ensures development, staging, and production data remain separate within shared indexes through automatic metadata tagging.
- **Incremental indexing** via `index_frames` keeps the vector store synchronized with new content, while `send_all_stories_to_pinecone` supports full data migrations.
- **Simple API** exposed through `semantic_search` in [`calliope/storage/vector_manager.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/storage/vector_manager.py) handles embedding, filtering, and retrieval in a single call.

## Frequently Asked Questions

### What embedding model does Calliope use for semantic search?

Calliope uses **OpenAI's embedding models** accessed through LangChain's `OpenAIEmbeddings` class. The implementation reads the API key from `settings.OPENAI_API_KEY` and initializes the model during both indexing and search operations in [`calliope/storage/vector_manager.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/storage/vector_manager.py). This approach generates high-dimensional vectors that capture semantic relationships between story frames rather than simple keyword matches.

### How does Calliope isolate development and production data in Pinecone?

Calliope implements **cloud-environment filtering** to share a single Pinecone index across multiple deployment stages. Each vector document includes metadata with an `"env"` field set via `get_cloud_environment()` from [`calliope/utils/google.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/utils/google.py). During queries, the system applies a metadata filter `{"env": {"$eq": cloud_env}}` to ensure only vectors from the current environment are returned, keeping development, staging, and production data logically separated.

### Where is the semantic search logic implemented in the codebase?

The core **semantic search** implementation resides in **[`calliope/storage/vector_manager.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/storage/vector_manager.py)**. This file contains the `semantic_search` function (lines 44-82) that orchestrates embedding generation, Pinecone connection, and environment filtering. It also houses the indexing functions `index_frames` (lines 98-112) and `send_all_stories_to_pinecone` that populate the vector store. Supporting configuration appears in [`calliope/settings.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/settings.py), while the HTTP API layer lives in [`calliope/routes/thoth.py`](https://github.com/chrisimmel/calliope/blob/main/calliope/routes/thoth.py).

### Can I perform a semantic search without indexing all story frames first?

No, **semantic search requires pre-indexing** to function. The `semantic_search` function queries the Pinecone vector store, which only contains embeddings for frames that have been processed through `index_frames` or `send_all_stories_to_pinecone`. Unindexed frames (where `StoryFrame.indexed_for_search = False`) exist only in the relational database and will not appear in semantic search results. For production deployments, you should run the indexing command (`calliope index-stories`) regularly to keep the vector store synchronized with new content.