How Calliope Implements Semantic Search: A Technical Deep Dive

Calliope implements semantic search by converting story frames into dense vector embeddings using OpenAI's models, storing them in Pinecone via LangChain, and filtering results by cloud environment to isolate development and production data.

The open-source Calliope project (chrisimmel/calliope) adds intelligent semantic search capabilities to its narrative engine without storing raw text in the primary database. Instead, the system leverages a vector database architecture that transforms story content into high-dimensional embeddings, enabling similarity-based retrieval across large story collections.

The Semantic Search Architecture

Calliope's semantic search system consists of three tightly integrated components: an embedding generation layer, a vector storage backend, and an environment-aware filtering mechanism.

Embedding Generation with OpenAI

At the core of the pipeline, Calliope uses OpenAI's embedding models accessed through LangChain's OpenAIEmbeddings class (langchain_community.embeddings). The system reads the API key from settings.OPENAI_API_KEY and initializes the embedding model during both indexing and search operations.

In calliope/storage/vector_manager.py, the embedding model is instantiated as:

embeddings = OpenAIEmbeddings(openai_api_key=keys.openai_api_key)

This model converts raw story text into 1536-dimensional vectors (or equivalent, depending on the OpenAI model version) that capture semantic meaning rather than simple keyword matches.

Vector Storage in Pinecone

Calliope persists embeddings in Pinecone using LangChain's PineconeVectorStore (langchain_pinecone.vectorstores). The vector store connects to an index named according to settings.SEMANTIC_SEARCH_INDEX (defaulting to story-semantic-search).

The implementation handles both write and read operations:

• Indexing: Uses PineconeVectorStore.afrom_texts() to asynchronously upload document chunks with metadata
• Querying: Uses PineconeVectorStore.from_existing_index() to connect to the index for similarity searches

Environment-Aware Metadata Filtering

To support multiple deployment stages while sharing a single Pinecone index (particularly useful for free-tier constraints), Calliope implements cloud-environment isolation. Each embedded document carries metadata including an "env" field set via get_cloud_environment() from calliope/utils/google.py.

During queries, the system applies a metadata filter to ensure only vectors from the current environment are returned:

filter = {"env": {"$eq": cloud_env}}
docsearch.similarity_search_with_score(query, k=max_results, filter=filter)

This filtering occurs in calliope/storage/vector_manager.py (lines 74-76) and ensures that development, staging, and production data remain logically separated without requiring separate Pinecone indexes.

Indexing Pipeline: From Story Frames to Vectors

Calliope provides two primary pathways for populating the vector database, both implemented in calliope/storage/vector_manager.py.

Batch Processing with index_frames

The index_frames function handles incremental updates when new story frames are added to the system. It queries the relational database (via calliope/tables/story.py) for frames where indexed_for_search = False, processes them in batches, and marks them as indexed upon successful vectorization.

The function uses RecursiveCharacterTextSplitter to divide long frame text into overlapping chunks before embedding, ensuring that semantic context is preserved across chunk boundaries while respecting token limits.

Key implementation details (lines 98-112 in vector_manager.py):

• Retrieves unindexed frames from the StoryFrame table
• Splits text into chunks with configurable overlap
• Creates LangChain Document objects with metadata (story CUID, frame number, environment)
• Asynchronously uploads to Pinecone via afrom_texts
• Updates the indexed_for_search flag in the relational database

Full Re-indexing with send_all_stories_to_pinecone

For initial setup or complete data refreshes, Calliope provides send_all_stories_to_pinecone. This function processes every story in the database regardless of the indexed flag, making it suitable for migrating existing content into the vector store or recovering from index corruption.

Both indexing functions respect the cloud environment constraints, ensuring that vectors are tagged with the correct environment metadata during upload.

Querying the Vector Store

The semantic_search Function

The primary interface for retrieving semantically similar content is the semantic_search function (lines 44-82 in calliope/storage/vector_manager.py). This synchronous function provides a simple API that handles embedding generation, vector store connection, and environment filtering automatically.

The function signature accepts optional API keys for Pinecone and OpenAI, falling back to the configured settings if not provided:

def semantic_search(
    query: str,
    pinecone_api_key: Optional[str] = None,
    openai_api_key: Optional[str] = None,
    max_results: int = 20
) -> List[Tuple[Document, float]]:

Execution flow:

1. Initializes OpenAIEmbeddings with the provided or configured API key
2. Connects to the existing Pinecone index using PineconeVectorStore.from_existing_index
3. Determines the current cloud environment via get_cloud_environment()
4. Constructs a metadata filter {"env": {"$eq": cloud_env}}
5. Executes similarity_search_with_score with the query, result limit, and environment filter
6. Returns scored documents containing story metadata and text snippets

The function includes logging for observability, tracking when the search initiates and completes.

Key Implementation Files

The semantic search capability is distributed across several modules in the Calliope codebase:

• calliope/storage/vector_manager.py – Core implementation containing semantic_search, index_frames, send_all_stories_to_pinecone, and the embedding/Pinecone integration logic (lines 44-82, 98-112).

• calliope/settings.py – Configuration defaults for SEMANTIC_SEARCH_INDEX, PINECONE_API_KEY, and OPENAI_API_KEY.

• calliope/tables/story.py – ORM definitions for Story and StoryFrame models, including the indexed_for_search boolean flag used by the indexing pipeline.

• calliope/utils/google.py – Provides get_cloud_environment() for cloud-environment detection and metadata tagging.

• calliope/routes/thoth.py – FastAPI route handlers that expose semantic search via HTTP endpoints by wrapping semantic_search.

• calliope/commands/index_stories.py – CLI command (calliope index-stories) that triggers index_frames and performs "heartbeat" searches to maintain Pinecone connection warmth.

Summary

Calliope implements semantic search through a modular pipeline that keeps raw story data in PostgreSQL while storing semantic representations in Pinecone:

• OpenAI embeddings (via LangChain) convert story frames into dense vectors that capture semantic meaning beyond keywords.
• Pinecone vector store provides scalable similarity search with metadata filtering capabilities.
• Environment isolation ensures development, staging, and production data remain separate within shared indexes through automatic metadata tagging.
• Incremental indexing via index_frames keeps the vector store synchronized with new content, while send_all_stories_to_pinecone supports full data migrations.
• Simple API exposed through semantic_search in calliope/storage/vector_manager.py handles embedding, filtering, and retrieval in a single call.

Frequently Asked Questions

What embedding model does Calliope use for semantic search?

Calliope uses OpenAI's embedding models accessed through LangChain's OpenAIEmbeddings class. The implementation reads the API key from settings.OPENAI_API_KEY and initializes the model during both indexing and search operations in calliope/storage/vector_manager.py. This approach generates high-dimensional vectors that capture semantic relationships between story frames rather than simple keyword matches.

How does Calliope isolate development and production data in Pinecone?

Calliope implements cloud-environment filtering to share a single Pinecone index across multiple deployment stages. Each vector document includes metadata with an "env" field set via get_cloud_environment() from calliope/utils/google.py. During queries, the system applies a metadata filter {"env": {"$eq": cloud_env}} to ensure only vectors from the current environment are returned, keeping development, staging, and production data logically separated.

Where is the semantic search logic implemented in the codebase?

The core semantic search implementation resides in calliope/storage/vector_manager.py. This file contains the semantic_search function (lines 44-82) that orchestrates embedding generation, Pinecone connection, and environment filtering. It also houses the indexing functions index_frames (lines 98-112) and send_all_stories_to_pinecone that populate the vector store. Supporting configuration appears in calliope/settings.py, while the HTTP API layer lives in calliope/routes/thoth.py.

Can I perform a semantic search without indexing all story frames first?

No, semantic search requires pre-indexing to function. The semantic_search function queries the Pinecone vector store, which only contains embeddings for frames that have been processed through index_frames or send_all_stories_to_pinecone. Unindexed frames (where StoryFrame.indexed_for_search = False) exist only in the relational database and will not appear in semantic search results. For production deployments, you should run the indexing command (calliope index-stories) regularly to keep the vector store synchronized with new content.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}