How to Define Data Schemas for CocoIndex: A Complete Guide to Vector and Table Structures

Define data schemas for CocoIndex by declaring VectorSchema objects for embeddings, implementing the VectorSchemaProvider protocol for dynamic dimensions, and composing them into TableSchema or CollectionSchema objects that you mount to connectors like SQLite or Qdrant.

CocoIndex separates schema declaration from data-processing logic, allowing you to pre-define how external storage structures should look before any data flows. In the cocoindex-io/cocoindex repository, schemas tell the engine the exact shape of your data—including embedding dimensions, nullability constraints, and primary keys—so it can automatically create and reconcile tables and vector collections. Understanding how to define data schemas for CocoIndex is essential for building declarative data pipelines that sync seamlessly with downstream storage systems.

Core Schema Building Blocks

VectorSchema and MultiVectorSchema

The foundation of any embedding-related schema starts in python/cocoindex/resources/schema.py. The VectorSchema class (lines 23-28) defines the data type and dimensionality of a single embedding vector using np.dtype and a size parameter. For advanced use cases, MultiVectorSchema (lines 49-55) wraps a VectorSchema with additional multi-vector metadata.

import numpy as np
from cocoindex.resources import schema as cs

# Define a 384-dimensional float32 embedding column

embed_schema = cs.VectorSchema(dtype=np.dtype(np.float32), size=384)

VectorSchemaProvider Protocol

For dynamic schema resolution at runtime, implement the VectorSchemaProvider protocol defined at lines 17-21 in python/cocoindex/resources/schema.py. This protocol requires a __coco_vector_schema__ method that returns a VectorSchema object. According to the cocoindex source code, embedding operators like SentenceTransformerEmbedder in python/cocoindex/ops/sentence_transformers.py implement this protocol to automatically expose their output dimensions.

from cocoindex.ops import sentence_transformers as st

class MyEmbedder(st.SentenceTransformerEmbedder):
    """Wraps a SentenceTransformer model and provides its VectorSchema."""
    ...
    

# The engine calls `await embedder.__coco_vector_schema__()` at runtime

# to resolve dimensions dynamically.

Column and Table Definitions

For relational databases, ColumnDef in python/cocoindex/connectors/sqlite/_target.py (lines 68-77) describes individual columns including SQL type, nullability, optional encoder, and whether the column stores vectors. The TableSchema class (lines 83-94) aggregates these into a full table definition with primary keys and optional row-type metadata.

PostgreSQL connectors follow an identical pattern in python/cocoindex/connectors/postgres/_target.py, ensuring consistent schema definition across SQL backends.

Collection Schemas for Vector Databases

When targeting Qdrant, use CollectionSchema defined in python/cocoindex/connectors/qdrant/_target.py (lines 8-33). This bundles one or more QdrantVectorDef objects that specify both the vector schema and distance metric (cosine, Euclidean, etc.).

from cocoindex.connectors.qdrant import _target as qdrant_target
from cocoindex.resources import schema as cs
import numpy as np

collection_schema = await qdrant_target.CollectionSchema.create(
    vectors={
        "embedding": qdrant_target.QdrantVectorDef(
            schema=cs.VectorSchema(dtype=np.dtype(np.float32), size=384),
            distance="cosine",
        )
    }
)

The Schema Declaration Workflow

CocoIndex follows a four-phase pattern for schema handling. As implemented in the cocoindex source code, the workflow proceeds as follows:

1. Declare vector shapes – Instantiate VectorSchema directly or create a class implementing VectorSchemaProvider.
2. Plug into column definitions – Mark columns with is_vector=True and let the engine pull the VectorSchema via get_vector_schema.
3. Build table/collection schemas – Pass dictionaries of ColumnDef or QdrantVectorDef objects, or use auto-inspection helpers like TableSchema.from_class and CollectionSchema.create.
4. Mount the schema – Pass the schema object to connector-specific declare_*_target functions, which sync the external store and reconcile future changes.

Practical Implementation Examples

Declaring a Simple Vector Schema

For fixed-dimension embeddings, instantiate VectorSchema directly with the appropriate NumPy dtype and size.

import numpy as np
from cocoindex.resources import schema as cs

# 384-dimensional float32 embeddings

vector_schema = cs.VectorSchema(dtype=np.dtype(np.float32), size=384)

Building a SQLite Table Schema with Vector Columns

Use TableSchema.from_class to auto-inspect a Python dataclass, then override specific columns with your vector schema. In python/cocoindex/connectors/sqlite/_target.py, the from_class method accepts a column_overrides dictionary to specify vector columns.

from dataclasses import dataclass
import numpy as np
from cocoindex.resources import schema as cs
from cocoindex.connectors.sqlite import _target as sqlite_target

@dataclass
class Document:
    id: int
    title: str
    content: str
    embedding: np.ndarray  # vector column

# Override the inferred type for `embedding`

vector_schema = cs.VectorSchema(dtype=np.dtype(np.float32), size=384)

table_schema = await sqlite_target.TableSchema.from_class(
    record_type=Document,
    primary_key=["id"],
    column_overrides={"embedding": vector_schema},
)

Creating a Qdrant Collection Schema

For vector databases, define named vector fields with specific distance metrics using the Qdrant connector.

from cocoindex.connectors.qdrant import _target as qdrant_target
from cocoindex.resources import schema as cs
import numpy as np

collection_schema = await qdrant_target.CollectionSchema.create(
    vectors={
        "embedding": qdrant_target.QdrantVectorDef(
            schema=cs.VectorSchema(dtype=np.dtype(np.float32), size=384),
            distance="cosine",
        ),
        "sparse_embedding": qdrant_target.QdrantVectorDef(
            schema=cs.VectorSchema(dtype=np.dtype(np.float32), size=768),
            distance="dot",
        )
    }
)

Mounting and Reconciling Schemas

Once defined, mount the schema using coco.use_mount with the appropriate connector function. The target schema persists as declarative state, and subsequent data operations reconcile against it.

import pathlib
from cocoindex import coco
from cocoindex.connectors.sqlite import declare_table_target

# Mount the schema from the previous example

db_path = pathlib.Path("./my.db")
target = await coco.use_mount(
    declare_table_target,
    db_path,
    table_schema,            # The TableSchema we built

    table_name="documents",
)

Summary

• VectorSchema in python/cocoindex/resources/schema.py defines embedding dimensions and data types using NumPy dtypes.
• Implement VectorSchemaProvider when you need runtime schema resolution from models like Sentence Transformers or LiteLLM.
• Use TableSchema.from_class for SQL connectors and CollectionSchema.create for Qdrant to auto-inspect Python record types.
• Override auto-inferred types using column_overrides or explicit vector definitions when mixing standard and vector columns.
• Mount schemas using connector-specific functions like declare_table_target to enable automatic table creation and schema reconciliation.

Frequently Asked Questions

What is the difference between VectorSchema and VectorSchemaProvider?

VectorSchema is a static struct that hardcodes the dtype and dimensionality of a vector. VectorSchemaProvider is a protocol that dynamic components like embedding models implement to expose their output dimensions at runtime. When you define data schemas for CocoIndex, use VectorSchema for fixed embeddings and VectorSchemaProvider when the model determines the shape.

How do I auto-generate schemas from Python dataclasses?

Use the TableSchema.from_class method (available in sqlite/_target.py and postgres/_target.py) or CollectionSchema.create for Qdrant. These class methods inspect your dataclass, NamedTuple, or Pydantic model fields and generate appropriate column definitions. Pass column_overrides to specify vector schemas for fields that should store embeddings.

Can I define multiple vector fields in a single collection?

Yes. When creating a CollectionSchema for Qdrant, provide a dictionary mapping field names to QdrantVectorDef objects. Each definition can specify different dimensions, data types, and distance metrics, allowing you to store dense and sparse vectors in the same collection.

Where are the core schema definitions located in the codebase?

The base schema types reside in python/cocoindex/resources/schema.py, including VectorSchema, MultiVectorSchema, and the VectorSchemaProvider protocol. Connector-specific implementations live in python/cocoindex/connectors/sqlite/_target.py for SQLite tables, python/cocoindex/connectors/postgres/_target.py for PostgreSQL, and python/cocoindex/connectors/qdrant/_target.py for Qdrant collections.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}