How Vector Embeddings Are Stored and Queried in SurrealDB for Semantic Search

Open Notebook stores vector embeddings as native array<float> fields in a dedicated source_embedding table, using bulk INSERT operations for efficient storage and SurrealQL subqueries for lightweight existence checks and similarity search retrieval.

Open Notebook leverages SurrealDB's native array support to manage dense vector embeddings for semantic search across document chunks. This implementation, found in the lfnovo/open-notebook repository, stores embeddings in a dedicated table linked to source documents, enabling efficient similarity matching and retrieval. Understanding how these vectors are stored and queried reveals the architectural patterns that make the system both performant and scalable.

Storage Schema in the source_embedding Table

Open Notebook persists every vector embedding in a dedicated SurrealDB table called source_embedding. Each record represents a single chunk of the original source text and contains the vector along with metadata required for reconstruction and querying.

Table Structure and Field Types

The schema is declared by the SourceEmbedding model in open_notebook/domain/notebook.py. The class inherits from ObjectModel and sets table_name = "source_embedding", defining the following fields:

• source (RecordID): The SurrealDB ID of the parent Source record, establishing the foreign key relationship.
• order (int): The chunk's 0-based position in the original document, enabling accurate reassembly of text.
• content (string): The raw text of the chunk, retained for debugging and regeneration scenarios.
• embedding (array<float>): The dense vector returned by the embedding model, stored as a native SurrealDB array.

This design ensures that vector data remains tightly coupled with its textual context while maintaining queryable relationships to parent documents.

Insertion Pipeline for Vector Embeddings

When a source document is processed, the system executes a deterministic pipeline to generate and store embeddings. This operation is handled by the embed_source command defined in commands/embedding_commands.py.

The embed_source Command Workflow

The command performs five distinct steps to ensure data consistency and efficiency:

1. Load the source and verify it contains processable text.
2. Delete any existing embeddings for the same source to maintain idempotency.
3. Chunk the full text according to content type (PDF, HTML, or plain text).
4. Generate embeddings for each chunk via the unified generate_embeddings utility.
5. Bulk-INSERT a list of dictionaries into the source_embedding table.

This approach prevents duplicate embeddings and ensures that regenerating vectors for an updated source replaces stale data atomically.

Bulk Insert Implementation

The system uses a single batch INSERT operation to minimize round-trips and allow SurrealDB to index the vector column efficiently:

# commands/embedding_commands.py – embed_source_command

source = await Source.get(input_data.source_id)

# … chunking logic omitted …

embeddings = await generate_embeddings(chunks, command_id=get_command_id(input_data))

records = [
    {
        "source": ensure_record_id(input_data.source_id),
        "order": idx,
        "content": chunk,
        "embedding": embedding,
    }
    for idx, (chunk, embedding) in enumerate(zip(chunks, embeddings))
]

await repo_insert("source_embedding", records)   # bulk INSERT

By passing a list of records to repo_insert, SurrealDB stores the vectors as native array fields without requiring serialization into binary blobs or external storage.

Querying Patterns and Retrieval Strategies

Various API components query the source_embedding table to check existence, count stored chunks, or retrieve vectors for similarity calculations.

Existence Checks Without Payload Transfer

To determine whether a source has embeddings without transferring heavy vector data, the routers use a boolean subquery in api/routers/sources.py:

# api/routers/sources.py – SELECT clause

SELECT
    id, asset, created, title, updated, topics, command,
    (SELECT VALUE count() FROM source_insight WHERE source = $parent.id GROUP ALL)[0].count OR 0 AS insights_count,
    (SELECT VALUE id FROM source_embedding WHERE source = $parent.id LIMIT 1) != [] AS embedded
FROM source

The embedded field evaluates to true when at least one row exists in source_embedding for the source. This lightweight check avoids pulling the full array<float> payload across the network when only a boolean status is required.

Chunk Counting and Metadata

The Source domain model provides a method to count stored chunks without retrieving the embeddings themselves. In open_notebook/domain/notebook.py, the get_embedded_chunks method executes:

# open_notebook/domain/notebook.py – get_embedded_chunks()

result = await repo_query(
    """
    SELECT count() AS chunks FROM source_embedding WHERE source=$id GROUP ALL
    """,
    {"id": ensure_record_id(self.id)},
)
return result[0]["chunks"] if result else 0

This pattern supports UI indicators showing how many chunks are indexed for a given source document.

Similarity Search Retrieval

For semantic search operations, the system retrieves all embeddings for a specific source using an ordered query:

# Pattern used in search services

emb_rows = await repo_query(
    """
    SELECT embedding, order FROM source_embedding WHERE source=$source_id ORDER BY order
    """,
    {"source_id": ensure_record_id(source_id)},
)
embeddings = [row["embedding"] for row in emb_rows]

SurrealDB's native array type allows the embedding field to be used directly in vector-distance operations (such as the <-> operator) without transformation, or exported to external ANN libraries for approximate nearest neighbor search.

Design Benefits of Native Array Storage

This architecture leverages SurrealDB-specific features to optimize for the semantic search workload:

• Chunk-level granularity: Each vector corresponds to a manageable text chunk (typically 512-1024 tokens), keeping dimensionality stable and enabling fine-grained similarity matching.
• Fast bulk writes: Single-statement INSERT batches reduce network round-trips and allow SurrealDB to build efficient indexes on the vector column.
• Composable relationships: The source foreign key enables JOIN operations that combine metadata (titles, topics) with embedding data in a single request.
• Storage efficiency: Native array<float> fields avoid the overhead of JSON serialization or base64 encoding required by text-based vector storage schemes.

Summary

• Storage Location: Vector embeddings reside in the source_embedding table as native array<float> fields, defined in open_notebook/domain/notebook.py.
• Write Pattern: The embed_source command in commands/embedding_commands.py uses bulk INSERT operations after deleting existing records to ensure idempotency.
• Existence Checking: Lightweight boolean queries in api/routers/sources.py verify embedding status without transferring vector payloads.
• Retrieval Strategy: Ordered SELECT queries fetch embeddings by source ID for similarity search, leveraging SurrealDB's native array support for distance calculations.
• Chunk Architecture: Each record represents a text chunk with ordering metadata, enabling document reconstruction and granular semantic matching.

Frequently Asked Questions

What data type does SurrealDB use for vector embeddings in Open Notebook?

SurrealDB stores vector embeddings as array<float> native fields. This type preserves the dense vector structure without requiring serialization, allowing direct use in mathematical operations and distance calculations within SurrealQL queries.

How does Open Notebook ensure idempotency when storing embeddings?

The embed_source command first executes a DELETE operation to remove any existing records in source_embedding matching the source ID before inserting new vectors. This pattern, implemented in commands/embedding_commands.py, ensures that reprocessing a source replaces stale embeddings rather than creating duplicates.

Can SurrealDB perform native vector similarity calculations on these embeddings?

Yes. Because embeddings are stored as native arrays, SurrealDB can apply vector operators like <-> (Euclidean distance) directly within queries. The retrieved arrays can also be exported to Python-based ANN libraries for approximate nearest neighbor search when working with large vector collections.

What is the typical granularity for stored embeddings?

Open Notebook stores one embedding per text chunk, with each chunk representing approximately 512-1024 tokens of the original document. The order field in source_embedding records maintains the sequence, allowing reconstruction of the full document context from retrieved chunks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}