How Vector Embeddings Are Stored and Queried in SurrealDB: Inside Open Notebook

Open Notebook stores vector embeddings as native array<float> values in a dedicated source_embedding table, using bulk INSERT operations for efficient writes and SurrealQL sub-queries for fast existence checks and retrieval.

The lfnovo/open-notebook repository implements a production-ready vector storage layer using SurrealDB as its primary database. Understanding how vector embeddings are stored and queried in SurrealDB reveals an architecture optimized for RAG (Retrieval-Augmented Generation) pipelines, leveraging native array types, chunk-level granularity, and idempotent write patterns.

Schema Design: The source_embedding Table

Vector embeddings are persisted in a dedicated table called source_embedding, defined by the SourceEmbedding model in open_notebook/domain/notebook.py (lines 304–307). This table stores individual text chunks rather than entire documents, enabling fine-grained similarity search.

Each record contains four critical fields:

• source – A RecordID linking to the parent document in the source table
• order – An integer (0-based) preserving the chunk sequence for document reconstruction
• content – The raw text string of the chunk (used for debugging and regeneration)
• embedding – The dense vector as an array<float> (e.g., 1536 dimensions for OpenAI models)

This schema aligns with standard chunking strategies (512–1024 tokens), keeping vector dimensionality stable while enabling precise semantic matching.

Storage Strategy: Bulk Insertion and Idempotency

The embed_source command in commands/embedding_commands.py orchestrates the complete embedding lifecycle from generation to persistence.

The Storage Pipeline

The implementation follows a strict idempotent pattern to prevent duplicate data:

1. Load the source document and verify text content exists
2. Delete existing embeddings for the source (lines 52–57), ensuring clean slate updates
3. Chunk the text according to content type (PDF, HTML, or plain text)
4. Generate embeddings via the generate_embeddings utility
5. Bulk-INSERT all records in a single operation (lines 92–104)

The bulk insertion uses the repository pattern:

# commands/embedding_commands.py

records = [
    {
        "source": ensure_record_id(input_data.source_id),
        "order": idx,
        "content": chunk,
        "embedding": embedding,
    }
    for idx, (chunk, embedding) in enumerate(zip(chunks, embeddings))
]

await repo_insert("source_embedding", records)

Single-round-trip writes via repo_insert("source_embedding", records) minimize network overhead and allow SurrealDB to index the vector column efficiently as native arrays.

Query Patterns: Existence Checks and Retrieval

The architecture optimizes for read performance through three distinct query patterns, avoiding unnecessary vector data transfer when only metadata is needed.

Checking Embedding Existence

To determine if a source has embeddings without retrieving the vectors, the API routers in api/routers/sources.py (lines 197–200) use a boolean sub-query:

# SurrealQL SELECT clause

SELECT
    id, asset, created, title,
    (SELECT VALUE id FROM source_embedding 
     WHERE source = $parent.id LIMIT 1) != [] AS embedded
FROM source

This pattern evaluates to true only when at least one record exists, avoiding the memory overhead of fetching large vector arrays.

Counting Stored Chunks

The Source model in open_notebook/domain/notebook.py (lines 442–452) provides a helper method for pagination and UI indicators:

async def get_embedded_chunks(self):
    result = await repo_query(
        """
        SELECT count() AS chunks 
        FROM source_embedding 
        WHERE source=$id 
        GROUP ALL
        """,
        {"id": ensure_record_id(self.id)},
    )
    return result[0]["chunks"] if result else 0

Retrieving Vectors for Similarity Search

For semantic search operations, the system retrieves vectors in their original order:

emb_rows = await repo_query(
    """
    SELECT embedding, order 
    FROM source_embedding 
    WHERE source=$source_id 
    ORDER BY order
    """,
    {"source_id": ensure_record_id(source_id)},
)
embeddings = [row["embedding"] for row in emb_rows]

Because SurrealDB stores these as native array<float> values, the results can be fed directly into vector distance operators (e.g., <-> for cosine similarity) or external ANN libraries without serialization overhead.

Key Implementation Files

File	Role
open_notebook/domain/notebook.py	Defines SourceEmbedding model and chunk counting logic
commands/embedding_commands.py	Implements embed_source command with idempotent bulk insertion
api/routers/sources.py	Contains existence check queries and API endpoints
open_notebook/utils/embedding.py	Provides generate_embeddings utility used by the command

Summary

• Native Array Storage: Vectors persist as array<float> in source_embedding, compatible with SurrealDB's vector search operators
• Chunk-Level Granularity: Records represent text chunks (not full documents) with order fields enabling reconstruction
• Idempotent Writes: The embed_source command deletes existing records before insertion, preventing duplicates
• Optimized Queries: Boolean sub-queries check for existence without fetching vector data, while ORDER BY clauses preserve chunk sequences
• Single-Statement Bulk Operations: Batch inserts reduce round-trips and improve indexing performance

Frequently Asked Questions

What data type does SurrealDB use for vector embeddings in this implementation?

SurrealDB stores the embeddings as array<float> (native arrays of floating-point numbers). This native type allows the database to apply vector-specific operators like <-> (distance calculations) directly on the stored data without requiring external vector databases or conversion layers.

How does Open Notebook handle updates to existing source embeddings?

The system implements idempotent updates through the embed_source command. Before inserting new embeddings, it executes a DELETE operation to remove all existing records for that source ID (lines 52–57 in commands/embedding_commands.py). This ensures that re-processing a source never creates duplicate chunks while maintaining the latest vector representations.

What query pattern efficiently checks if embeddings exist without retrieving them?

The implementation uses a boolean existence sub-query: (SELECT VALUE id FROM source_embedding WHERE source = $parent.id LIMIT 1) != []. This pattern returns true or false without serializing the potentially large embedding arrays, optimizing API responses in api/routers/sources.py.

Why are embeddings stored at the chunk level rather than the document level?

Chunk-level storage (each record ≈ 512–1024 tokens) provides superior semantic search precision. Large documents contain multiple concepts; storing vectors per chunk allows the system to match specific passages rather than averaged document representations. The order field enables reconstruction of the original document when needed, while the source field maintains referential integrity for joins with metadata tables.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}