# How Vector Embeddings Are Stored and Queried in SurrealDB for Semantic Search

> Learn how SurrealDB stores and queries vector embeddings for semantic search. Discover efficient storage techniques and powerful retrieval methods to enhance your search capabilities.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: deep-dive
- Published: 2026-06-22

---

**Open Notebook stores vector embeddings as native `array<float>` fields in a dedicated `source_embedding` table, using bulk INSERT operations for efficient storage and SurrealQL subqueries for lightweight existence checks and similarity search retrieval.**

Open Notebook leverages SurrealDB's native array support to manage dense vector embeddings for semantic search across document chunks. This implementation, found in the `lfnovo/open-notebook` repository, stores embeddings in a dedicated table linked to source documents, enabling efficient similarity matching and retrieval. Understanding how these vectors are stored and queried reveals the architectural patterns that make the system both performant and scalable.

## Storage Schema in the `source_embedding` Table

Open Notebook persists every vector embedding in a dedicated SurrealDB table called **`source_embedding`**. Each record represents a single **chunk** of the original source text and contains the vector along with metadata required for reconstruction and querying.

### Table Structure and Field Types

The schema is declared by the `SourceEmbedding` model in **[`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py)**. The class inherits from `ObjectModel` and sets `table_name = "source_embedding"`, defining the following fields:

- **`source`** (`RecordID`): The SurrealDB ID of the parent `Source` record, establishing the foreign key relationship.
- **`order`** (`int`): The chunk's 0-based position in the original document, enabling accurate reassembly of text.
- **`content`** (`string`): The raw text of the chunk, retained for debugging and regeneration scenarios.
- **`embedding`** (`array<float>`): The dense vector returned by the embedding model, stored as a native SurrealDB array.

This design ensures that vector data remains tightly coupled with its textual context while maintaining queryable relationships to parent documents.

## Insertion Pipeline for Vector Embeddings

When a source document is processed, the system executes a deterministic pipeline to generate and store embeddings. This operation is handled by the **`embed_source`** command defined in **[`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py)**.

### The `embed_source` Command Workflow

The command performs five distinct steps to ensure data consistency and efficiency:

1. **Load** the source and verify it contains processable text.
2. **Delete** any existing embeddings for the same source to maintain idempotency.
3. **Chunk** the full text according to content type (PDF, HTML, or plain text).
4. **Generate** embeddings for each chunk via the unified `generate_embeddings` utility.
5. **Bulk-INSERT** a list of dictionaries into the `source_embedding` table.

This approach prevents duplicate embeddings and ensures that regenerating vectors for an updated source replaces stale data atomically.

### Bulk Insert Implementation

The system uses a single batch INSERT operation to minimize round-trips and allow SurrealDB to index the vector column efficiently:

```python

# commands/embedding_commands.py – embed_source_command

source = await Source.get(input_data.source_id)

# … chunking logic omitted …

embeddings = await generate_embeddings(chunks, command_id=get_command_id(input_data))

records = [
    {
        "source": ensure_record_id(input_data.source_id),
        "order": idx,
        "content": chunk,
        "embedding": embedding,
    }
    for idx, (chunk, embedding) in enumerate(zip(chunks, embeddings))
]

await repo_insert("source_embedding", records)   # bulk INSERT

```

By passing a list of records to `repo_insert`, SurrealDB stores the vectors as native array fields without requiring serialization into binary blobs or external storage.

## Querying Patterns and Retrieval Strategies

Various API components query the `source_embedding` table to check existence, count stored chunks, or retrieve vectors for similarity calculations.

### Existence Checks Without Payload Transfer

To determine whether a source has embeddings without transferring heavy vector data, the routers use a boolean subquery in **[`api/routers/sources.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/sources.py)**:

```python

# api/routers/sources.py – SELECT clause

SELECT
    id, asset, created, title, updated, topics, command,
    (SELECT VALUE count() FROM source_insight WHERE source = $parent.id GROUP ALL)[0].count OR 0 AS insights_count,
    (SELECT VALUE id FROM source_embedding WHERE source = $parent.id LIMIT 1) != [] AS embedded
FROM source

```

The `embedded` field evaluates to `true` when at least one row exists in `source_embedding` for the source. This lightweight check avoids pulling the full `array<float>` payload across the network when only a boolean status is required.

### Chunk Counting and Metadata

The `Source` domain model provides a method to count stored chunks without retrieving the embeddings themselves. In **[`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py)**, the `get_embedded_chunks` method executes:

```python

# open_notebook/domain/notebook.py – get_embedded_chunks()

result = await repo_query(
    """
    SELECT count() AS chunks FROM source_embedding WHERE source=$id GROUP ALL
    """,
    {"id": ensure_record_id(self.id)},
)
return result[0]["chunks"] if result else 0

```

This pattern supports UI indicators showing how many chunks are indexed for a given source document.

### Similarity Search Retrieval

For semantic search operations, the system retrieves all embeddings for a specific source using an ordered query:

```python

# Pattern used in search services

emb_rows = await repo_query(
    """
    SELECT embedding, order FROM source_embedding WHERE source=$source_id ORDER BY order
    """,
    {"source_id": ensure_record_id(source_id)},
)
embeddings = [row["embedding"] for row in emb_rows]

```

SurrealDB's native array type allows the `embedding` field to be used directly in vector-distance operations (such as the `<->` operator) without transformation, or exported to external ANN libraries for approximate nearest neighbor search.

## Design Benefits of Native Array Storage

This architecture leverages SurrealDB-specific features to optimize for the semantic search workload:

- **Chunk-level granularity**: Each vector corresponds to a manageable text chunk (typically 512-1024 tokens), keeping dimensionality stable and enabling fine-grained similarity matching.
- **Fast bulk writes**: Single-statement INSERT batches reduce network round-trips and allow SurrealDB to build efficient indexes on the vector column.
- **Composable relationships**: The `source` foreign key enables JOIN operations that combine metadata (titles, topics) with embedding data in a single request.
- **Storage efficiency**: Native `array<float>` fields avoid the overhead of JSON serialization or base64 encoding required by text-based vector storage schemes.

## Summary

- **Storage Location**: Vector embeddings reside in the `source_embedding` table as native `array<float>` fields, defined in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py).
- **Write Pattern**: The `embed_source` command in [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py) uses bulk INSERT operations after deleting existing records to ensure idempotency.
- **Existence Checking**: Lightweight boolean queries in [`api/routers/sources.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/sources.py) verify embedding status without transferring vector payloads.
- **Retrieval Strategy**: Ordered SELECT queries fetch embeddings by source ID for similarity search, leveraging SurrealDB's native array support for distance calculations.
- **Chunk Architecture**: Each record represents a text chunk with ordering metadata, enabling document reconstruction and granular semantic matching.

## Frequently Asked Questions

### What data type does SurrealDB use for vector embeddings in Open Notebook?

SurrealDB stores vector embeddings as **`array<float>`** native fields. This type preserves the dense vector structure without requiring serialization, allowing direct use in mathematical operations and distance calculations within SurrealQL queries.

### How does Open Notebook ensure idempotency when storing embeddings?

The `embed_source` command first executes a DELETE operation to remove any existing records in `source_embedding` matching the source ID before inserting new vectors. This pattern, implemented in [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py), ensures that reprocessing a source replaces stale embeddings rather than creating duplicates.

### Can SurrealDB perform native vector similarity calculations on these embeddings?

Yes. Because embeddings are stored as native arrays, SurrealDB can apply vector operators like `<->` (Euclidean distance) directly within queries. The retrieved arrays can also be exported to Python-based ANN libraries for approximate nearest neighbor search when working with large vector collections.

### What is the typical granularity for stored embeddings?

Open Notebook stores one embedding per **text chunk**, with each chunk representing approximately 512-1024 tokens of the original document. The `order` field in `source_embedding` records maintains the sequence, allowing reconstruction of the full document context from retrieved chunks.