# How Vector Embeddings Are Stored and Queried in SurrealDB: Inside Open Notebook

> Discover how Open Notebook stores and queries vector embeddings in SurrealDB. Learn about native array storage, efficient bulk inserts, and fast retrieval with SurrealQL.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-14

---

**Open Notebook stores vector embeddings as native `array<float>` values in a dedicated `source_embedding` table, using bulk INSERT operations for efficient writes and SurrealQL sub-queries for fast existence checks and retrieval.**

The `lfnovo/open-notebook` repository implements a production-ready vector storage layer using SurrealDB as its primary database. Understanding how vector embeddings are stored and queried in SurrealDB reveals an architecture optimized for RAG (Retrieval-Augmented Generation) pipelines, leveraging native array types, chunk-level granularity, and idempotent write patterns.

## Schema Design: The `source_embedding` Table

Vector embeddings are persisted in a dedicated table called **`source_embedding`**, defined by the `SourceEmbedding` model in **[`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py)** (lines 304–307). This table stores individual text chunks rather than entire documents, enabling fine-grained similarity search.

Each record contains four critical fields:

- **`source`** – A `RecordID` linking to the parent document in the `source` table
- **`order`** – An integer (0-based) preserving the chunk sequence for document reconstruction
- **`content`** – The raw text string of the chunk (used for debugging and regeneration)
- **`embedding`** – The dense vector as an `array<float>` (e.g., 1536 dimensions for OpenAI models)

This schema aligns with standard chunking strategies (512–1024 tokens), keeping vector dimensionality stable while enabling precise semantic matching.

## Storage Strategy: Bulk Insertion and Idempotency

The **`embed_source`** command in **[`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py)** orchestrates the complete embedding lifecycle from generation to persistence.

### The Storage Pipeline

The implementation follows a strict idempotent pattern to prevent duplicate data:

1. **Load** the source document and verify text content exists
2. **Delete** existing embeddings for the source (lines 52–57), ensuring clean slate updates
3. **Chunk** the text according to content type (PDF, HTML, or plain text)
4. **Generate** embeddings via the `generate_embeddings` utility
5. **Bulk-INSERT** all records in a single operation (lines 92–104)

The bulk insertion uses the repository pattern:

```python

# commands/embedding_commands.py

records = [
    {
        "source": ensure_record_id(input_data.source_id),
        "order": idx,
        "content": chunk,
        "embedding": embedding,
    }
    for idx, (chunk, embedding) in enumerate(zip(chunks, embeddings))
]

await repo_insert("source_embedding", records)

```

**Single-round-trip writes** via `repo_insert("source_embedding", records)` minimize network overhead and allow SurrealDB to index the vector column efficiently as native arrays.

## Query Patterns: Existence Checks and Retrieval

The architecture optimizes for read performance through three distinct query patterns, avoiding unnecessary vector data transfer when only metadata is needed.

### Checking Embedding Existence

To determine if a source has embeddings without retrieving the vectors, the API routers in **[`api/routers/sources.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/sources.py)** (lines 197–200) use a boolean sub-query:

```python

# SurrealQL SELECT clause

SELECT
    id, asset, created, title,
    (SELECT VALUE id FROM source_embedding 
     WHERE source = $parent.id LIMIT 1) != [] AS embedded
FROM source

```

This pattern evaluates to `true` only when at least one record exists, avoiding the memory overhead of fetching large vector arrays.

### Counting Stored Chunks

The `Source` model in **[`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py)** (lines 442–452) provides a helper method for pagination and UI indicators:

```python
async def get_embedded_chunks(self):
    result = await repo_query(
        """
        SELECT count() AS chunks 
        FROM source_embedding 
        WHERE source=$id 
        GROUP ALL
        """,
        {"id": ensure_record_id(self.id)},
    )
    return result[0]["chunks"] if result else 0

```

### Retrieving Vectors for Similarity Search

For semantic search operations, the system retrieves vectors in their original order:

```python
emb_rows = await repo_query(
    """
    SELECT embedding, order 
    FROM source_embedding 
    WHERE source=$source_id 
    ORDER BY order
    """,
    {"source_id": ensure_record_id(source_id)},
)
embeddings = [row["embedding"] for row in emb_rows]

```

Because SurrealDB stores these as native `array<float>` values, the results can be fed directly into vector distance operators (e.g., `<->` for cosine similarity) or external ANN libraries without serialization overhead.

## Key Implementation Files

| File | Role |
|------|------|
| [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) | Defines `SourceEmbedding` model and chunk counting logic |
| [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py) | Implements `embed_source` command with idempotent bulk insertion |
| [`api/routers/sources.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/sources.py) | Contains existence check queries and API endpoints |
| [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py) | Provides `generate_embeddings` utility used by the command |

## Summary

- **Native Array Storage**: Vectors persist as `array<float>` in `source_embedding`, compatible with SurrealDB's vector search operators
- **Chunk-Level Granularity**: Records represent text chunks (not full documents) with `order` fields enabling reconstruction
- **Idempotent Writes**: The `embed_source` command deletes existing records before insertion, preventing duplicates
- **Optimized Queries**: Boolean sub-queries check for existence without fetching vector data, while `ORDER BY` clauses preserve chunk sequences
- **Single-Statement Bulk Operations**: Batch inserts reduce round-trips and improve indexing performance

## Frequently Asked Questions

### What data type does SurrealDB use for vector embeddings in this implementation?

SurrealDB stores the embeddings as **`array<float>`** (native arrays of floating-point numbers). This native type allows the database to apply vector-specific operators like `<->` (distance calculations) directly on the stored data without requiring external vector databases or conversion layers.

### How does Open Notebook handle updates to existing source embeddings?

The system implements **idempotent updates** through the `embed_source` command. Before inserting new embeddings, it executes a DELETE operation to remove all existing records for that source ID (lines 52–57 in [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py)). This ensures that re-processing a source never creates duplicate chunks while maintaining the latest vector representations.

### What query pattern efficiently checks if embeddings exist without retrieving them?

The implementation uses a **boolean existence sub-query**: `(SELECT VALUE id FROM source_embedding WHERE source = $parent.id LIMIT 1) != []`. This pattern returns `true` or `false` without serializing the potentially large `embedding` arrays, optimizing API responses in [`api/routers/sources.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/sources.py).

### Why are embeddings stored at the chunk level rather than the document level?

**Chunk-level storage** (each record ≈ 512–1024 tokens) provides superior semantic search precision. Large documents contain multiple concepts; storing vectors per chunk allows the system to match specific passages rather than averaged document representations. The `order` field enables reconstruction of the original document when needed, while the `source` field maintains referential integrity for joins with metadata tables.