# How SurrealDB Handles Vector Embeddings and Semantic Search Across Sources

> SurrealDB efficiently handles vector embeddings and semantic search using its fn::vector_search function. Discover how it enables fast k-nearest-neighbor queries and cosine-similarity searches across all your data.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: deep-dive
- Published: 2026-06-19

---

**SurrealDB stores dense vector embeddings in record `embed` fields and executes fast k-nearest-neighbor queries using the built-in `fn::vector_search` function, enabling cosine-similarity searches across sources and notes with configurable similarity thresholds.**

Open Notebook leverages SurrealDB's native vector capabilities to power semantic search without external databases. The application stores every piece of content—from web sources to user notes—as records containing high-dimensional embeddings in an `embed` field. This architecture allows SurrealDB to perform lightning-fast similarity calculations directly within the database engine using built-in scalar functions.

## How SurrealDB Stores Vector Embeddings

Open Notebook persists **vector embeddings** alongside content records to enable semantic retrieval. When a user adds or updates content, the system generates a dense vector representation and saves it to the record's `embed` field using SurrealDB's native array type for high-dimensional data.

### Generating Embeddings with ModelManager

The embedding pipeline lives in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py). The `generate_embedding` function (lines 48‑‑52) converts raw text into normalized vectors using a unified interface that wraps the configured embedding model via `ModelManager`:

```python
from open_notebook.utils.embedding import generate_embedding

# Generate embedding for user query

embed = await generate_embedding("What are the benefits of vector search?")

```

For batch operations, the utility provides a corresponding batch method to efficiently process multiple documents. All vectors are normalized before storage to ensure consistent cosine similarity calculations during search.

## The Vector Search Implementation

SurrealDB exposes a built-in scalar function **`fn::vector_search`** that performs k-nearest-neighbor (KNN) queries over any table containing vector data. Open Notebook utilizes this function to execute semantic search without transferring data to external vector stores.

### SurrealQL Query Structure

The `vector_search` function in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) (lines 738‑‑766) constructs a SurrealQL statement that accepts five parameters:

```sql
SELECT * FROM fn::vector_search(
    $embed,          -- embedding vector
    $results,       -- maximum results to return
    $source,        -- boolean to include sources table
    $note,          -- boolean to include notes table
    $minimum_score  -- similarity threshold (0.0 to 1.0)
);

```

SurrealDB computes the **cosine similarity** between the query vector and every stored `embed` vector, filters results below the threshold, sorts by similarity, and returns the top matches. This executes entirely within the database engine for optimal performance.

## End-to-End Semantic Search Workflow

The semantic search pipeline spans multiple modules, from initial vectorization to final API delivery.

### Vector Generation and Persistence

When content requires embedding, `Source.vectorize()` in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) (lines 477‑‑499) enqueues an `embed_source` command. This command eventually persists the vector to the record:

```python

# Queue embedding job

command_id = await source.vectorize()

# The embed_source command stores the vector in SurrealDB

await embed_source_command(source_id, embed)  # Updates embed field

```

### Query Execution and Fallback Logic

The `vector_search` function handles query routing and error recovery. According to the source code in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) (lines 11‑‑19), the system implements a fallback mechanism: if a traditional text search throws a "position overflow" error (a known SurrealDB bug with large highlight fields), the code automatically switches to vector search to guarantee results rather than returning a 500 error.

### API Integration

The public search endpoint in [`api/routers/search.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/search.py) (lines 19‑‑24) forwards requests to the domain layer. Clients can trigger semantic search via HTTP parameters:

```python

# Client request: GET /search?type=vector&keyword=privacy&results=10&source=true&note=true

# Router delegates to domain function

results = await vector_search(
    keyword="privacy",
    results=10,
    source=True,
    note=True,
    minimum_score=0.2,
)

```

All database operations execute asynchronously through the repository helper `repo_query` (defined in [`open_notebook/database/repository.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/database/repository.py) at line 65), which sends raw SurrealQL to the driver and returns parsed objects.

## Performance and Error Handling

Open Notebook optimizes for reliability by combining synchronous vector storage with resilient query patterns. The asynchronous `repo_query` helper ensures non-blocking database calls, while the fallback logic guarantees that search requests always return relevant results even when text indexing encounters edge-case errors.

## Summary

- **Storage**: SurrealDB stores normalized vectors in record `embed` fields as high-dimensional arrays.
- **Search**: The `fn::vector_search` function performs native cosine-similarity KNN queries across configurable tables.
- **Pipeline**: `generate_embedding` creates vectors, `Source.vectorize()` persists them, and `vector_search()` queries them using SurrealQL.
- **Resilience**: Automatic fallback from text to vector search prevents failures due to "position overflow" errors.
- **Source files**: Key logic resides in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py) (generation), [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) (search and persistence), and [`api/routers/search.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/search.py) (API routing).

## Frequently Asked Questions

### What embedding models does Open Notebook support?

Open Notebook supports any embedding model compatible with the `ModelManager` utility in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py). The system uses a unified interface to call the configured model, allowing integration with both local and remote embedding providers as long as they return normalized float vectors.

### How does SurrealDB calculate similarity between vectors?

SurrealDB computes **cosine similarity** between the query embedding and stored `embed` vectors. The `fn::vector_search` function sorts results by this similarity score and filters out any matches below the `$minimum_score` threshold (ranging from 0 to 1), returning only the most semantically relevant results.

### Can I search across both sources and notes simultaneously?

Yes. The `vector_search` function accepts boolean parameters `$source` and `$note` that control which tables to include in the query. Setting both to `true` searches across both content types in a single query, while setting one to `false` restricts results to the specific table.

### What happens when text search fails in Open Notebook?

When a text search query triggers a "position overflow" error—a known SurrealDB issue with large highlight fields—the system automatically falls back to vector search. This fallback logic in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) (lines 11‑‑19) ensures users always receive search results rather than encountering a 500 server error.