# How Open Notebook Handles Vector Embeddings for Semantic Search in SurrealDB

> Discover how Open Notebook manages vector embeddings for semantic search in SurrealDB. Learn about its utility layer, async job commands, and native vector similarity functions.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-12

---

**Open Notebook stores and queries vector embeddings entirely within SurrealDB using a unified utility layer, asynchronous job commands, and native database vector similarity functions.**

Open Notebook is an open-source knowledge management system that leverages SurrealDB as its primary data store. To enable semantic search across sources and notes, the application implements a complete embedding pipeline that keeps vectors inside the database rather than relying on external vector stores, creating a single source of truth for all semantic data.

## Embedding Generation Architecture

The foundation of Open Notebook's semantic search resides in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py). This module provides model-agnostic utilities that normalize text of any length into dense vectors suitable for storage and comparison.

### Handling Variable-Length Inputs

The `generate_embedding()` function processes text through an intelligent routing strategy. For short inputs, it calls the embedding model directly. For longer content, it automatically invokes `chunk_text()` from [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py) to split the text into manageable segments, processes these chunks in batches using `generate_embeddings()`, and consolidates the results via `mean_pool_embeddings()` to produce a single normalized vector regardless of original input size.

### Configuration and Resilience

The embedding pipeline respects three environment variables defined at the top of [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py):

- `OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE` controls the throughput for batch processing
- `EMBEDDING_MAX_RETRIES` defines the retry count for transient API failures
- `EMBEDDING_RETRY_DELAY` specifies the wait duration between retry attempts

This configuration ensures robust handling of rate limits and network interruptions during vectorization.

## Persisting Embeddings via Asynchronous Commands

When a source document is created or updated, Open Notebook triggers the `embed_source` command defined in [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py). This command runs asynchronously through the job queue to prevent HTTP connection pool exhaustion during large document processing.

### Idempotent Storage Pattern

The `embed_source` command implements an idempotent write pattern. It first loads the Source record and deletes any existing `source_embedding` rows for that specific source. It then detects the appropriate content type, chunks the full text, and calls `generate_embeddings()` to obtain vectors for each chunk. Finally, it bulk-inserts records into SurrealDB using `repo_insert("source_embedding", records)`.

Each stored embedding record contains the source reference, chunk order, original content, and the vector embedding itself:

```python

# Inside commands/embedding_commands.py

embeddings = await generate_embeddings(chunks, command_id=cmd_id)
records = [
    {
        "source": ensure_record_id(input_data.source_id),
        "order": idx,
        "content": chunk,
        "embedding": embedding,
    }
    for idx, (chunk, embedding) in enumerate(zip(chunks, embeddings))
]
await repo_insert("source_embedding", records)

```

## Querying Vectors with SurrealDB Native Functions

Search requests entering through [`api/routers/search.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/search.py) delegate to the `vector_search` function in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py). This domain function orchestrates the conversion of user queries into database-ready vector searches using SurrealDB's built-in capabilities.

### Query Vectorization

The `vector_search` function first transforms the user's keyword string into an embedding by calling `generate_embedding()`, automatically handling chunking and mean pooling if the query exceeds length thresholds. This ensures the query vector is compatible with the stored embeddings regardless of input length.

### Cosine Similarity Search

The function passes the query vector along with pagination and filtering parameters to SurrealDB's native `fn::vector_search` function:

```sql
SELECT * FROM fn::vector_search($embed, $results, $source, $note, $minimum_score);

```

SurrealDB computes cosine similarity between the query vector and all stored `source_embedding` vectors, returning the top-N matching sources (and optionally notes) whose similarity exceeds the specified `minimum_score`. This calculation occurs entirely within the database engine, eliminating network overhead for vector comparisons.

## Practical Implementation Examples

### Generating a Query Embedding

```python
from open_notebook.utils.embedding import generate_embedding

# Convert a search term into a normalized embedding vector

query_vector = await generate_embedding("What is the impact of climate change on Arctic ecosystems?")

```

### Triggering Source Vectorization

```python

# Assume `source` is an instance of open_notebook.domain.notebook.Source

command_id = await source.vectorize()   # returns the job ID for tracking

```

### Executing Semantic Search

```python
from open_notebook.domain.notebook import vector_search

# Get the 10 most relevant sources for the query vector

results = await vector_search(
    keyword="climate change Arctic",
    results=10,
    source=True,
    note=False,
    minimum_score=0.25,
)

```

## Summary

- Open Notebook stores all vector embeddings inside SurrealDB, maintaining a single source of truth for semantic data rather than using external vector databases.
- The `generate_embedding()` utility in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py) handles automatic text chunking and mean pooling to normalize variable-length inputs into fixed-size vectors.
- Asynchronous `embed_source` commands in [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py) prevent resource exhaustion while bulk-inserting chunk embeddings via `repo_insert("source_embedding", records)`.
- Search queries are vectorized and passed to SurrealDB's native `fn::vector_search` function for cosine similarity calculation against stored `source_embedding` records.
- Environment variables `OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE`, `EMBEDDING_MAX_RETRIES`, and `EMBEDDING_RETRY_DELAY` provide operational control over the embedding pipeline resilience.

## Frequently Asked Questions

### Where does Open Notebook store vector embeddings?

Open Notebook stores vector embeddings directly inside SurrealDB as records in the `source_embedding` table. This design eliminates the need for external vector databases and allows the application to use SurrealDB's native `fn::vector_search` function for cosine similarity calculations against the stored vectors.

### How does Open Notebook handle documents that exceed the embedding model's context window?

The `generate_embedding()` function in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py) automatically detects long inputs and routes them through `chunk_text()` from [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py). It embeds each chunk separately using `generate_embeddings()`, then applies `mean_pool_embeddings()` to combine the chunk vectors into a single normalized vector representing the entire document.

### What happens if the embedding API fails during processing?

The embedding utilities implement automatic retry logic configured by the `EMBEDDING_MAX_RETRIES` and `EMBEDDING_RETRY_DELAY` environment variables. For source documents, the `embed_source` command runs asynchronously via the job queue, isolating failures and preventing HTTP connection pool exhaustion from blocking the main application thread.

### How can I trigger semantic search in my own Open Notebook instance?

Import the `vector_search` function from [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) and pass your query string along with parameters for result count, content type filters (`source` and `note` booleans), and minimum similarity score. The function automatically vectorizes your query and returns matching records from SurrealDB's `source_embedding` table using the database's built-in vector similarity function.