# How Vector Embeddings Are Managed and Queried in SurrealDB for Semantic Search

> Learn how to manage and query vector embeddings in SurrealDB for semantic search using native vector similarity searches and cosine scoring for efficient retrieval.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-07-04

---

**Open Notebook generates dense vector embeddings for documents and queries, then executes native vector similarity searches using SurrealDB's built-in `fn::vector_search` function to perform semantic retrieval with cosine similarity scoring.**

The `lfnovo/open-notebook` repository implements a complete semantic search pipeline that stores floating-point embeddings in SurrealDB's native vector columns and queries them using built-in nearest-neighbor functions. This architecture combines intelligent text chunking with SurrealDB's approximate nearest-neighbor indexing to deliver relevant results across source documents and notes.

## Embedding Generation and Chunking Strategy

The embedding pipeline lives in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py), specifically within the `generate_embedding` function (lines 9-75). This utility handles variable-length inputs by automatically chunking long texts and mean-pooling the results into a single dense vector.

The function implements a two-path strategy:

1. **Short text path**: When input tokens are below the `CHUNK_SIZE` threshold, the text is embedded directly via the configured model
2. **Long text path**: Exceeding inputs are split into chunks, each embedded separately, then combined using mean-pooling

```python
async def generate_embedding(
    text: str,
    content_type: Optional[ContentType] = None,
    file_path: Optional[str] = None,
    command_id: Optional[str] = None,
) -> List[float]:
    if not text or not text.strip():
        raise ValueError("Cannot generate embedding for empty text")
    text_tokens = token_count(text)

    # Short text → embed directly

    if text_tokens <= CHUNK_SIZE:
        embeddings = await generate_embeddings([text], command_id=command_id)
        return embeddings[0]

    # Long text → chunk, embed each chunk, then mean‑pool

    chunks = chunk_text(text, content_type=content_type, file_path=file_path)
    embeddings = await generate_embeddings(chunks, command_id=command_id)
    pooled = await mean_pool_embeddings(embeddings)
    return pooled

```

This approach ensures that every document—regardless of length—produces a single, fixed-dimension vector compatible with SurrealDB's `vector` column type.

## SurrealDB Vector Search Implementation

The core search logic resides in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) within the `vector_search` function (lines 738-754). This function bridges the embedding generation layer with SurrealDB's native vector capabilities by invoking the `fn::vector_search` SurrealQL function.

SurrealDB automatically builds an approximate nearest-neighbour index on vector columns, enabling efficient similarity lookups without manual index management. The function signature exposed to the application is:

```sql
fn::vector_search( embed VECTOR, results INT, source BOOL, note BOOL, minimum_score FLOAT )

```

- **`embed`**: The query embedding produced by `generate_embedding`
- **`results`**: Maximum number of matches to return
- **`source`** / **`note`**: Booleans controlling which record types are included
- **`minimum_score`**: Cosine-similarity threshold (defaults to approximately 0.2)

The Python implementation validates inputs, generates the query embedding, and executes the SurrealQL statement:

```python
async def vector_search(
    keyword: str,
    results: int,
    source: bool = True,
    note: bool = True,
    minimum_score=0.2,
):
    if not keyword:
        raise InvalidInputError("Search keyword cannot be empty")
    # ① Build embedding vector for the query

    from open_notebook.utils.embedding import generate_embedding
    embed = await generate_embedding(keyword)

    # ② Execute SurrealQL vector search

    search_results = await repo_query(
        """
        SELECT * FROM fn::vector_search($embed, $results, $source, $note, $minimum_score);
        """,
        {
            "embed": embed,
            "results": results,
            "source": source,
            "note": note,
            "minimum_score": minimum_score,
        },
    )
    return search_results

```

SurrealDB returns matching records enriched with computed similarity scores, which Open Notebook passes directly to API consumers.

## Database Query Execution

The `repo_query` function in [`open_notebook/database/repository.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/database/repository.py) (lines 65-76) handles the actual database communication. This generic helper opens a SurrealDB connection, sends the parameterized SurrealQL, parses `RecordID` objects, and returns plain Python dictionaries to the domain layer.

This abstraction allows the vector search logic to remain agnostic of connection management while ensuring proper resource cleanup and result serialization.

## API Layer and Fallback Strategy

The search endpoint in [`api/routers/search.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/search.py) exposes vector search to clients via a dedicated API route. When the request type is set to `"vector"`, the router delegates to the `vector_search` domain function after verifying that an embedding model is configured.

A critical resilience feature exists in the fallback logic: when `fn::text_search` fails due to a *highlight‑position overflow* (a SurrealDB limitation on very large strings), the system automatically falls back to vector search. This guarantees result delivery even when text-based highlighting encounters size constraints.

```python

# api/routers/search.py  (excerpt)

@router.post("/search", response_model=SearchResponse)
async def search(req: SearchRequest):
    if req.type == "vector":
        # Ensure an embedding model is configured; otherwise raise 400

        results = await vector_search(
            keyword=req.query,
            results=req.limit,
            source=req.include_source,
            note=req.include_note,
            minimum_score=req.min_score,
        )
    else:
        results = await text_search(...)
    return SearchResponse(matches=results)

```

## Summary

- **Dense vector storage**: Open Notebook stores floating-point embeddings in SurrealDB's native `vector` columns, which automatically maintain approximate nearest-neighbour indexes
- **Intelligent chunking**: The `generate_embedding` utility in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py) handles long documents by chunking and mean-pooling to produce single-vector representations
- **Native SurrealQL search**: Vector similarity queries execute via `fn::vector_search`, accepting parameters for result limits, source/note filtering, and minimum cosine similarity thresholds
- **Resilient architecture**: The system falls back from text search to vector search when encountering SurrealDB's highlight-position overflow limitations on large strings
- **Clean abstraction**: The `repo_query` helper in [`open_notebook/database/repository.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/database/repository.py) manages connection lifecycle and result parsing, keeping domain logic database-agnostic

## Frequently Asked Questions

### How does SurrealDB store vector embeddings in Open Notebook?

SurrealDB stores each embedding in a column of type `vector` and automatically builds an approximate nearest-neighbour index. According to the `lfnovo/open-notebook` source code, embeddings are passed as floating-point arrays to the database, where SurrealDB handles the indexing and similarity computation natively.

### What happens when a text search fails due to large document sizes?

When `fn::text_search` encounters a highlight-position overflow—a known SurrealDB limitation with very large strings—the code in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) (lines 11-20) automatically falls back to the vector search path. This ensures users always receive relevant results even when exact text matching is unavailable.

### How does Open Notebook handle documents that exceed the embedding model's token limit?

The `generate_embedding` function in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py) automatically detects oversized inputs using `token_count`, splits them into chunks via `chunk_text`, generates embeddings for each chunk, and combines them using `mean_pool_embeddings`. This produces a single vector representation regardless of original document length.

### What parameters control the scope and quality of vector search results?

The `fn::vector_search` function accepts five parameters: `embed` (the query vector), `results` (maximum matches), `source` and `note` (booleans filtering which record types to search), and `minimum_score` (a cosine similarity threshold, typically defaulting to 0.2). These are passed from the API layer through the `vector_search` domain function in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py).