# Open Notebook Vector Search Implementation Details: SurrealDB Integration and Embedding Pipeline

> Explore open notebook vector search implementation details. Learn how embedding pipelines and SurrealDB integration power efficient cosine similarity matching for your content.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: internals
- Published: 2026-06-17

---

**Open Notebook implements vector search by embedding queries through a chunked mean-pooling pipeline and querying SurrealDB's built-in `fn::vector_search` function to perform cosine similarity matching against stored content embeddings.**

Open Notebook stores all content—sources, notes, and notebooks—in SurrealDB and leverages the database's native vector-search capabilities for semantic retrieval. The architecture combines a Python-based embedding pipeline with SurrealDB's stored procedures to deliver efficient similarity search across potentially long-form content.

## Embedding Pipeline: From Text to Vectors

The vector search process begins in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py), where the `generate_embedding` function transforms user queries into dense vectors suitable for similarity comparison.

### Handling Long Queries with Chunking

For queries exceeding the configured `CHUNK_SIZE` token limit, the system implements a sophisticated chunking strategy:

- The text is split into smaller chunks using `chunk_text()` from [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py)
- Each chunk is embedded individually through the model manager ([`open_notebook/ai/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/models.py))
- The resulting vectors are aggregated using `mean_pool_embeddings()` (lines 55-89 in [`embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/embedding.py)) to produce a single normalized query vector

This approach ensures that even lengthy user inputs generate a properly normalized embedding without truncation or information loss. For short queries, the system bypasses chunking and calls `generate_embeddings([text])` directly.

```python

# From open_notebook/utils/embedding.py (lines 260-274)

embed = await generate_embedding(keyword)

```

## SurrealDB Vector Search Execution

Once the embedding vector is generated, the search logic delegates to SurrealDB's built-in vector similarity function through a stored procedure call.

### The vector_search Function

The `vector_search` function in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) (lines 38-64) executes the search by calling `fn::vector_search` via the `repo_query` wrapper:

```python

# From open_notebook/domain/notebook.py

search_results = await repo_query(
    """
    SELECT * FROM fn::vector_search($embed, $results, $source, $note, $minimum_score);
    """,
    {
        "embed": embed,
        "results": results,
        "source": source,
        "note": note,
        "minimum_score": minimum_score,
    },
)

```

The `repo_query` function serves as a thin async wrapper around the SurrealDB driver, executing raw SurrealQL statements. The `fn::vector_search` stored procedure performs cosine similarity calculations against pre-computed embeddings stored in the database, returning the top-N most similar records filtered by the `minimum_score` threshold (defaulting to 0.2).

## API Layer and Fallback Strategy

The search functionality is exposed through a FastAPI router that validates configuration and handles edge cases.

### HTTP Endpoint Configuration

The vector search endpoint is defined in [`api/routers/search.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/search.py) (lines 21-34), which validates that an embedding model is configured before processing requests:

```python

# From api/routers/search.py

if search_request.type == "vector":
    results = await vector_search(
        keyword=search_request.query,
        results=search_request.limit,
        source=search_request.search_sources,
        note=search_request.search_notes,
        minimum_score=search_request.minimum_score,
    )

```

If no embedding model is configured, the API returns a 400 error with a clear message. The same endpoint supports traditional text search when the request type is set to `"text"`.

### Robust Fallback Handling

The implementation includes a resilience mechanism in the `text_search` function ([`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py), lines 11-18). When SurrealDB's text search encounters a "position overflow" error—a known limitation when highlighting large strings—the system automatically falls back to vector search:

```python
except RuntimeError as e:
    if "position overflow" in str(e):
        logger.warning(...)
        return await vector_search(keyword, results, source, note)

```

This ensures users receive semantic results even when exact text matching fails due to database limitations.

## Practical Implementation Examples

### Direct Python Integration

To execute a vector search programmatically within the Open Notebook backend:

```python
from open_notebook.domain.notebook import vector_search

# Search across both sources and notes

results = await vector_search(
    keyword="machine learning",
    results=10,
    source=True,
    note=True,
    minimum_score=0.3,
)

for r in results:
    print(r.id, r.title, r._score)  # _score contains cosine similarity

```

### HTTP API Request

Clients can access vector search through the REST API:

```bash
curl -X POST http://localhost:5055/search \
  -H "Content-Type: application/json" \
  -d '{
        "type": "vector",
        "query": "artificial intelligence",
        "limit": 5,
        "search_sources": true,
        "search_notes": false,
        "minimum_score": 0.25
      }'

```

The response includes matching records with fields such as `id`, `title`, `content`, and the computed similarity score.

## Summary

- **Open Notebook** stores all content embeddings in SurrealDB and uses the native `fn::vector_search` stored procedure for similarity matching.
- The embedding pipeline in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py) handles long queries through chunking and mean-pooling to maintain vector quality.
- The `vector_search` function in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) serves as the primary interface between the application and SurrealDB's vector capabilities.
- The API layer automatically falls back from text search to vector search when encountering SurrealDB position overflow errors.
- Configuration parameters like `minimum_score` (default 0.2) and `CHUNK_SIZE` allow fine-tuning of search behavior.

## Frequently Asked Questions

### How does Open Notebook handle long queries in vector search?

According to the source code in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py), queries exceeding the `CHUNK_SIZE` token limit are split into smaller chunks using `chunk_text()`, embedded individually, and then aggregated via `mean_pool_embeddings()` to produce a single normalized vector. This ensures semantic integrity regardless of input length.

### What database function performs the similarity search in SurrealDB?

Open Notebook utilizes SurrealDB's built-in stored procedure `fn::vector_search`, which performs cosine similarity calculations against pre-computed embeddings. The function is called through `repo_query()` in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) and accepts parameters for the embedding vector, result limits, source/note flags, and minimum similarity threshold.

### What happens when text search fails in Open Notebook?

When the `text_search` function encounters a SurrealDB "position overflow" error during highlighting of large strings, it automatically catches the `RuntimeError` and falls back to `vector_search()` with the same parameters. This fallback mechanism is implemented in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) (lines 11-18).

### How do I configure the minimum similarity threshold for vector searches?

The `minimum_score` parameter controls the similarity threshold and defaults to 0.2. When calling `vector_search()` directly or via the API endpoint at `/search`, specify this parameter to filter out low-similarity matches. The value represents the cosine similarity cutoff, with higher values returning more strictly matched results.