# Implementing Semantic Search Across Notebook Sources in Open Notebook

> Implement semantic search in Open Notebook using SurrealDB vector search. Learn about automatic fallback from text to vector search for seamless results. Enhance your notebook experience.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-22

---

**Open Notebook implements semantic search across notebook sources using SurrealDB's vector search capabilities with automatic fallback from text search to vector search when highlight position overflows occur.**

The open-source Open Notebook project (lfnovo/open-notebook) stores every notebook, source document, and note in SurrealDB, exposing two complementary search strategies through its REST API. By combining full-text indexing with vector embeddings, the system delivers relevant results even when traditional text search hits database limitations. Understanding how these components interact enables developers to leverage both semantic and lexical search effectively.

## How Semantic Search Works in Open Notebook

Open Notebook provides dual search capabilities that work together to ensure robust information retrieval across all stored content.

### Text Search with Automatic Fallback

The system defaults to SurrealDB's built-in `fn::text_search` function, which utilizes the database's full-text index to return matches with highlight positions. When the highlight position overflows—a known SurrealDB limitation—the code automatically catches this error and transparently invokes `vector_search` as a fallback, logging the transition for monitoring purposes.

### Vector Search Architecture

For semantic queries, the system embeds the search query using the configured embedding model via the **Esperanto** AI-provider wrapper, then executes `fn::vector_search` in SurrealDB. This performs a nearest-neighbor lookup on stored embeddings, returning the top-k most similar source-note pairs along with similarity scores.

## Core Implementation Details

The search functionality spans several critical files in the repository, each handling specific responsibilities within the architecture.

### Domain Layer: [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py)

The `text_search` function (lines 696-735) serves as an async wrapper around `fn::text_search`, catching highlight-position overflow errors and triggering the fallback to `vector_search`. The `vector_search` function (lines 738-766) builds query embeddings using the ModelManager and returns nearest-neighbor results, raising a clear HTTP 400 error upstream if no embedding model is configured.

### API Layer: [`api/routers/search.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/search.py)

The POST `/search` endpoint (lines 10-44) validates request payloads and dispatches to either `text_search` or `vector_search` based on the `type` field. It returns a structured `SearchResponse` and validates that embedding models are available before attempting vector operations.

### Service Layer: [`api/search_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/search_service.py)

This thin wrapper (lines 13-30) handles operational logging before forwarding requests to the router functions, maintaining clean separation between domain logic and infrastructure concerns.

### Embedding Infrastructure: [`open_notebook/ai/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/models.py)

Provides the `ModelManager` class that integrates with the Esperanto AI-provider wrapper to generate query embeddings for semantic search operations.

## Search Request Workflow

1. **Client Request**: The frontend sends a JSON payload specifying the query type, search scope, and result limits.
2. **Router Dispatch**: [`api/routers/search.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/search.py) receives the request and selects the appropriate search strategy based on the `type` field.
3. **Embedding Generation**: For vector searches, `vector_search` invokes the ModelManager to convert the query into a dense vector.
4. **Database Lookup**: The generated embedding passes to `fn::vector_search`, which performs the nearest-neighbor lookup in SurrealDB.
5. **Response Serialization**: The router formats results into a `SearchResponse` and returns them to the client.

## Implementation Examples

### Calling the Search Endpoint via Python

```python
import httpx

payload = {
    "query": "semantic search implementation",
    "type": "vector",
    "limit": 5,
    "search_sources": True,
    "search_notes": True,
    "minimum_score": 0.3,
}

async def search():
    async with httpx.AsyncClient() as client:
        resp = await client.post(
            "http://localhost:5055/api/search", json=payload
        )
        resp.raise_for_status()
        return resp.json()

results = await search()
print(results)

```

### Triggering Automatic Fallback

No additional client code is required to handle text search overflow. Simply request `"type": "text"` and the backend automatically logs a warning and returns vector results if the overflow occurs.

### Manual Query Embedding

```python
from open_notebook.ai.models import ModelManager

async def embed_query(query: str) -> list[float]:
    manager = ModelManager()
    embedder = await manager.get_embedding_model()
    return await embedder.embed_text(query)

embedding = await embed_query("Explain LangGraph")

```

## Summary

- Open Notebook stores all content in SurrealDB and exposes both text and vector search strategies through a unified API.
- The `text_search` function in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) automatically falls back to vector search when highlight position overflows occur.
- Vector search uses the Esperanto AI-provider wrapper and `fn::vector_search` for nearest-neighbor lookup on stored embeddings.
- The POST `/search` endpoint in [`api/routers/search.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/search.py) validates requests and returns structured `SearchResponse` objects.
- Developers can trigger semantic search explicitly with `"type": "vector"` or rely on automatic fallback from text search.

## Frequently Asked Questions

### What happens if the text search highlight position overflows?

When SurrealDB's `fn::text_search` returns a highlight position overflow error, the `text_search` function in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) automatically catches this exception and transparently invokes `vector_search` as a fallback. The system logs this transition while returning semantic results to the user without requiring client-side handling.

### How does Open Notebook generate embeddings for semantic search?

The system uses the **Esperanto** AI-provider wrapper accessed through `ModelManager` in [`open_notebook/ai/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/models.py). When processing a vector search request, the `vector_search` function builds the query embedding using the configured embedding model before passing it to SurrealDB's `fn::vector_search` function.

### What error occurs if I request vector search without configuring an embedding model?

The API returns an HTTP 400 error with a clear message indicating that no embedding model is configured. This validation occurs in the `vector_search` function (lines 738-766 of [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py)) and is propagated through the `/api/search` endpoint before attempting any database operations.

### Can I search across both notebook sources and notes simultaneously?

Yes. The search endpoint accepts boolean parameters `search_sources` and `search_notes` in the request payload. When both are set to `true`, the system queries across all stored content types and returns unified results ranked by relevance or similarity score.