# How the Open Notebook Search Service Performs Vector and Full‑Text Search

> Learn how Open Notebook's search service performs vector and full-text search using a single FastAPI endpoint and SurrealDB for efficient querying.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-09

---

**Open Notebook routes every search request through a single FastAPI endpoint that delegates to SurrealDB either via a built‑in text index for full‑text search or via an embedded query vector for semantic similarity search, depending on the request type.**

The `lfnovo/open-notebook` repository implements a unified search architecture that supports both classic keyword matching and modern semantic retrieval. Understanding how the search service performs vector and full‑text search helps developers extend the system or debug ranking issues. Both modes share the same `POST /search` entry point but rely on separate SurrealDB functions and distinct preprocessing pipelines.

## Full‑Text Search Flow

When a `SearchRequest` arrives with `type = "text"`, the FastAPI router in [`api/routers/search.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/search.py) validates the payload and invokes `text_search()` from [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py). This function constructs a SurrealQL statement that calls the built‑in database function `fn::text_search`, which tokenises the query and matches it against indexed titles, notes, and full‑text fields.

SurrealDB handles tokenisation, ranking, and result retrieval natively. The database returns matching source and note records directly, which the router wraps into a uniform `SearchResponse` before sending them to the client.

## Vector Search Flow

For semantic queries (`type = "vector"`), the router first verifies that an embedding model is available through `model_manager.get_embedding_model()`. If configured, it calls `vector_search()` in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py), passing the user’s query string along with pagination and filtering flags.

Inside `vector_search()`, the query is sent to `generate_embedding()` in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py). If the text exceeds `CHUNK_SIZE`, it is split via `chunk_text()` and processed in batches of `EMBEDDING_BATCH_SIZE` (default `50`). The resulting chunk embeddings are fused into a single unit‑length vector by `mean_pool_embeddings()`.

This final embedding is passed to SurrealDB’s custom function `fn::vector_search`, which performs a similarity search against stored `source_embedding` and `note_embedding` vectors. Results are filtered by the `minimum_score` threshold and returned in similarity order.

## Embedding Generation Pipeline

The utilities in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py) centralise all embedding logic for the application. Long inputs are chunked in a content‑type aware manner, batched to respect provider rate limits, and pooled so that a single vector represents the entire query regardless of original length.

This same pipeline is reused by background jobs that embed sources and notes, ensuring consistency between indexed documents and search queries.

## Code Examples

### Using the Python Search Client

The `SearchService` wrapper in [`api/search_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/search_service.py) forwards calls to the FastAPI endpoint through the shared `api_client`.

```python
from api.search_service import search_service

# Full‑text search

results = search_service.search(
    query="machine learning ethics",
    search_type="text",
    limit=10,
    search_sources=True,
    search_notes=True,
)
print("Full‑text hits:", results)

# Vector (semantic) search

results = search_service.search(
    query="Explain transformer attention mechanisms",
    search_type="vector",
    limit=5,
    minimum_score=0.3,
)
print("Vector hits:", results)

```

### Calling the FastAPI Endpoint Directly

You can also exercise the router with a `TestClient` or any HTTP client.

```python
from fastapi.testclient import TestClient
from api.main import app

client = TestClient(app)

# Full‑text request

resp = client.post(
    "/search",
    json={
        "query": "privacy first AI",
        "type": "text",
        "limit": 15,
        "search_sources": True,
        "search_notes": False,
    },
)
print(resp.json())

# Vector request

resp = client.post(
    "/search",
    json={
        "query": "What are the advantages of vector search?",
        "type": "vector",
        "limit": 5,
        "minimum_score": 0.25,
    },
)
print(resp.json())

```

### Invoking `vector_search` Directly

For custom workflows, import `vector_search` from the domain layer and await its results.

```python
from open_notebook.domain.notebook import vector_search

async def find_similar_sources(query: str):
    matches = await vector_search(
        keyword=query,
        results=8,
        source=True,
        note=False,
        minimum_score=0.2,
    )
    return matches

```

## Summary

- **Full‑text search** is handled by `text_search()` in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py), which invokes SurrealDB’s `fn::text_search` against a built‑in text index.
- **Vector search** is handled by `vector_search()` in the same file, embedding the query via `generate_embedding()` and querying SurrealDB with `fn::vector_search`.
- **Embedding utilities** in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py) manage chunking, batching, and mean‑pooling so long queries still produce a single comparable vector.
- **Routing and validation** happen in [`api/routers/search.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/search.py), ensuring a uniform `SearchResponse` regardless of search mode.

## Frequently Asked Questions

### What is the difference between full‑text and vector search in Open Notebook?

Full‑text search relies on SurrealDB’s native text index to tokenise and rank keywords against source and note fields. Vector search converts the query into an embedding vector and performs a similarity search against pre‑computed `source_embedding` and `note_embedding` vectors stored in SurrealDB.

### How does Open Notebook handle long queries during vector search?

If a query exceeds `CHUNK_SIZE`, `generate_embedding()` splits the text via `chunk_text()`, embeds the chunks in batches of `EMBEDDING_BATCH_SIZE` (default `50`), and combines them with `mean_pool_embeddings()` into one unit‑length vector before searching.

### Which file validates that an embedding model is configured before vector search?

The FastAPI router in [`api/routers/search.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/search.py) checks `model_manager.get_embedding_model()` before dispatching a request to `vector_search()`. If no model is configured, the request fails at the API layer.

### Can I adjust the minimum similarity threshold for vector results?

Yes. Both the API endpoint and the direct `vector_search()` function accept a `minimum_score` parameter. The SurrealDB `fn::vector_search` function uses this value to filter out low‑similarity matches before returning the result set.