# Full-Text and Vector Search in Open Notebooks: Implementation Deep Dive

> Discover how Open Notebook integrates full-text and vector search with SurrealDB. Learn about hybrid search and semantic understanding by exploring the lfnovo/open-notebook repo.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: deep-dive
- Published: 2026-07-02

---

**Open Notebook implements hybrid search using SurrealDB's built‑in full‑text engine with automatic fallback to vector similarity search when literal keyword matching fails or semantic understanding is required.**

The open‑source Open Notebook project (`lfnovo/open‑notebook`) stores notebooks, sources, and notes in SurrealDB, exposing two complementary search strategies orchestrated by a thin service layer. This article examines the technical implementation of full‑text and vector search in notebooks, tracing the execution path from the FastAPI endpoint down to the database functions.

## Search Architecture Overview

Open Notebook provides **full‑text search** for literal keyword matches and **vector (embedding) search** for semantic similarity. These strategies operate within the [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) module, where the `text_search` coroutine attempts the full‑text path first. When SurrealDB throws a *position overflow* error—common with large or multi‑byte text chunks—the system transparently falls back to the `vector_search` coroutine. Both methods raise `DatabaseOperationError` on unrecoverable failures, ensuring the API surface returns explicit errors rather than empty results.

## The SearchService Abstraction

The entry point for all search operations is [`api/search_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/search_service.py), which defines a singleton `search_service` that forwards request parameters to the underlying API client.

```python
from api.search_service import search_service

# Service forwards to api_client.search with normalized parameters

results = search_service.search(
    query="machine learning",
    search_type="text",
    limit=20
)

```

This thin wrapper ensures the front‑end communicates through a consistent interface while the heavy lifting occurs in the domain layer.

## Full‑Text Search Implementation

The core full‑text logic resides in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) within the `text_search` coroutine. It executes SurrealDB's native `fn::text_search` function against the raw text of sources and notes.

```python
search_results = await repo_query(
    """
    select *
    from fn::text_search($keyword, $results, $source, $note)
    """,
    {"keyword": keyword, "results": results,
     "source": source, "note": note},
)

```

This query searches across both **sources** and **notes** simultaneously, returning records that contain literal keyword matches. The function accepts boolean flags to toggle searching across sources, notes, or both, along with a `minimum_score` threshold to filter relevance.

## Vector Search and Embedding Generation

When full‑text search is insufficient or explicitly requested, the `vector_search` coroutine in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) performs semantic similarity matching. The process involves two steps:

1. **Embedding generation**: The query text is converted to a vector embedding via `open_notebook.utils.embedding.generate_embedding`.
2. **Similarity lookup**: SurrealDB's `fn::vector_search` compares the query embedding against pre‑computed embeddings stored for sources and notes.

```python

# Inside vector_search coroutine

query_embedding = await generate_embedding(query)
similar_results = await repo_query(
    """
    select *, vector::similarity($query_embedding, embedding) as score
    from source, note
    """,
    {"query_embedding": query_embedding}
)

```

This approach captures conceptual meaning rather than exact lexical matches, enabling discovery of relevant content even when keywords differ.

## Automatic Fallback Strategy

The system implements resilient error handling for edge cases in full‑text processing. When `text_search` encounters a *position overflow* exception (SurrealDB error code relating to text indexing limits on large content), it catches the error and automatically triggers the `vector_search` path.

```python

# Exception handling in notebook.py

except SurrealDBError as e:
    if "position overflow" in str(e):
        logger.warning("Full-text position overflow, falling back to vector search")
        return await vector_search(keyword, results, source, note)
    raise DatabaseOperationError(f"Search failed: {e}")

```

This transparent fallback ensures users receive meaningful results even when the text engine hits indexing limitations on very large documents.

## Assembling Complete Context

For downstream workflows requiring the full content of a notebook—such as podcast generation or long‑form LLM prompts—the `Notebook.get_context` method in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) retrieves the complete text corpus.

```python
nb = await Notebook.get("notebook:my-notebook-id")
full_context_md = await nb.get_context()  # Returns markdown with sources and notes

```

This method loads the `full_text` field from each source and the `content` field from each note, then formats them into structured markdown sections (e.g., `## Source: ...`) suitable for language model consumption.

## Practical Usage Examples

### Performing a Full‑Text Search

```python
from api.search_service import search_service

results = search_service.search(
    query="privacy first",
    search_type="text",  # Optional; default behavior

    limit=10,
    search_sources=True,
    search_notes=True,
    minimum_score=0.2,
)

# Returns list of dicts with source/note metadata

```

### Forcing Vector Search for Semantic Queries

```python
results = search_service.search(
    query="how to fine‑tune a transformer?",
    search_type="vector",  # Bypasses full‑text completely

    limit=10,
)

```

### HTTP API Call

```bash
curl -X GET "http://localhost:5055/search?query=machine%20learning&limit=20" \
     -H "Accept: application/json"

```

### Retrieving Notebook Context for LLMs

```python
from open_notebook.domain.notebook import Notebook

async def get_notebook_corpus(notebook_id: str):
    nb = await Notebook.get(f"notebook:{notebook_id}")
    if nb:
        return await nb.get_context()  # Markdown formatted string

    return None

```

## Summary

- **Dual search strategy**: Open Notebook combines SurrealDB's `fn::text_search` for literal matching with `fn::vector_search` for semantic similarity, implemented in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py).
- **Automatic resilience**: Full‑text search automatically falls back to vector search when encountering *position overflow* errors, ensuring continuous service availability.
- **Service layer abstraction**: [`api/search_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/search_service.py) provides a singleton interface that normalizes requests before forwarding to the SurrealDB backend via the API client.
- **Embedding pipeline**: Vector search relies on [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py) to generate query embeddings at runtime, compared against pre‑computed source and note vectors.
- **Context assembly**: The `Notebook.get_context` method aggregates full source texts and note contents into markdown format for downstream language model workflows.

## Frequently Asked Questions

### What database powers the search functionality in Open Notebook?

Open Notebook uses **SurrealDB** as its primary datastore, leveraging native database functions (`fn::text_search` and `fn::vector_search`) to perform both full‑text and vector operations without external search engines. This tight integration simplifies deployment while maintaining ACID compliance for notebook data.

### When does the system fallback from full‑text to vector search?

The system triggers the fallback when SurrealDB raises a *position overflow* exception during full‑text indexing, which typically occurs with very large or multi‑byte text chunks. The `text_search` coroutine catches this specific error in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) and transparently redirects the query to the `vector_search` path.

### How are embeddings generated for vector search?

Embeddings are generated at query time using the `generate_embedding` utility in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py). This function converts the user's search query into a vector representation that matches the pre‑computed embeddings stored alongside source and note records in SurrealDB, enabling cosine similarity comparisons.

### Can I retrieve the complete text content of all sources within a notebook?

Yes. The `Notebook.get_context` method asynchronously loads the `full_text` of every source and the `content` of every note associated with a notebook, then returns them as a formatted markdown string. This is particularly useful for feeding comprehensive notebook context into podcast generation pipelines or large language model prompts.