Full-Text and Vector Search in Open Notebooks: Implementation Deep Dive

Open Notebook implements hybrid search using SurrealDB's built‑in full‑text engine with automatic fallback to vector similarity search when literal keyword matching fails or semantic understanding is required.

The open‑source Open Notebook project (lfnovo/open‑notebook) stores notebooks, sources, and notes in SurrealDB, exposing two complementary search strategies orchestrated by a thin service layer. This article examines the technical implementation of full‑text and vector search in notebooks, tracing the execution path from the FastAPI endpoint down to the database functions.

Search Architecture Overview

Open Notebook provides full‑text search for literal keyword matches and vector (embedding) search for semantic similarity. These strategies operate within the open_notebook/domain/notebook.py module, where the text_search coroutine attempts the full‑text path first. When SurrealDB throws a position overflow error—common with large or multi‑byte text chunks—the system transparently falls back to the vector_search coroutine. Both methods raise DatabaseOperationError on unrecoverable failures, ensuring the API surface returns explicit errors rather than empty results.

The SearchService Abstraction

The entry point for all search operations is api/search_service.py, which defines a singleton search_service that forwards request parameters to the underlying API client.

from api.search_service import search_service

# Service forwards to api_client.search with normalized parameters

results = search_service.search(
    query="machine learning",
    search_type="text",
    limit=20
)

This thin wrapper ensures the front‑end communicates through a consistent interface while the heavy lifting occurs in the domain layer.

Full‑Text Search Implementation

The core full‑text logic resides in open_notebook/domain/notebook.py within the text_search coroutine. It executes SurrealDB's native fn::text_search function against the raw text of sources and notes.

search_results = await repo_query(
    """
    select *
    from fn::text_search($keyword, $results, $source, $note)
    """,
    {"keyword": keyword, "results": results,
     "source": source, "note": note},
)

This query searches across both sources and notes simultaneously, returning records that contain literal keyword matches. The function accepts boolean flags to toggle searching across sources, notes, or both, along with a minimum_score threshold to filter relevance.

Vector Search and Embedding Generation

When full‑text search is insufficient or explicitly requested, the vector_search coroutine in open_notebook/domain/notebook.py performs semantic similarity matching. The process involves two steps:

1. Embedding generation: The query text is converted to a vector embedding via open_notebook.utils.embedding.generate_embedding.
2. Similarity lookup: SurrealDB's fn::vector_search compares the query embedding against pre‑computed embeddings stored for sources and notes.

# Inside vector_search coroutine

query_embedding = await generate_embedding(query)
similar_results = await repo_query(
    """
    select *, vector::similarity($query_embedding, embedding) as score
    from source, note
    """,
    {"query_embedding": query_embedding}
)

This approach captures conceptual meaning rather than exact lexical matches, enabling discovery of relevant content even when keywords differ.

Automatic Fallback Strategy

The system implements resilient error handling for edge cases in full‑text processing. When text_search encounters a position overflow exception (SurrealDB error code relating to text indexing limits on large content), it catches the error and automatically triggers the vector_search path.

# Exception handling in notebook.py

except SurrealDBError as e:
    if "position overflow" in str(e):
        logger.warning("Full-text position overflow, falling back to vector search")
        return await vector_search(keyword, results, source, note)
    raise DatabaseOperationError(f"Search failed: {e}")

This transparent fallback ensures users receive meaningful results even when the text engine hits indexing limitations on very large documents.

Assembling Complete Context

For downstream workflows requiring the full content of a notebook—such as podcast generation or long‑form LLM prompts—the Notebook.get_context method in open_notebook/domain/notebook.py retrieves the complete text corpus.

nb = await Notebook.get("notebook:my-notebook-id")
full_context_md = await nb.get_context()  # Returns markdown with sources and notes

This method loads the full_text field from each source and the content field from each note, then formats them into structured markdown sections (e.g., ## Source: ...) suitable for language model consumption.

Practical Usage Examples

Performing a Full‑Text Search

from api.search_service import search_service

results = search_service.search(
    query="privacy first",
    search_type="text",  # Optional; default behavior

    limit=10,
    search_sources=True,
    search_notes=True,
    minimum_score=0.2,
)

# Returns list of dicts with source/note metadata

Forcing Vector Search for Semantic Queries

results = search_service.search(
    query="how to fine‑tune a transformer?",
    search_type="vector",  # Bypasses full‑text completely

    limit=10,
)

HTTP API Call

curl -X GET "http://localhost:5055/search?query=machine%20learning&limit=20" \
     -H "Accept: application/json"

Retrieving Notebook Context for LLMs

from open_notebook.domain.notebook import Notebook

async def get_notebook_corpus(notebook_id: str):
    nb = await Notebook.get(f"notebook:{notebook_id}")
    if nb:
        return await nb.get_context()  # Markdown formatted string

    return None

Summary

• Dual search strategy: Open Notebook combines SurrealDB's fn::text_search for literal matching with fn::vector_search for semantic similarity, implemented in open_notebook/domain/notebook.py.
• Automatic resilience: Full‑text search automatically falls back to vector search when encountering position overflow errors, ensuring continuous service availability.
• Service layer abstraction: api/search_service.py provides a singleton interface that normalizes requests before forwarding to the SurrealDB backend via the API client.
• Embedding pipeline: Vector search relies on open_notebook/utils/embedding.py to generate query embeddings at runtime, compared against pre‑computed source and note vectors.
• Context assembly: The Notebook.get_context method aggregates full source texts and note contents into markdown format for downstream language model workflows.

Frequently Asked Questions

What database powers the search functionality in Open Notebook?

Open Notebook uses SurrealDB as its primary datastore, leveraging native database functions (fn::text_search and fn::vector_search) to perform both full‑text and vector operations without external search engines. This tight integration simplifies deployment while maintaining ACID compliance for notebook data.

When does the system fallback from full‑text to vector search?

The system triggers the fallback when SurrealDB raises a position overflow exception during full‑text indexing, which typically occurs with very large or multi‑byte text chunks. The text_search coroutine catches this specific error in open_notebook/domain/notebook.py and transparently redirects the query to the vector_search path.

How are embeddings generated for vector search?

Embeddings are generated at query time using the generate_embedding utility in open_notebook/utils/embedding.py. This function converts the user's search query into a vector representation that matches the pre‑computed embeddings stored alongside source and note records in SurrealDB, enabling cosine similarity comparisons.

Can I retrieve the complete text content of all sources within a notebook?

Yes. The Notebook.get_context method asynchronously loads the full_text of every source and the content of every note associated with a notebook, then returns them as a formatted markdown string. This is particularly useful for feeding comprehensive notebook context into podcast generation pipelines or large language model prompts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}