How Open Notebook Combines Full-Text and Vector Search for Hybrid Retrieval

Open Notebook's search service exposes a single /search endpoint that intelligently switches between full-text keyword matching and semantic vector retrieval based on the request's type parameter, while automatically falling back to vector search when the full-text engine fails.

Open Notebook's search architecture merges traditional keyword matching with modern semantic retrieval through a unified API surface. According to the lfnovo/open-notebook source code, the service routes queries between SurrealDB's native text functions and embedding-based similarity search, ensuring users receive relevant results even when individual search components encounter errors.

The Unified Search API

The entry point for all search operations resides in api/routers/search.py, which defines the /search endpoint. This router accepts a SearchRequest payload containing a type field that determines the search strategy.

When type equals "vector", the router invokes vector_search; otherwise, it defaults to text_search. This design provides a single API surface for both search modalities while maintaining clear separation between the underlying implementations.

How Full-Text Search Works

Full-text search leverages SurrealDB's built-in text indexing for fast, exact keyword matching. In open_notebook/domain/notebook.py, the text_search method executes SurrealDB's fn::text_search function against stored document chunks.

This approach excels for precise term matching and returns raw database rows quickly. The function searches across indexed content and returns matches containing the exact query terms, making it ideal for keyword-heavy queries where semantic understanding is unnecessary.

How Vector Search Works

Vector search enables semantic retrieval by comparing query embeddings against stored chunk embeddings. When the router calls vector_search (also in open_notebook/domain/notebook.py), the system first generates a query embedding via open_notebook.utils.embedding.generate_embedding.

The function then executes SurrealDB's fn::vector_search to find the most similar vectors in the database. This method captures conceptual meaning rather than exact lexical matches, making it suitable for queries like "how do transformer models understand language" where keyword matching might fail.

Automatic Fallback Mechanism

The search service implements graceful degradation through an automatic fallback mechanism inside text_search. If SurrealDB's fn::text_search raises a "position overflow" error—a known edge case in the database's full-text engine—the service logs a warning and immediately invokes vector_search instead.

This ensures users always receive results even when the primary search engine fails. The response maintains the original search_type value of "text" in the payload, though the actual results derive from the vector engine, providing seamless error recovery without client-side handling.

Implementation Examples

Text Search (Default)

The following example demonstrates a standard keyword search against sources and notes:

import requests

payload = {
    "query": "attention mechanism",
    "type": "text",      # can be omitted – "text" is the default

    "limit": 20,
    "search_sources": True,
    "search_notes": True,
}
resp = requests.post("http://localhost:5055/search", json=payload)
print(resp.json())

Vector Search

For semantic queries requiring conceptual understanding, specify "vector" as the search type:

payload = {
    "query": "how do transformer models understand language",
    "type": "vector",
    "limit": 10,
    "search_sources": True,
    "search_notes": True,
    "minimum_score": 0.3,
}
resp = requests.post("http://localhost:5055/search", json=payload)
print(resp.json())

Response Structure

Both search modes return a SearchResponse object (defined in api/models.py) containing:

• total_count: The number of matching results
• search_type: The actual search mode used ("text" or "vector")
• Results array with matched chunks

When the automatic fallback triggers, the search_type remains "text" (reflecting the original request) despite the results coming from the vector engine.

Summary

• Open Notebook exposes a single /search endpoint in api/routers/search.py that handles both search modalities through the type parameter
• Full-text search uses SurrealDB's fn::text_search for fast keyword matching via text_search in open_notebook/domain/notebook.py
• Vector search generates embeddings via generate_embedding and uses SurrealDB's fn::vector_search for semantic similarity via vector_search in the same file
• The system automatically falls back to vector search when full-text search encounters "position overflow" errors, ensuring continuous availability
• The search_type field in responses indicates which engine actually processed the query, even when fallback occurs

Frequently Asked Questions

How does Open Notebook decide between full-text and vector search?

Open Notebook checks the type field in the SearchRequest payload. If type equals "vector", the router calls vector_search; otherwise, it defaults to text_search. This logic resides in api/routers/search.py lines 17-45.

What happens if the full-text search fails?

The text_search function in open_notebook/domain/notebook.py (lines 111-126) includes a try-catch block that detects "position overflow" errors from SurrealDB. When this specific error occurs, the function automatically invokes vector_search and returns those results instead, ensuring the user never receives an empty response due to backend engine failures.

Which search method is faster?

Full-text search executes faster because it leverages SurrealDB's native text indexes without requiring embedding generation. Vector search incurs additional latency (typically 1-2 seconds) due to the embedding generation step via open_notebook.utils.embedding.generate_embedding before querying the vector database.

Can I force vector search even if I initially requested text search?

No explicit client-side parameter forces this switch; however, the automatic fallback mechanism in open_notebook/domain/notebook.py will silently upgrade your text search to vector search if the full-text engine raises a "position overflow" error. For guaranteed vector search, explicitly set "type": "vector" in your request payload.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}