How the Open Notebook Search Service Performs Vector and Full‑Text Search

Open Notebook routes every search request through a single FastAPI endpoint that delegates to SurrealDB either via a built‑in text index for full‑text search or via an embedded query vector for semantic similarity search, depending on the request type.

The lfnovo/open-notebook repository implements a unified search architecture that supports both classic keyword matching and modern semantic retrieval. Understanding how the search service performs vector and full‑text search helps developers extend the system or debug ranking issues. Both modes share the same POST /search entry point but rely on separate SurrealDB functions and distinct preprocessing pipelines.

Full‑Text Search Flow

When a SearchRequest arrives with type = "text", the FastAPI router in api/routers/search.py validates the payload and invokes text_search() from open_notebook/domain/notebook.py. This function constructs a SurrealQL statement that calls the built‑in database function fn::text_search, which tokenises the query and matches it against indexed titles, notes, and full‑text fields.

SurrealDB handles tokenisation, ranking, and result retrieval natively. The database returns matching source and note records directly, which the router wraps into a uniform SearchResponse before sending them to the client.

Vector Search Flow

For semantic queries (type = "vector"), the router first verifies that an embedding model is available through model_manager.get_embedding_model(). If configured, it calls vector_search() in open_notebook/domain/notebook.py, passing the user’s query string along with pagination and filtering flags.

Inside vector_search(), the query is sent to generate_embedding() in open_notebook/utils/embedding.py. If the text exceeds CHUNK_SIZE, it is split via chunk_text() and processed in batches of EMBEDDING_BATCH_SIZE (default 50). The resulting chunk embeddings are fused into a single unit‑length vector by mean_pool_embeddings().

This final embedding is passed to SurrealDB’s custom function fn::vector_search, which performs a similarity search against stored source_embedding and note_embedding vectors. Results are filtered by the minimum_score threshold and returned in similarity order.

Embedding Generation Pipeline

The utilities in open_notebook/utils/embedding.py centralise all embedding logic for the application. Long inputs are chunked in a content‑type aware manner, batched to respect provider rate limits, and pooled so that a single vector represents the entire query regardless of original length.

This same pipeline is reused by background jobs that embed sources and notes, ensuring consistency between indexed documents and search queries.

Code Examples

Using the Python Search Client

The SearchService wrapper in api/search_service.py forwards calls to the FastAPI endpoint through the shared api_client.

from api.search_service import search_service

# Full‑text search

results = search_service.search(
    query="machine learning ethics",
    search_type="text",
    limit=10,
    search_sources=True,
    search_notes=True,
)
print("Full‑text hits:", results)

# Vector (semantic) search

results = search_service.search(
    query="Explain transformer attention mechanisms",
    search_type="vector",
    limit=5,
    minimum_score=0.3,
)
print("Vector hits:", results)

Calling the FastAPI Endpoint Directly

You can also exercise the router with a TestClient or any HTTP client.

from fastapi.testclient import TestClient
from api.main import app

client = TestClient(app)

# Full‑text request

resp = client.post(
    "/search",
    json={
        "query": "privacy first AI",
        "type": "text",
        "limit": 15,
        "search_sources": True,
        "search_notes": False,
    },
)
print(resp.json())

# Vector request

resp = client.post(
    "/search",
    json={
        "query": "What are the advantages of vector search?",
        "type": "vector",
        "limit": 5,
        "minimum_score": 0.25,
    },
)
print(resp.json())

Invoking vector_search Directly

For custom workflows, import vector_search from the domain layer and await its results.

from open_notebook.domain.notebook import vector_search

async def find_similar_sources(query: str):
    matches = await vector_search(
        keyword=query,
        results=8,
        source=True,
        note=False,
        minimum_score=0.2,
    )
    return matches

Summary

• Full‑text search is handled by text_search() in open_notebook/domain/notebook.py, which invokes SurrealDB’s fn::text_search against a built‑in text index.
• Vector search is handled by vector_search() in the same file, embedding the query via generate_embedding() and querying SurrealDB with fn::vector_search.
• Embedding utilities in open_notebook/utils/embedding.py manage chunking, batching, and mean‑pooling so long queries still produce a single comparable vector.
• Routing and validation happen in api/routers/search.py, ensuring a uniform SearchResponse regardless of search mode.

Frequently Asked Questions

What is the difference between full‑text and vector search in Open Notebook?

Full‑text search relies on SurrealDB’s native text index to tokenise and rank keywords against source and note fields. Vector search converts the query into an embedding vector and performs a similarity search against pre‑computed source_embedding and note_embedding vectors stored in SurrealDB.

How does Open Notebook handle long queries during vector search?

If a query exceeds CHUNK_SIZE, generate_embedding() splits the text via chunk_text(), embeds the chunks in batches of EMBEDDING_BATCH_SIZE (default 50), and combines them with mean_pool_embeddings() into one unit‑length vector before searching.

Which file validates that an embedding model is configured before vector search?

The FastAPI router in api/routers/search.py checks model_manager.get_embedding_model() before dispatching a request to vector_search(). If no model is configured, the request fails at the API layer.

Can I adjust the minimum similarity threshold for vector results?

Yes. Both the API endpoint and the direct vector_search() function accept a minimum_score parameter. The SurrealDB fn::vector_search function uses this value to filter out low‑similarity matches before returning the result set.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}