Design of the Search Service for Full-Text and Vector Search in Open Notebook

Open Notebook implements a unified search service that forwards queries from a thin client layer to a FastAPI backend, which executes either full-text or vector similarity searches against SurrealDB depending on the specified search type.

The search service for full-text and vector search in Open Notebook (lfnovo/open-notebook) provides a clean architectural abstraction that separates client-side convenience from backend execution complexity. This design allows the system to leverage SurrealDB's hybrid database capabilities—supporting both traditional text indexing and modern vector storage—while exposing a simple, consistent Python API to the rest of the application.

Three-Layer Service Architecture

The search implementation consists of three distinct layers that handle request serialization, transport, and execution.

HTTP Client Layer (api/client.py)

The foundation resides in api/client.py, where the APIClient class provides generic HTTP capabilities through the _make_request method. The APIClient.search method (lines 16-35) constructs a JSON payload containing query, type, limit, search_sources, search_notes, and minimum_score, then transmits a POST request to the /api/search endpoint.

Service Abstraction Layer (api/search_service.py)

Built atop the HTTP client, api/search_service.py exposes the primary SearchService.search interface (lines 18-38) consumed by the rest of the codebase. This thin wrapper immediately forwards parameters to api_client.search and normalizes the response by extracting and returning response.get("results", []), ensuring callers always receive a standardized list of result dictionaries regardless of backend implementation details.

FastAPI Backend Router (api/routers/search.py)

The FastAPI router receives the HTTP request at /api/search and orchestrates actual query execution. It parses the incoming JSON body, determines whether to invoke full-text indexing or vector similarity lookup based on the search_type parameter, and delegates to the repository layer (open_notebook/database/repository.py) to construct and execute the appropriate SurrealQL queries against the database.

Full-Text vs. Vector Search Implementation

The backend supports dual query modes that share common pagination and filtering logic but operate on different data structures.

Full-Text Search Mode

When search_type="text" (the default), the backend executes a textual match against the content fields of Sources and Notes. The router translates the query into SurrealQL using WHERE content CONTAINS <term> clauses or equivalent full-text index lookups, allowing traditional keyword-based retrieval across the knowledge base.

Vector Similarity Search

For search_type="vector", the system first generates query embeddings via the embedding service (open_notebook/utils/embedding.py). It then performs a k-NN similarity search using the SurrealQL expression VECTOR_DISTANCE(query_embedding, stored_embedding) < threshold, ordering results by computed distance. The minimum_score parameter filters results, returning only matches with similarity scores greater than or equal to the specified threshold.

End-to-End Workflow

The complete search execution follows a standardized pipeline:

1. Caller invokes SearchService.search(query, search_type, ...).
2. The service forwards the request to APIClient.search, which builds the JSON payload.
3. APIClient._make_request sends a POST to the /api/search endpoint.
4. The FastAPI router parses the JSON, selects the appropriate query engine (text or vector), and calls repository functions.
5. The repository issues SurrealQL against SurrealDB, retrieving matched Source and Note records with computed similarity scores (for vector mode).
6. Results are packaged into a {"results": [...]} payload and returned to the client.
7. SearchService.search extracts the list of result dictionaries and returns them to the caller.

Implementation Examples

The following examples demonstrate how to invoke the search service from client code:

# Example 1 – simple full-text search

from api.search_service import search_service

results = search_service.search(
    query="privacy‑preserving AI",
    search_type="text",      # default, can be omitted

    limit=20,
    search_sources=True,
    search_notes=True,
    minimum_score=0.0        # ignored for text mode

)

for r in results:
    print(r["type"], r["title"], r["score"])

# Example 2 – vector similarity search

results = search_service.search(
    query="How does embedding‑based retrieval work?",
    search_type="vector",
    limit=10,
    search_sources=True,
    search_notes=False,
    minimum_score=0.25      # only keep results with similarity ≥ 0.25

)

# Each result contains a `score` (higher = more similar)

for hit in results:
    print(f"{hit['type']} – {hit['title']} (score={hit['score']:.2f})")

Key Source Files

File	Role
[api/client.py](https://github.com/lfnovo/open-notebook/blob/main/api/client.py)	Implements the generic HTTP client and the search request construction (lines 16‑35).
[api/search_service.py](https://github.com/lfnovo/open-notebook/blob/main/api/search_service.py)	Provides the thin wrapper that forwards calls to the HTTP client (lines 18‑38).
api/routers/search.py	FastAPI router that receives POST requests, routes to text or vector engines, and queries SurrealDB.
open_notebook/database/repository.py	Contains SurrealQL queries that perform actual text or vector lookups against stored records.
open_notebook/utils/embedding.py	Generates query embeddings required for vector search execution.

Summary

• Three-layer design: Client HTTP transport, service abstraction, and FastAPI backend router separate concerns cleanly.
• Dual-mode support: The service handles both text (keyword) and vector (semantic) searches through the search_type parameter.
• SurrealDB integration: Full-text uses CONTAINS clauses while vector uses VECTOR_DISTANCE calculations against stored embeddings.
• Consistent interface: SearchService.search always returns a normalized list from response.get("results", []), insulating callers from backend changes.

Frequently Asked Questions

How does the search service decide between full-text and vector search?

The decision occurs in the FastAPI router (api/routers/search.py) based on the search_type parameter sent in the JSON payload. When search_type="text" (the default), the backend executes SurrealQL text matching; when search_type="vector", it generates embeddings and performs vector distance calculations. Both paths share the same endpoint and response format.

What database queries execute under the hood for vector searches?

The repository layer constructs SurrealQL queries using the VECTOR_DISTANCE(query_embedding, stored_embedding) function, comparing the query's embedding (generated via open_notebook/utils/embedding.py) against vector columns stored in SurrealDB. Results filter by the minimum_score threshold and order by calculated distance.

Can the search service query both Sources and Notes simultaneously?

Yes. The search_sources and search_notes boolean parameters (passed through APIClient.search) allow callers to scope searches to one or both content types. The backend applies these filters when querying the repository, ensuring only requested entity types return in the results list.

What happens if the backend search implementation changes?

The thin service architecture protects client code from breaking changes. Since SearchService.search is the primary interface and it normalizes responses to response.get("results", []), modifications to SurrealQL syntax, indexing strategies, or even database migrations within the repository layer do not require updates to calling code in the client.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}