# Design of the Search Service for Full-Text and Vector Search in Open Notebook

> Explore the unified search service for full-text and vector search in Open Notebook. Learn how it queries SurrealDB via a FastAPI backend for efficient results.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: architecture
- Published: 2026-06-25

---

**Open Notebook implements a unified search service that forwards queries from a thin client layer to a FastAPI backend, which executes either full-text or vector similarity searches against SurrealDB depending on the specified search type.**

The search service for full-text and vector search in Open Notebook (lfnovo/open-notebook) provides a clean architectural abstraction that separates client-side convenience from backend execution complexity. This design allows the system to leverage SurrealDB's hybrid database capabilities—supporting both traditional text indexing and modern vector storage—while exposing a simple, consistent Python API to the rest of the application.

## Three-Layer Service Architecture

The search implementation consists of three distinct layers that handle request serialization, transport, and execution.

### HTTP Client Layer (api/client.py)

The foundation resides in [`api/client.py`](https://github.com/lfnovo/open-notebook/blob/main/api/client.py), where the `APIClient` class provides generic HTTP capabilities through the `_make_request` method. The `APIClient.search` method (lines 16-35) constructs a JSON payload containing `query`, `type`, `limit`, `search_sources`, `search_notes`, and `minimum_score`, then transmits a **POST** request to the `/api/search` endpoint.

### Service Abstraction Layer (api/search_service.py)

Built atop the HTTP client, [`api/search_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/search_service.py) exposes the primary `SearchService.search` interface (lines 18-38) consumed by the rest of the codebase. This thin wrapper immediately forwards parameters to `api_client.search` and normalizes the response by extracting and returning `response.get("results", [])`, ensuring callers always receive a standardized list of result dictionaries regardless of backend implementation details.

### FastAPI Backend Router (api/routers/search.py)

The FastAPI router receives the HTTP request at `/api/search` and orchestrates actual query execution. It parses the incoming JSON body, determines whether to invoke **full-text** indexing or **vector similarity** lookup based on the `search_type` parameter, and delegates to the repository layer ([`open_notebook/database/repository.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/database/repository.py)) to construct and execute the appropriate SurrealQL queries against the database.

## Full-Text vs. Vector Search Implementation

The backend supports dual query modes that share common pagination and filtering logic but operate on different data structures.

### Full-Text Search Mode

When `search_type="text"` (the default), the backend executes a textual match against the `content` fields of **Sources** and **Notes**. The router translates the query into SurrealQL using `WHERE content CONTAINS <term>` clauses or equivalent full-text index lookups, allowing traditional keyword-based retrieval across the knowledge base.

### Vector Similarity Search

For `search_type="vector"`, the system first generates query embeddings via the embedding service ([`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py)). It then performs a **k-NN** similarity search using the SurrealQL expression `VECTOR_DISTANCE(query_embedding, stored_embedding) < threshold`, ordering results by computed distance. The `minimum_score` parameter filters results, returning only matches with similarity scores greater than or equal to the specified threshold.

## End-to-End Workflow

The complete search execution follows a standardized pipeline:

1. **Caller** invokes `SearchService.search(query, search_type, ...)`.
2. The service forwards the request to `APIClient.search`, which builds the JSON payload.
3. `APIClient._make_request` sends a **POST** to the `/api/search` endpoint.
4. The FastAPI router parses the JSON, selects the appropriate query engine (text or vector), and calls repository functions.
5. The repository issues SurrealQL against SurrealDB, retrieving matched **Source** and **Note** records with computed similarity scores (for vector mode).
6. Results are packaged into a `{"results": [...]}` payload and returned to the client.
7. `SearchService.search` extracts the list of result dictionaries and returns them to the caller.

## Implementation Examples

The following examples demonstrate how to invoke the search service from client code:

```python

# Example 1 – simple full-text search

from api.search_service import search_service

results = search_service.search(
    query="privacy‑preserving AI",
    search_type="text",      # default, can be omitted

    limit=20,
    search_sources=True,
    search_notes=True,
    minimum_score=0.0        # ignored for text mode

)

for r in results:
    print(r["type"], r["title"], r["score"])

```

```python

# Example 2 – vector similarity search

results = search_service.search(
    query="How does embedding‑based retrieval work?",
    search_type="vector",
    limit=10,
    search_sources=True,
    search_notes=False,
    minimum_score=0.25      # only keep results with similarity ≥ 0.25

)

# Each result contains a `score` (higher = more similar)

for hit in results:
    print(f"{hit['type']} – {hit['title']} (score={hit['score']:.2f})")

```

## Key Source Files

| File | Role |
|------|------|
| [[`api/client.py`](https://github.com/lfnovo/open-notebook/blob/main/api/client.py)](https://github.com/lfnovo/open-notebook/blob/main/api/client.py) | Implements the generic HTTP client and the `search` request construction (lines 16‑35). |
| [[`api/search_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/search_service.py)](https://github.com/lfnovo/open-notebook/blob/main/api/search_service.py) | Provides the thin wrapper that forwards calls to the HTTP client (lines 18‑38). |
| [`api/routers/search.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/search.py) | FastAPI router that receives POST requests, routes to text or vector engines, and queries SurrealDB. |
| [`open_notebook/database/repository.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/database/repository.py) | Contains SurrealQL queries that perform actual text or vector lookups against stored records. |
| [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py) | Generates query embeddings required for vector search execution. |

## Summary

- **Three-layer design**: Client HTTP transport, service abstraction, and FastAPI backend router separate concerns cleanly.
- **Dual-mode support**: The service handles both `text` (keyword) and `vector` (semantic) searches through the `search_type` parameter.
- **SurrealDB integration**: Full-text uses `CONTAINS` clauses while vector uses `VECTOR_DISTANCE` calculations against stored embeddings.
- **Consistent interface**: `SearchService.search` always returns a normalized list from `response.get("results", [])`, insulating callers from backend changes.

## Frequently Asked Questions

### How does the search service decide between full-text and vector search?

The decision occurs in the FastAPI router ([`api/routers/search.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/search.py)) based on the `search_type` parameter sent in the JSON payload. When `search_type="text"` (the default), the backend executes SurrealQL text matching; when `search_type="vector"`, it generates embeddings and performs vector distance calculations. Both paths share the same endpoint and response format.

### What database queries execute under the hood for vector searches?

The repository layer constructs SurrealQL queries using the `VECTOR_DISTANCE(query_embedding, stored_embedding)` function, comparing the query's embedding (generated via [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py)) against vector columns stored in SurrealDB. Results filter by the `minimum_score` threshold and order by calculated distance.

### Can the search service query both Sources and Notes simultaneously?

Yes. The `search_sources` and `search_notes` boolean parameters (passed through `APIClient.search`) allow callers to scope searches to one or both content types. The backend applies these filters when querying the repository, ensuring only requested entity types return in the results list.

### What happens if the backend search implementation changes?

The thin service architecture protects client code from breaking changes. Since `SearchService.search` is the primary interface and it normalizes responses to `response.get("results", [])`, modifications to SurrealQL syntax, indexing strategies, or even database migrations within the repository layer do not require updates to calling code in the client.