How Search Queries Are Routed Through Vector and Full-Text Search in SurrealDB

Open Notebook routes search queries through distinct SurrealDB stored functions—fn::text_search for keyword matching and fn::vector_search for cosine similarity—using a FastAPI router that delegates to domain helpers in open_notebook/domain/notebook.py.

In the open-source lfnovo/open-notebook project, the search architecture implements a clean separation between HTTP handling, business logic, and database operations. Understanding how search queries are routed through vector and full-text search in SurrealDB reveals a pipeline that moves from FastAPI endpoints through Python domain layers to SurrealQL stored procedures defined in migration files.

API Entry Point and Route Selection

The routing logic begins in api/routers/search.py, where the search_knowledge_base endpoint inspects the request type to determine which search strategy to employ.

# api/routers/search.py

@router.post("/search", response_model=SearchResponse)
async def search_knowledge_base(search_request: SearchRequest):
    if search_request.type == "vector":
        results = await vector_search(
            keyword=search_request.query,
            results=search_request.limit,
            source=search_request.search_sources,
            note=search_request.search_notes,
            minimum_score=search_request.minimum_score,
        )
    else:  # text search

        results = await text_search(
            keyword=search_request.query,
            results=search_request.limit,
            source=search_request.search_sources,
            note=search_request.search_notes,
        )
    return SearchResponse(results=results or [], total_count=len(results or []), search_type=search_request.type)

The router validates the incoming SearchRequest payload and branches based on the type field. Vector searches trigger embedding generation and similarity scoring, while full-text searches leverage SurrealDB's built-in text indexing capabilities.

Domain Layer Abstraction

Both search paths converge in open_notebook/domain/notebook.py, which provides async helper functions that sanitize inputs and construct the SurrealQL queries. These functions do not execute search logic directly; instead, they invoke stored database functions via the repo_query wrapper.

Full-Text Search Implementation

The text_search() function forwards keyword queries to the fn::text_search stored procedure:

# open_notebook/domain/notebook.py

async def text_search(keyword: str, results: int, source: bool = True, note: bool = True):
    search_results = await repo_query(
        """
        select *
        from fn::text_search($keyword, $results, $source, $note)
        """,
        {"keyword": keyword, "results": results, "source": source, "note": note},
    )
    return search_results

Vector Search Implementation

The vector_search() function requires an additional preprocessing step. Before querying SurrealDB, it must convert the query string into a numerical embedding using the configured model:

async def vector_search(
    keyword: str, results: int, source: bool = True, note: bool = True, minimum_score=0.2,
):
    from open_notebook.utils.embedding import generate_embedding
    embed = await generate_embedding(keyword)      # converts query → vector

    search_results = await repo_query(
        """
        SELECT * FROM fn::vector_search($embed, $results, $source, $note, $minimum_score);
        """,
        {
            "embed": embed,
            "results": results,
            "source": source,
            "note": note,
            "minimum_score": minimum_score,
        },
    )
    return search_results

The embedding generation logic resides in open_notebook/utils/embedding.py, which handles model interaction and text chunking when queries exceed the model's context window.

SurrealDB Stored Functions

The actual search execution occurs within SurrealDB itself, defined in open_notebook/database/migrations/4.surrealql. These stored functions perform the heavy lifting of indexing, scoring, and filtering.

Full-Text Search Function (fn::text_search)

This function uses SurrealDB's native search:: operators to match keywords across multiple fields including source titles, full-text content, insights, and note content:

DEFINE FUNCTION IF NOT EXISTS fn::text_search($query_text: string, $match_count: int, $sources:bool, $show_notes:bool) {
    let $source_title_search = IF $sources {(SELECT id, title,
        search::highlight('`', '`', 1) as content,
        id as parent_id,
        math::max(search::score(1)) AS relevance FROM source WHERE title @1@ $query_text GROUP BY id)} ELSE { [] };
    /* similar blocks for source_embedding, source_full_text, source_insight,
       note_title, note_content */
    let $final_results = array::union($source_results, $note_results );
    RETURN (select id, parent_id, title, math::max(relevance) as relevance
        from $final_results where id is not None
        group by id, parent_id, title ORDER BY relevance DESC LIMIT $match_count);
};

The function aggregates results from multiple sub-queries using search::highlight for result formatting and search::score for relevance ranking, then returns the top $match_count matches.

Vector Search Function (fn::vector_search)

For semantic search, this function computes cosine similarity between the query vector and stored embeddings across source_embedding, source_insight, and note records:

DEFINE FUNCTION IF NOT EXISTS fn::vector_search($query: array<float>, $match_count: int,
    $sources: bool, $show_notes: bool, $min_similarity: float) {
    let $source_embedding_search = IF $sources {(SELECT source.id as id, source.title as title,
        content, source.id as parent_id,
        vector::similarity::cosine(embedding, $query) as similarity
        FROM source_embedding
        WHERE vector::similarity::cosine(embedding, $query) >= $min_similarity
        ORDER BY similarity DESC LIMIT $match_count)} ELSE { [] };
    /* similar blocks for source_insight and note */
    let $all_results = array::union(
        array::union($source_embedding_search, $source_insight_search),
        $note_content_search
    );
    RETURN (select id, parent_id, title, math::max(similarity) as similarity,
        array::flatten(content) as matches
        from $all_results where id is not None
        group by id, parent_id, title ORDER BY similarity DESC LIMIT $match_count);
};

The function filters results based on the $min_similarity threshold before applying the final limit, ensuring only semantically relevant matches return to the client.

Complete Request Flow

The end-to-end routing of search queries follows this sequence:

1. Client submits a POST /api/search request with JSON payload specifying type as either "vector" or "text"
2. FastAPI router (api/routers/search.py) validates the request and dispatches to the appropriate domain helper
3. Vector path only: generate_embedding() in open_notebook/utils/embedding.py converts the query string into a float[] array
4. Domain helper (text_search or vector_search in open_notebook/domain/notebook.py) constructs the SurrealQL statement and calls repo_query()
5. SurrealDB executes the corresponding stored function (fn::text_search or fn::vector_search), performing the actual index scan, scoring, and filtering
6. Results flow back through the helper to the router, which wraps them in a SearchResponse and returns them to the client

Practical Usage Examples

You can trigger these search paths directly through the domain layer when working within the backend:

from open_notebook.domain.notebook import vector_search, text_search

# Full-text search across sources and notes

text_results = await text_search(
    keyword="machine learning",
    results=5,
    source=True,
    note=True
)

# Vector search with minimum similarity threshold

vector_results = await vector_search(
    keyword="machine learning",
    results=5,
    minimum_score=0.25,
    source=True,
    note=False
)

Alternatively, use the API client to access the endpoints remotely:

from open_notebook.api.client import APIClient

client = APIClient(base_url="http://localhost:5055")

# Full-text search

text_res = client.search(
    query="quantum computing",
    type="text",
    limit=10,
    search_sources=True,
    search_notes=True,
)

# Vector search (requires configured embedding model)

vector_res = client.search(
    query="quantum computing",
    type="vector",
    limit=10,
    minimum_score=0.3,
)

Summary

• FastAPI router in api/routers/search.py serves as the entry point, branching requests based on the type parameter to either vector_search() or text_search()
• Domain helpers in open_notebook/domain/notebook.py abstract the SurrealDB interaction, preparing parameters and invoking stored functions via repo_query()
• Embedding generation occurs only in the vector path through generate_embedding() in open_notebook/utils/embedding.py, transforming text queries into float arrays
• SurrealQL stored functions defined in open_notebook/database/migrations/4.surrealql execute the actual search logic: fn::text_search uses search::score and search::highlight, while fn::vector_search uses vector::similarity::cosine
• Cosine similarity thresholds in vector search and relevance scoring in full-text search provide configurable quality controls for result sets

Frequently Asked Questions

What determines whether a search uses vector or full-text retrieval?

The type field in the SearchRequest payload determines the routing path. When set to "vector", the system generates embeddings and queries fn::vector_search; any other value routes to text_search() and invokes fn::text_search instead.

How does the vector search generate embeddings for queries?

The vector_search() function imports generate_embedding() from open_notebook/utils/embedding.py, which uses the configured embedding model (via Esperanto) to convert the query string into a float array. This array becomes the $query parameter passed to the SurrealDB fn::vector_search function.

What SurrealDB functions perform the actual search operations?

The system relies on two stored functions defined in open_notebook/database/migrations/4.surrealql: fn::text_search for lexical matching using SurrealDB's full-text operators, and fn::vector_search for semantic similarity using vector::similarity::cosine calculations against stored embeddings.

Can I adjust the similarity threshold for vector searches?

Yes. The minimum_score parameter (default 0.2) in vector_search() maps to the $min_similarity parameter in the SurrealDB function. The stored procedure filters results where vector::similarity::cosine(embedding, $query) >= $min_similarity, allowing you to control result precision by adjusting this value in the API request or domain helper call.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}