How Vector Embeddings Are Managed and Queried in SurrealDB for Semantic Search

Open Notebook generates dense vector embeddings for documents and queries, then executes native vector similarity searches using SurrealDB's built-in fn::vector_search function to perform semantic retrieval with cosine similarity scoring.

The lfnovo/open-notebook repository implements a complete semantic search pipeline that stores floating-point embeddings in SurrealDB's native vector columns and queries them using built-in nearest-neighbor functions. This architecture combines intelligent text chunking with SurrealDB's approximate nearest-neighbor indexing to deliver relevant results across source documents and notes.

Embedding Generation and Chunking Strategy

The embedding pipeline lives in open_notebook/utils/embedding.py, specifically within the generate_embedding function (lines 9-75). This utility handles variable-length inputs by automatically chunking long texts and mean-pooling the results into a single dense vector.

The function implements a two-path strategy:

1. Short text path: When input tokens are below the CHUNK_SIZE threshold, the text is embedded directly via the configured model
2. Long text path: Exceeding inputs are split into chunks, each embedded separately, then combined using mean-pooling

async def generate_embedding(
    text: str,
    content_type: Optional[ContentType] = None,
    file_path: Optional[str] = None,
    command_id: Optional[str] = None,
) -> List[float]:
    if not text or not text.strip():
        raise ValueError("Cannot generate embedding for empty text")
    text_tokens = token_count(text)

    # Short text → embed directly

    if text_tokens <= CHUNK_SIZE:
        embeddings = await generate_embeddings([text], command_id=command_id)
        return embeddings[0]

    # Long text → chunk, embed each chunk, then mean‑pool

    chunks = chunk_text(text, content_type=content_type, file_path=file_path)
    embeddings = await generate_embeddings(chunks, command_id=command_id)
    pooled = await mean_pool_embeddings(embeddings)
    return pooled

This approach ensures that every document—regardless of length—produces a single, fixed-dimension vector compatible with SurrealDB's vector column type.

SurrealDB Vector Search Implementation

The core search logic resides in open_notebook/domain/notebook.py within the vector_search function (lines 738-754). This function bridges the embedding generation layer with SurrealDB's native vector capabilities by invoking the fn::vector_search SurrealQL function.

SurrealDB automatically builds an approximate nearest-neighbour index on vector columns, enabling efficient similarity lookups without manual index management. The function signature exposed to the application is:

fn::vector_search( embed VECTOR, results INT, source BOOL, note BOOL, minimum_score FLOAT )

• embed: The query embedding produced by generate_embedding
• results: Maximum number of matches to return
• source / note: Booleans controlling which record types are included
• minimum_score: Cosine-similarity threshold (defaults to approximately 0.2)

The Python implementation validates inputs, generates the query embedding, and executes the SurrealQL statement:

async def vector_search(
    keyword: str,
    results: int,
    source: bool = True,
    note: bool = True,
    minimum_score=0.2,
):
    if not keyword:
        raise InvalidInputError("Search keyword cannot be empty")
    # ① Build embedding vector for the query

    from open_notebook.utils.embedding import generate_embedding
    embed = await generate_embedding(keyword)

    # ② Execute SurrealQL vector search

    search_results = await repo_query(
        """
        SELECT * FROM fn::vector_search($embed, $results, $source, $note, $minimum_score);
        """,
        {
            "embed": embed,
            "results": results,
            "source": source,
            "note": note,
            "minimum_score": minimum_score,
        },
    )
    return search_results

SurrealDB returns matching records enriched with computed similarity scores, which Open Notebook passes directly to API consumers.

Database Query Execution

The repo_query function in open_notebook/database/repository.py (lines 65-76) handles the actual database communication. This generic helper opens a SurrealDB connection, sends the parameterized SurrealQL, parses RecordID objects, and returns plain Python dictionaries to the domain layer.

This abstraction allows the vector search logic to remain agnostic of connection management while ensuring proper resource cleanup and result serialization.

API Layer and Fallback Strategy

The search endpoint in api/routers/search.py exposes vector search to clients via a dedicated API route. When the request type is set to "vector", the router delegates to the vector_search domain function after verifying that an embedding model is configured.

A critical resilience feature exists in the fallback logic: when fn::text_search fails due to a highlight‑position overflow (a SurrealDB limitation on very large strings), the system automatically falls back to vector search. This guarantees result delivery even when text-based highlighting encounters size constraints.

# api/routers/search.py  (excerpt)

@router.post("/search", response_model=SearchResponse)
async def search(req: SearchRequest):
    if req.type == "vector":
        # Ensure an embedding model is configured; otherwise raise 400

        results = await vector_search(
            keyword=req.query,
            results=req.limit,
            source=req.include_source,
            note=req.include_note,
            minimum_score=req.min_score,
        )
    else:
        results = await text_search(...)
    return SearchResponse(matches=results)

Summary

• Dense vector storage: Open Notebook stores floating-point embeddings in SurrealDB's native vector columns, which automatically maintain approximate nearest-neighbour indexes
• Intelligent chunking: The generate_embedding utility in open_notebook/utils/embedding.py handles long documents by chunking and mean-pooling to produce single-vector representations
• Native SurrealQL search: Vector similarity queries execute via fn::vector_search, accepting parameters for result limits, source/note filtering, and minimum cosine similarity thresholds
• Resilient architecture: The system falls back from text search to vector search when encountering SurrealDB's highlight-position overflow limitations on large strings
• Clean abstraction: The repo_query helper in open_notebook/database/repository.py manages connection lifecycle and result parsing, keeping domain logic database-agnostic

Frequently Asked Questions

How does SurrealDB store vector embeddings in Open Notebook?

SurrealDB stores each embedding in a column of type vector and automatically builds an approximate nearest-neighbour index. According to the lfnovo/open-notebook source code, embeddings are passed as floating-point arrays to the database, where SurrealDB handles the indexing and similarity computation natively.

What happens when a text search fails due to large document sizes?

When fn::text_search encounters a highlight-position overflow—a known SurrealDB limitation with very large strings—the code in open_notebook/domain/notebook.py (lines 11-20) automatically falls back to the vector search path. This ensures users always receive relevant results even when exact text matching is unavailable.

How does Open Notebook handle documents that exceed the embedding model's token limit?

The generate_embedding function in open_notebook/utils/embedding.py automatically detects oversized inputs using token_count, splits them into chunks via chunk_text, generates embeddings for each chunk, and combines them using mean_pool_embeddings. This produces a single vector representation regardless of original document length.

What parameters control the scope and quality of vector search results?

The fn::vector_search function accepts five parameters: embed (the query vector), results (maximum matches), source and note (booleans filtering which record types to search), and minimum_score (a cosine similarity threshold, typically defaulting to 0.2). These are passed from the API layer through the vector_search domain function in open_notebook/domain/notebook.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}