How Open Notebook Implements Vector Search with SurrealDB Embeddings

Open Notebook implements vector search by generating query embeddings through a unified helper and executing SurrealDB's native fn::vector_search function to perform nearest-neighbor lookups against stored document embeddings.

The open-source project lfnovo/open-notebook stores dense vector embeddings for every source document and note, enabling semantic similarity search through SurrealDB's built-in vector capabilities. When users submit a query, the system transforms text into floating-point vectors and leverages SurrealDB's approximate nearest-neighbor indexing to retrieve relevant records. This approach combines intelligent text chunking with database-native vector operations to deliver fast, accurate semantic search results.

The Vector Search Pipeline

Open Notebook's vector search flows through three core components: embedding generation, SurrealQL execution, and result parsing. The process begins when a user submits a search query through the API, triggering the vector_search function in open_notebook/domain/notebook.py. This function coordinates between the embedding utility and the database repository to return similarity-ranked results.

Generating Query Embeddings

The generate_embedding function in open_notebook/utils/embedding.py serves as the unified entry point for all embedding operations. It handles both short texts and long documents that exceed the model's token limit.

Handling Short Texts

For inputs under the CHUNK_SIZE threshold, the system passes the text directly to the configured embedding model:

# open_notebook/utils/embedding.py (excerpt)

if text_tokens <= CHUNK_SIZE:
    embeddings = await generate_embeddings([text], command_id=command_id)
    return embeddings[0]

Chunking and Pooling for Long Documents

When texts exceed the token limit, the system splits the content into chunks, generates embeddings for each chunk, and computes a mean-pooled average to create a single representative vector:

# Long text → chunk, embed each chunk, then mean‑pool

chunks = chunk_text(text, content_type=content_type, file_path=file_path)
embeddings = await generate_embeddings(chunks, command_id=command_id)
pooled = await mean_pool_embeddings(embeddings)
return pooled

This ensures that lengthy source documents maintain semantic coherence in the vector space without truncating content.

Executing SurrealDB Vector Search

Once the query embedding is generated, Open Notebook invokes SurrealDB's native vector search capabilities through a specialized SurrealQL function.

The vector_search Function Signature

SurrealDB provides the built-in function fn::vector_search with the following signature:

fn::vector_search(
    embed VECTOR, 
    results INT, 
    source BOOL, 
    note BOOL, 
    minimum_score FLOAT
)

• embed: The query embedding produced by Open Notebook's embedding utility
• results: Maximum number of matches to return
• source / note: Booleans controlling which record types (source documents or notes) to include
• minimum_score: Cosine-similarity threshold (defaults to approximately 0.2)

Implementation in notebook.py

The domain logic resides in open_notebook/domain/notebook.py (lines 738-754), which validates inputs and executes the SurrealQL query:

async def vector_search(
    keyword: str,
    results: int,
    source: bool = True,
    note: bool = True,
    minimum_score=0.2,
):
    if not keyword:
        raise InvalidInputError("Search keyword cannot be empty")
    
    # Build embedding vector for the query

    from open_notebook.utils.embedding import generate_embedding
    embed = await generate_embedding(keyword)

    # Execute SurrealQL vector search

    search_results = await repo_query(
        """
        SELECT * FROM fn::vector_search($embed, $results, $source, $note, $minimum_score);
        """,
        {
            "embed": embed,
            "results": results,
            "source": source,
            "note": note,
            "minimum_score": minimum_score,
        },
    )
    return search_results

The repo_query helper in open_notebook/database/repository.py (lines 65-76) manages the SurrealDB connection, sends the query, and parses RecordID objects into plain Python dictionaries.

Fallback from Text Search to Vector Search

Open Notebook implements resilience by falling back to vector search when text search encounters limitations. If fn::text_search fails due to a highlight-position overflow (a SurrealDB limitation on very large strings), the system automatically redirects to the vector search path. This fallback logic appears in the text_search function (lines 11-20 of notebook.py), ensuring users always receive results even when exact text matching fails on large documents.

API Integration

The search endpoint in api/routers/search.py exposes vector search to clients through a FastAPI router:

@router.post("/search", response_model=SearchResponse)
async def search(req: SearchRequest):
    if req.type == "vector":
        results = await vector_search(
            keyword=req.query,
            results=req.limit,
            source=req.include_source,
            note=req.include_note,
            minimum_score=req.min_score,
        )
    else:
        results = await text_search(...)
    return SearchResponse(matches=results)

This architecture separates the HTTP layer from the domain logic, allowing the vector_search function to be reused across different contexts while maintaining clean API contracts.

Summary

• Open Notebook stores dense vector embeddings for all source documents and notes in SurrealDB's native vector columns.
• The generate_embedding utility in open_notebook/utils/embedding.py handles text chunking and mean-pooling for long documents.
• Vector search executes through SurrealDB's fn::vector_search function, which performs approximate nearest-neighbor lookups against stored embeddings.
• The vector_search function in open_notebook/domain/notebook.py orchestrates embedding generation and SurrealQL execution with configurable similarity thresholds.
• A fallback mechanism ensures vector search activates when text search fails on large documents due to SurrealDB highlight limitations.

Frequently Asked Questions

How does Open Notebook handle very long documents for vector search?

Open Notebook processes long documents through an intelligent chunking strategy. When generate_embedding detects text exceeding the CHUNK_SIZE token limit, it splits the content into manageable chunks using chunk_text(), generates embeddings for each chunk, and then computes a mean-pooled average via mean_pool_embeddings() to create a single representative vector. This approach preserves semantic information from the entire document without exceeding model token limits.

What is the default similarity threshold for SurrealDB vector searches?

The default minimum similarity score is approximately 0.2, which corresponds to a cosine similarity threshold in the fn::vector_search function. This parameter filters out low-relevance matches while maintaining recall for semantically related content. Users can override this threshold through the API by specifying the minimum_score parameter in their search requests.

Why does Open Notebook use SurrealDB's native vector search instead of a separate vector database?

Open Notebook leverages SurrealDB's built-in fn::vector_search and native vector column types to simplify architecture and reduce operational complexity. SurrealDB automatically builds approximate nearest-neighbor indexes on vector columns, eliminating the need for separate infrastructure like Pinecone or Weaviate. This integration allows the application to perform semantic similarity queries using standard SurrealQL SELECT statements alongside traditional relational operations, keeping the data layer unified and reducing latency from network hops between separate services.

What happens when text search fails in Open Notebook?

When fn::text_search encounters a highlight-position overflow error—a known SurrealDB limitation when processing very large strings—the system automatically falls back to vector search. This fallback logic, implemented in open_notebook/domain/notebook.py, ensures that users receive relevant results even when exact text matching cannot complete. The application catches the specific error condition and redirects the query through the vector_search path, maintaining service availability and result quality.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}