Open Notebook Vector Search Implementation Details: SurrealDB Integration and Embedding Pipeline

Open Notebook implements vector search by embedding queries through a chunked mean-pooling pipeline and querying SurrealDB's built-in fn::vector_search function to perform cosine similarity matching against stored content embeddings.

Open Notebook stores all content—sources, notes, and notebooks—in SurrealDB and leverages the database's native vector-search capabilities for semantic retrieval. The architecture combines a Python-based embedding pipeline with SurrealDB's stored procedures to deliver efficient similarity search across potentially long-form content.

Embedding Pipeline: From Text to Vectors

The vector search process begins in open_notebook/utils/embedding.py, where the generate_embedding function transforms user queries into dense vectors suitable for similarity comparison.

Handling Long Queries with Chunking

For queries exceeding the configured CHUNK_SIZE token limit, the system implements a sophisticated chunking strategy:

• The text is split into smaller chunks using chunk_text() from open_notebook/utils/chunking.py
• Each chunk is embedded individually through the model manager (open_notebook/ai/models.py)
• The resulting vectors are aggregated using mean_pool_embeddings() (lines 55-89 in embedding.py) to produce a single normalized query vector

This approach ensures that even lengthy user inputs generate a properly normalized embedding without truncation or information loss. For short queries, the system bypasses chunking and calls generate_embeddings([text]) directly.

# From open_notebook/utils/embedding.py (lines 260-274)

embed = await generate_embedding(keyword)

SurrealDB Vector Search Execution

Once the embedding vector is generated, the search logic delegates to SurrealDB's built-in vector similarity function through a stored procedure call.

The vector_search Function

The vector_search function in open_notebook/domain/notebook.py (lines 38-64) executes the search by calling fn::vector_search via the repo_query wrapper:

# From open_notebook/domain/notebook.py

search_results = await repo_query(
    """
    SELECT * FROM fn::vector_search($embed, $results, $source, $note, $minimum_score);
    """,
    {
        "embed": embed,
        "results": results,
        "source": source,
        "note": note,
        "minimum_score": minimum_score,
    },
)

The repo_query function serves as a thin async wrapper around the SurrealDB driver, executing raw SurrealQL statements. The fn::vector_search stored procedure performs cosine similarity calculations against pre-computed embeddings stored in the database, returning the top-N most similar records filtered by the minimum_score threshold (defaulting to 0.2).

API Layer and Fallback Strategy

The search functionality is exposed through a FastAPI router that validates configuration and handles edge cases.

HTTP Endpoint Configuration

The vector search endpoint is defined in api/routers/search.py (lines 21-34), which validates that an embedding model is configured before processing requests:

# From api/routers/search.py

if search_request.type == "vector":
    results = await vector_search(
        keyword=search_request.query,
        results=search_request.limit,
        source=search_request.search_sources,
        note=search_request.search_notes,
        minimum_score=search_request.minimum_score,
    )

If no embedding model is configured, the API returns a 400 error with a clear message. The same endpoint supports traditional text search when the request type is set to "text".

Robust Fallback Handling

The implementation includes a resilience mechanism in the text_search function (open_notebook/domain/notebook.py, lines 11-18). When SurrealDB's text search encounters a "position overflow" error—a known limitation when highlighting large strings—the system automatically falls back to vector search:

except RuntimeError as e:
    if "position overflow" in str(e):
        logger.warning(...)
        return await vector_search(keyword, results, source, note)

This ensures users receive semantic results even when exact text matching fails due to database limitations.

Practical Implementation Examples

Direct Python Integration

To execute a vector search programmatically within the Open Notebook backend:

from open_notebook.domain.notebook import vector_search

# Search across both sources and notes

results = await vector_search(
    keyword="machine learning",
    results=10,
    source=True,
    note=True,
    minimum_score=0.3,
)

for r in results:
    print(r.id, r.title, r._score)  # _score contains cosine similarity

HTTP API Request

Clients can access vector search through the REST API:

curl -X POST http://localhost:5055/search \
  -H "Content-Type: application/json" \
  -d '{
        "type": "vector",
        "query": "artificial intelligence",
        "limit": 5,
        "search_sources": true,
        "search_notes": false,
        "minimum_score": 0.25
      }'

The response includes matching records with fields such as id, title, content, and the computed similarity score.

Summary

• Open Notebook stores all content embeddings in SurrealDB and uses the native fn::vector_search stored procedure for similarity matching.
• The embedding pipeline in open_notebook/utils/embedding.py handles long queries through chunking and mean-pooling to maintain vector quality.
• The vector_search function in open_notebook/domain/notebook.py serves as the primary interface between the application and SurrealDB's vector capabilities.
• The API layer automatically falls back from text search to vector search when encountering SurrealDB position overflow errors.
• Configuration parameters like minimum_score (default 0.2) and CHUNK_SIZE allow fine-tuning of search behavior.

Frequently Asked Questions

How does Open Notebook handle long queries in vector search?

According to the source code in open_notebook/utils/embedding.py, queries exceeding the CHUNK_SIZE token limit are split into smaller chunks using chunk_text(), embedded individually, and then aggregated via mean_pool_embeddings() to produce a single normalized vector. This ensures semantic integrity regardless of input length.

What database function performs the similarity search in SurrealDB?

Open Notebook utilizes SurrealDB's built-in stored procedure fn::vector_search, which performs cosine similarity calculations against pre-computed embeddings. The function is called through repo_query() in open_notebook/domain/notebook.py and accepts parameters for the embedding vector, result limits, source/note flags, and minimum similarity threshold.

What happens when text search fails in Open Notebook?

When the text_search function encounters a SurrealDB "position overflow" error during highlighting of large strings, it automatically catches the RuntimeError and falls back to vector_search() with the same parameters. This fallback mechanism is implemented in open_notebook/domain/notebook.py (lines 11-18).

How do I configure the minimum similarity threshold for vector searches?

The minimum_score parameter controls the similarity threshold and defaults to 0.2. When calling vector_search() directly or via the API endpoint at /search, specify this parameter to filter out low-similarity matches. The value represents the cosine similarity cutoff, with higher values returning more strictly matched results.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}