How Open Notebook Handles Vector Embeddings for Semantic Search in SurrealDB

Open Notebook stores and queries vector embeddings entirely within SurrealDB using a unified utility layer, asynchronous job commands, and native database vector similarity functions.

Open Notebook is an open-source knowledge management system that leverages SurrealDB as its primary data store. To enable semantic search across sources and notes, the application implements a complete embedding pipeline that keeps vectors inside the database rather than relying on external vector stores, creating a single source of truth for all semantic data.

Embedding Generation Architecture

The foundation of Open Notebook's semantic search resides in open_notebook/utils/embedding.py. This module provides model-agnostic utilities that normalize text of any length into dense vectors suitable for storage and comparison.

Handling Variable-Length Inputs

The generate_embedding() function processes text through an intelligent routing strategy. For short inputs, it calls the embedding model directly. For longer content, it automatically invokes chunk_text() from open_notebook/utils/chunking.py to split the text into manageable segments, processes these chunks in batches using generate_embeddings(), and consolidates the results via mean_pool_embeddings() to produce a single normalized vector regardless of original input size.

Configuration and Resilience

The embedding pipeline respects three environment variables defined at the top of open_notebook/utils/embedding.py:

• OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE controls the throughput for batch processing
• EMBEDDING_MAX_RETRIES defines the retry count for transient API failures
• EMBEDDING_RETRY_DELAY specifies the wait duration between retry attempts

This configuration ensures robust handling of rate limits and network interruptions during vectorization.

Persisting Embeddings via Asynchronous Commands

When a source document is created or updated, Open Notebook triggers the embed_source command defined in commands/embedding_commands.py. This command runs asynchronously through the job queue to prevent HTTP connection pool exhaustion during large document processing.

Idempotent Storage Pattern

The embed_source command implements an idempotent write pattern. It first loads the Source record and deletes any existing source_embedding rows for that specific source. It then detects the appropriate content type, chunks the full text, and calls generate_embeddings() to obtain vectors for each chunk. Finally, it bulk-inserts records into SurrealDB using repo_insert("source_embedding", records).

Each stored embedding record contains the source reference, chunk order, original content, and the vector embedding itself:

# Inside commands/embedding_commands.py

embeddings = await generate_embeddings(chunks, command_id=cmd_id)
records = [
    {
        "source": ensure_record_id(input_data.source_id),
        "order": idx,
        "content": chunk,
        "embedding": embedding,
    }
    for idx, (chunk, embedding) in enumerate(zip(chunks, embeddings))
]
await repo_insert("source_embedding", records)

Querying Vectors with SurrealDB Native Functions

Search requests entering through api/routers/search.py delegate to the vector_search function in open_notebook/domain/notebook.py. This domain function orchestrates the conversion of user queries into database-ready vector searches using SurrealDB's built-in capabilities.

Query Vectorization

The vector_search function first transforms the user's keyword string into an embedding by calling generate_embedding(), automatically handling chunking and mean pooling if the query exceeds length thresholds. This ensures the query vector is compatible with the stored embeddings regardless of input length.

Cosine Similarity Search

The function passes the query vector along with pagination and filtering parameters to SurrealDB's native fn::vector_search function:

SELECT * FROM fn::vector_search($embed, $results, $source, $note, $minimum_score);

SurrealDB computes cosine similarity between the query vector and all stored source_embedding vectors, returning the top-N matching sources (and optionally notes) whose similarity exceeds the specified minimum_score. This calculation occurs entirely within the database engine, eliminating network overhead for vector comparisons.

Practical Implementation Examples

Generating a Query Embedding

from open_notebook.utils.embedding import generate_embedding

# Convert a search term into a normalized embedding vector

query_vector = await generate_embedding("What is the impact of climate change on Arctic ecosystems?")

Triggering Source Vectorization

# Assume `source` is an instance of open_notebook.domain.notebook.Source

command_id = await source.vectorize()   # returns the job ID for tracking

Executing Semantic Search

from open_notebook.domain.notebook import vector_search

# Get the 10 most relevant sources for the query vector

results = await vector_search(
    keyword="climate change Arctic",
    results=10,
    source=True,
    note=False,
    minimum_score=0.25,
)

Summary

• Open Notebook stores all vector embeddings inside SurrealDB, maintaining a single source of truth for semantic data rather than using external vector databases.
• The generate_embedding() utility in open_notebook/utils/embedding.py handles automatic text chunking and mean pooling to normalize variable-length inputs into fixed-size vectors.
• Asynchronous embed_source commands in commands/embedding_commands.py prevent resource exhaustion while bulk-inserting chunk embeddings via repo_insert("source_embedding", records).
• Search queries are vectorized and passed to SurrealDB's native fn::vector_search function for cosine similarity calculation against stored source_embedding records.
• Environment variables OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE, EMBEDDING_MAX_RETRIES, and EMBEDDING_RETRY_DELAY provide operational control over the embedding pipeline resilience.

Frequently Asked Questions

Where does Open Notebook store vector embeddings?

Open Notebook stores vector embeddings directly inside SurrealDB as records in the source_embedding table. This design eliminates the need for external vector databases and allows the application to use SurrealDB's native fn::vector_search function for cosine similarity calculations against the stored vectors.

How does Open Notebook handle documents that exceed the embedding model's context window?

The generate_embedding() function in open_notebook/utils/embedding.py automatically detects long inputs and routes them through chunk_text() from open_notebook/utils/chunking.py. It embeds each chunk separately using generate_embeddings(), then applies mean_pool_embeddings() to combine the chunk vectors into a single normalized vector representing the entire document.

What happens if the embedding API fails during processing?

The embedding utilities implement automatic retry logic configured by the EMBEDDING_MAX_RETRIES and EMBEDDING_RETRY_DELAY environment variables. For source documents, the embed_source command runs asynchronously via the job queue, isolating failures and preventing HTTP connection pool exhaustion from blocking the main application thread.

How can I trigger semantic search in my own Open Notebook instance?

Import the vector_search function from open_notebook/domain/notebook.py and pass your query string along with parameters for result count, content type filters (source and note booleans), and minimum similarity score. The function automatically vectorizes your query and returns matching records from SurrealDB's source_embedding table using the database's built-in vector similarity function.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}