How the Open Notebook Embedding Service Vectorizes Sources and Stores Them in SurrealDB

The Open Notebook embedding service converts source text into high-dimensional vectors using batched embedding generation with mean-pooling for long documents, then persists these vectors as SourceEmbedding records in SurrealDB through an async repository layer.

Open Notebook, an open-source knowledge management system, implements a sophisticated embedding pipeline that transforms raw content into searchable vectors. When users ingest PDFs, web pages, or audio transcripts, the system processes this content through a LangGraph workflow that ultimately stores semantic representations in SurrealDB. Understanding how the embedding service vectorizes sources and stores them in SurrealDB reveals the architectural decisions that enable efficient semantic search across diverse content types.

Source Chunking and Text Preparation

Before vectorization, raw source content undergoes preprocessing to handle token limits and optimize embedding quality. The system extracts text from documents and invokes chunk_text from open_notebook/utils/chunking.py to split lengthy content into manageable segments.

This preprocessing ensures that no single chunk exceeds the configured CHUNK_SIZE threshold. For short documents that fit within token limits, the system embeds the entire text as a single unit. However, when dealing with large sources like lengthy PDFs or extensive transcripts, the pipeline splits content into discrete chunks that can be processed independently while maintaining semantic coherence.

Embedding Generation with Batch Processing

The core vectorization logic resides in open_notebook/utils/embedding.py, specifically within the generate_embeddings function. This utility orchestrates the transformation of text into dense vectors through an efficient batching strategy.

The service reads the configured batch size from the OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE environment variable, defaulting to 50 documents per batch. It then calls the embedding model's async aembed method on each batch, implementing exponential backoff through EMBEDDING_MAX_RETRIES and EMBEDDING_RETRY_DELAY parameters to handle transient API failures gracefully.

from open_notebook.utils.embedding import generate_embeddings

texts = ["The quick brown fox jumps over the lazy dog."]
embeddings = await generate_embeddings(texts, command_id="cmd-123")

# embeddings[0] contains a list[float] ready for SurrealDB storage

Handling Long Documents with Mean-Pooling

When source content exceeds the provider's token limit, the embedding service employs a sophisticated aggregation strategy. Rather than discarding overflow content or creating separate embeddings that fragment the document's meaning, the system processes individual chunks through mean_pool_embeddings (lines 55-108 in open_notebook/utils/embedding.py).

This function normalizes each chunk embedding, calculates the arithmetic mean across all chunk vectors, then re-normalizes the result to produce a single unit-length vector representing the entire source. This approach preserves the semantic essence of lengthy documents while maintaining compatibility with SurrealDB's vector storage requirements.

Retry Logic and Error Handling

The embedding layer implements robust fault tolerance through configurable retry mechanisms. If an embedding provider returns errors or rate limits, the service automatically retries failed batches using exponential backoff. This ensures that transient network issues or API throttling do not interrupt the ingestion pipeline, particularly critical when processing large imports through the background command system.

Persisting Vectors to SurrealDB

Once the final embedding vector is computed, the system persists it through the repository layer defined in open_notebook/database/repository.py. The process involves creating a SourceEmbedding object, a Pydantic model defined in open_notebook/domain/notebook.py that structures the data for SurrealDB storage.

The SourceEmbedding record contains:

• vector: A list<float> containing the high-dimensional embedding
• source_id: Reference to the parent source node
• model_name: Identifier of the embedding model used (e.g., "openai-text-embedding-ada-002")
• created_at: Timestamp of vectorization

from open_notebook.database.repository import repository
from open_notebook.domain.notebook import SourceEmbedding

async def store_source_vector(source_id: str, vector: list[float]):
    embed_record = SourceEmbedding(
        source_id=source_id,
        vector=vector,
        model_name="openai-text-embedding-ada-002",
    )
    await repository.save_source_embedding(embed_record)

The repository layer executes an async CREATE query against SurrealDB, storing the vector in the embedding record linked to the original Source node. SurrealDB's native vector type enables efficient similarity search operations, allowing the system to perform fast semantic queries across all vectorized content.

Background Processing Architecture

For production deployments handling large-scale imports, Open Notebook queues embedding work through the command system in commands/embedding_commands.py. The RebuildEmbeddings command (lines 34-1056) provides a background processing mechanism that:

1. Accepts lists of source IDs requiring vectorization
2. Logs progress and mismatches during processing
3. Retries failures automatically
4. Returns a RebuildEmbeddingsOutput with status and error details

This command integrates with the LangGraph source processing pipeline in open_notebook/graphs/source.py, which triggers the embedding step automatically after content extraction completes. The asynchronous architecture ensures that UI responsiveness remains high while large documents undergo vectorization.

from commands.embedding_commands import RebuildEmbeddingsInput, RebuildEmbeddings

result = await RebuildEmbeddings(
    input_data=RebuildEmbeddingsInput(source_ids=["src-42"]),
)

# Returns RebuildEmbeddingsOutput with processing status

Summary

• Text preprocessing occurs in open_notebook/utils/chunking.py, splitting content into token-compliant chunks before embedding.
• Batch embedding generation happens in open_notebook/utils/embedding.py via generate_embeddings, which processes 50 documents per batch by default and implements exponential backoff for retries.
• Mean-pooling aggregates chunk embeddings for long documents into single unit-length vectors through mean_pool_embeddings.
• Type-safe storage uses the SourceEmbedding Pydantic model from open_notebook/domain/notebook.py to structure data for SurrealDB.
• Async persistence flows through open_notebook/database/repository.py, leveraging SurrealDB's native vector type for similarity search.
• Background processing handles large imports via RebuildEmbeddings in commands/embedding_commands.py, ensuring reliable vectorization without blocking user operations.

Frequently Asked Questions

How does Open Notebook handle documents that exceed the embedding model's token limit?

When documents exceed token limits, the system splits them into chunks using chunk_text, embeds each chunk separately, then applies mean-pooling through mean_pool_embeddings in open_notebook/utils/embedding.py. This function normalizes individual chunk embeddings, averages them, and re-normalizes the result to create a single representative vector that captures the semantic meaning of the entire document while maintaining unit length.

What configuration options control the embedding service behavior?

The embedding service respects two key environment variables: OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE (defaulting to 50) controls how many texts are processed per API call, while EMBEDDING_MAX_RETRIES and EMBEDDING_RETRY_DELAY govern the exponential backoff strategy for handling rate limits and transient failures. These settings allow operators to tune the balance between throughput and API provider constraints.

How is the embedding vector linked to the original source in SurrealDB?

The system creates a SourceEmbedding record containing the vector and metadata, including the source_id field that references the parent source node. When the repository layer executes the CREATE query in SurrealDB, this establishes a graph relationship between the source content and its vector representation, enabling joins between semantic search results and original document metadata.

Can the embedding process run asynchronously without blocking the user interface?

Yes, Open Notebook implements background processing through the RebuildEmbeddings command in commands/embedding_commands.py. This command queues embedding jobs for processing outside the request-response cycle, logging progress and retrying failures automatically. The LangGraph source processing pipeline in open_notebook/graphs/source.py triggers this background workflow, ensuring that users can continue working while large documents undergo vectorization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}