# How the Open Notebook Embedding Service Vectorizes Sources and Stores Them in SurrealDB

> Learn how the Open Notebook embedding service vectorizes sources using batched generation and mean-pooling, then stores them in SurrealDB via an async repository layer.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-24

---

**The Open Notebook embedding service converts source text into high-dimensional vectors using batched embedding generation with mean-pooling for long documents, then persists these vectors as `SourceEmbedding` records in SurrealDB through an async repository layer.**

Open Notebook, an open-source knowledge management system, implements a sophisticated embedding pipeline that transforms raw content into searchable vectors. When users ingest PDFs, web pages, or audio transcripts, the system processes this content through a LangGraph workflow that ultimately stores semantic representations in SurrealDB. Understanding how the embedding service vectorizes sources and stores them in SurrealDB reveals the architectural decisions that enable efficient semantic search across diverse content types.

## Source Chunking and Text Preparation

Before vectorization, raw source content undergoes preprocessing to handle token limits and optimize embedding quality. The system extracts text from documents and invokes `chunk_text` from [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py) to split lengthy content into manageable segments.

This preprocessing ensures that no single chunk exceeds the configured `CHUNK_SIZE` threshold. For short documents that fit within token limits, the system embeds the entire text as a single unit. However, when dealing with large sources like lengthy PDFs or extensive transcripts, the pipeline splits content into discrete chunks that can be processed independently while maintaining semantic coherence.

## Embedding Generation with Batch Processing

The core vectorization logic resides in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py), specifically within the `generate_embeddings` function. This utility orchestrates the transformation of text into dense vectors through an efficient batching strategy.

The service reads the configured batch size from the `OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE` environment variable, defaulting to **50** documents per batch. It then calls the embedding model's async `aembed` method on each batch, implementing exponential backoff through `EMBEDDING_MAX_RETRIES` and `EMBEDDING_RETRY_DELAY` parameters to handle transient API failures gracefully.

```python
from open_notebook.utils.embedding import generate_embeddings

texts = ["The quick brown fox jumps over the lazy dog."]
embeddings = await generate_embeddings(texts, command_id="cmd-123")

# embeddings[0] contains a list[float] ready for SurrealDB storage

```

### Handling Long Documents with Mean-Pooling

When source content exceeds the provider's token limit, the embedding service employs a sophisticated aggregation strategy. Rather than discarding overflow content or creating separate embeddings that fragment the document's meaning, the system processes individual chunks through `mean_pool_embeddings` (lines 55-108 in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py)).

This function normalizes each chunk embedding, calculates the arithmetic mean across all chunk vectors, then re-normalizes the result to produce a single unit-length vector representing the entire source. This approach preserves the semantic essence of lengthy documents while maintaining compatibility with SurrealDB's vector storage requirements.

### Retry Logic and Error Handling

The embedding layer implements robust fault tolerance through configurable retry mechanisms. If an embedding provider returns errors or rate limits, the service automatically retries failed batches using exponential backoff. This ensures that transient network issues or API throttling do not interrupt the ingestion pipeline, particularly critical when processing large imports through the background command system.

## Persisting Vectors to SurrealDB

Once the final embedding vector is computed, the system persists it through the repository layer defined in [`open_notebook/database/repository.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/database/repository.py). The process involves creating a `SourceEmbedding` object, a Pydantic model defined in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) that structures the data for SurrealDB storage.

The `SourceEmbedding` record contains:
- **vector**: A `list<float>` containing the high-dimensional embedding
- **source_id**: Reference to the parent source node
- **model_name**: Identifier of the embedding model used (e.g., "openai-text-embedding-ada-002")
- **created_at**: Timestamp of vectorization

```python
from open_notebook.database.repository import repository
from open_notebook.domain.notebook import SourceEmbedding

async def store_source_vector(source_id: str, vector: list[float]):
    embed_record = SourceEmbedding(
        source_id=source_id,
        vector=vector,
        model_name="openai-text-embedding-ada-002",
    )
    await repository.save_source_embedding(embed_record)

```

The repository layer executes an async `CREATE` query against SurrealDB, storing the vector in the `embedding` record linked to the original `Source` node. SurrealDB's native vector type enables efficient similarity search operations, allowing the system to perform fast semantic queries across all vectorized content.

## Background Processing Architecture

For production deployments handling large-scale imports, Open Notebook queues embedding work through the command system in [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py). The `RebuildEmbeddings` command (lines 34-1056) provides a background processing mechanism that:

1. Accepts lists of source IDs requiring vectorization
2. Logs progress and mismatches during processing
3. Retries failures automatically
4. Returns a `RebuildEmbeddingsOutput` with status and error details

This command integrates with the LangGraph source processing pipeline in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py), which triggers the embedding step automatically after content extraction completes. The asynchronous architecture ensures that UI responsiveness remains high while large documents undergo vectorization.

```python
from commands.embedding_commands import RebuildEmbeddingsInput, RebuildEmbeddings

result = await RebuildEmbeddings(
    input_data=RebuildEmbeddingsInput(source_ids=["src-42"]),
)

# Returns RebuildEmbeddingsOutput with processing status

```

## Summary

- **Text preprocessing** occurs in [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py), splitting content into token-compliant chunks before embedding.
- **Batch embedding generation** happens in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py) via `generate_embeddings`, which processes 50 documents per batch by default and implements exponential backoff for retries.
- **Mean-pooling** aggregates chunk embeddings for long documents into single unit-length vectors through `mean_pool_embeddings`.
- **Type-safe storage** uses the `SourceEmbedding` Pydantic model from [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) to structure data for SurrealDB.
- **Async persistence** flows through [`open_notebook/database/repository.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/database/repository.py), leveraging SurrealDB's native vector type for similarity search.
- **Background processing** handles large imports via `RebuildEmbeddings` in [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py), ensuring reliable vectorization without blocking user operations.

## Frequently Asked Questions

### How does Open Notebook handle documents that exceed the embedding model's token limit?

When documents exceed token limits, the system splits them into chunks using `chunk_text`, embeds each chunk separately, then applies mean-pooling through `mean_pool_embeddings` in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py). This function normalizes individual chunk embeddings, averages them, and re-normalizes the result to create a single representative vector that captures the semantic meaning of the entire document while maintaining unit length.

### What configuration options control the embedding service behavior?

The embedding service respects two key environment variables: `OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE` (defaulting to 50) controls how many texts are processed per API call, while `EMBEDDING_MAX_RETRIES` and `EMBEDDING_RETRY_DELAY` govern the exponential backoff strategy for handling rate limits and transient failures. These settings allow operators to tune the balance between throughput and API provider constraints.

### How is the embedding vector linked to the original source in SurrealDB?

The system creates a `SourceEmbedding` record containing the vector and metadata, including the `source_id` field that references the parent source node. When the repository layer executes the `CREATE` query in SurrealDB, this establishes a graph relationship between the source content and its vector representation, enabling joins between semantic search results and original document metadata.

### Can the embedding process run asynchronously without blocking the user interface?

Yes, Open Notebook implements background processing through the `RebuildEmbeddings` command in [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py). This command queues embedding jobs for processing outside the request-response cycle, logging progress and retrying failures automatically. The LangGraph source processing pipeline in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) triggers this background workflow, ensuring that users can continue working while large documents undergo vectorization.