# Embedding Service: How Open Notebook Vectorizes Content and Stores Embeddings in SurrealDB

> Learn how Open Notebook vectorizes content with SurrealDB for semantic search. Discover efficient storage and retrieval of text embeddings.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-07-05

---

**Open Notebook converts text from sources, notes, and insights into vector embeddings using content-aware chunking and stores them in SurrealDB for semantic search.**

The embedding service in `lfnovo/open-notebook` provides a robust, asynchronous pipeline that transforms raw text into searchable vectors. This system handles everything from intelligent content chunking to batch embedding generation and persistence in SurrealDB. Understanding how the embedding service vectorizes content and stores it in SurrealDB is essential for developers customizing semantic search capabilities or troubleshooting vector generation issues.

## Content Chunking and Type Detection

Before generating vectors, the system analyzes and splits text into optimal chunks based on content type.

### Automatic Content Type Detection

The **`detect_content_type`** function in [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py) examines the file extension and runs heuristics on the text to determine the appropriate splitting strategy. This ensures that HTML documents, Markdown files, and plain text are processed with specialized splitters that preserve semantic boundaries.

### Splitting Strategies

The chunking module applies different strategies based on the detected content type:

- **HTML** → Uses `HTMLHeaderTextSplitter` to preserve document structure
- **Markdown** → Uses `MarkdownHeaderTextSplitter` to maintain header hierarchy
- **Other formats** → Falls back to a plain-text recursive splitter

The system respects a **`CHUNK_SIZE`** of approximately 400 tokens. When chunks exceed this limit, they are re-split using the plain-text splitter. Any chunk smaller than **`MIN_CHUNK_SIZE`** (default 5 tokens) is discarded to avoid noise.

## Embedding Generation with Batching and Retries

Once chunked, text is converted to vectors using the configured embedding model via `model_manager.get_embedding_model()`.

### Batch Processing Configuration

The embedding utility in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py) processes text in batches controlled by the **`OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE`** environment variable (default 50). Each batch is retried up to three times to handle transient failures from the embedding API.

### Mean Pooling for Long Documents

For content that exceeds the token limit and is split into multiple chunks, the system computes embeddings for each chunk individually, then **mean-pools** the results. The vectors are unit-normed, averaged, and re-normed to produce a single representative embedding that retains the original length-scale of the document.

## Storing Embeddings in SurrealDB

The system persists vectors differently depending on the content type, using the repository pattern to abstract SurrealDB operations.

### Source Embeddings

The **`embed_source_command`** in [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py) handles bulk embedding storage for sources:

1. Deletes previous `source_embedding` records for the source using `repo_query`
2. Generates embeddings for every chunk
3. Bulk-inserts rows into the `source_embedding` table using `repo_insert`

Each row contains `source`, `order`, `content`, and the resulting `embedding` vector.

### Note and Insight Embeddings

- **`embed_note_command`**: Generates a single embedding (mean-pooled if necessary) for the note’s Markdown content and updates the record directly via `repo_query` using `UPDATE … SET embedding = …`
- **`embed_insight_command`**: Works identically to the note command but targets the `source_insight` table

### Repository Abstraction

All database operations use generic repository helpers (`repo_insert`, `repo_query`) defined in [`open_notebook/database/repository.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/database/repository.py). These wrappers handle SurrealDB’s async driver and ensure proper record ID formatting.

## API Interface and Async Processing

The public HTTP endpoint **`POST /embed`** in [`api/routers/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/embedding.py) provides the external interface for vectorization.

The endpoint validates that an embedding model is configured, then either queues an async command (`embed_source` or `embed_note`) or calls domain-model convenience methods (`Source.vectorize()`, `Note.save()`). It returns a `command_id` that callers can poll to track job status.

```bash
curl -X POST http://localhost:5055/embedding/embed \
  -H "Content-Type: application/json" \
  -d '{
        "item_id": "source:abcd1234",
        "item_type": "source",
        "async_processing": true
      }'

```

Response:

```json
{
  "success": true,
  "message": "Embedding queued for background processing",
  "item_id": "source:abcd1234",
  "item_type": "source",
  "command_id": "cmd-2024-07-05-001"
}

```

## Rebuilding the Embedding Corpus

When switching embedding models or reprocessing the entire corpus, the **`rebuild_embeddings_command`** in [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py) orchestrates bulk re-vectorization.

The command collects IDs of existing items (or all items depending on filters), then submits individual `embed_*` commands for each one. This enables the system to re-vectorize the entire corpus asynchronously after a model change.

```python
from commands.embedding_commands import rebuild_embeddings_command, RebuildEmbeddingsInput
import asyncio

async def rebuild_all():
    inp = RebuildEmbeddingsInput(mode="all", include_sources=True,
                                 include_notes=True, include_insights=True)
    result = await rebuild_embeddings_command(inp)
    print(f"Submitted {result.jobs_submitted} jobs (failed: {result.failed_submissions})")

asyncio.run(rebuild_all())

```

## Manual Vectorization Example

For custom workflows, you can manually orchestrate the chunking and embedding pipeline:

```python
from open_notebook.utils.embedding import generate_embedding
from open_notebook.utils.chunking import detect_content_type, chunk_text
from open_notebook.database.repository import repo_insert, repo_query
from open_notebook.domain.notebook import Source
import asyncio

async def vectorize_source(source_id: str):
    # Load the source record

    source = await Source.get(source_id)
    if not source or not source.full_text:
        raise ValueError("Source not found or empty")

    # Detect content type and chunk the text

    ctype = detect_content_type(source.full_text, source.asset.file_path if source.asset else None)
    chunks = chunk_text(source.full_text, content_type=ctype)

    # Generate embeddings for every chunk

    embeddings = await generate_embeddings(chunks)

    # Build rows for the source_embedding table

    rows = [
        {
            "source": source.id,
            "order": i,
            "content": chunk,
            "embedding": emb,
        }
        for i, (chunk, emb) in enumerate(zip(chunks, embeddings))
    ]

    # Delete any old embeddings and insert the new ones

    await repo_query("DELETE source_embedding WHERE source = $sid", {"sid": source.id})
    await repo_insert("source_embedding", rows)

# Run

asyncio.run(vectorize_source("source:abcd1234"))

```

## Summary

- **Content-aware chunking** uses `detect_content_type` in [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py) to apply HTML, Markdown, or plain-text splitters with a default chunk size of ~400 tokens and minimum size of 5 tokens.
- **Batch embedding generation** respects `OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE` (default 50) with automatic retries, implementing mean pooling for multi-chunk documents.
- **SurrealDB persistence** uses `repo_insert` and `repo_query` wrappers to store vectors in the `source_embedding`, `source_insight`, or note tables depending on content type.
- **Async command architecture** allows the `POST /embed` endpoint to queue background jobs via `embed_source_command`, `embed_note_command`, or `embed_insight_command`.
- **Bulk rebuilding** is available through `rebuild_embeddings_command` to re-vectorize the entire corpus after model changes.

## Frequently Asked Questions

### How does Open Notebook handle different file types when chunking text?

The system uses `detect_content_type` in [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py) to inspect file extensions and run heuristics on the text. Based on this detection, it selects `HTMLHeaderTextSplitter` for HTML content, `MarkdownHeaderTextSplitter` for Markdown files, or a recursive plain-text splitter for other formats. This ensures that document structure is preserved during the chunking process.

### What happens when text exceeds the embedding model's token limit?

When text exceeds the `CHUNK_SIZE` of approximately 400 tokens, the system splits it into multiple chunks. For long documents split this way, embeddings are generated for each chunk individually and then mean-pooled to create a single unit-normed vector that represents the entire document while retaining the original length-scale.

### How can I rebuild all embeddings after changing the embedding model?

Use the `rebuild_embeddings_command` in [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py). This command accepts a `RebuildEmbeddingsInput` configuration allowing you to specify `mode="all"` and boolean flags for `include_sources`, `include_notes`, and `include_insights`. It collects the relevant IDs and submits individual embedding jobs for each item, returning the count of jobs submitted and any failed submissions.

### What is the default batch size for embedding generation and can it be changed?

The default batch size is 50, controlled by the environment variable `OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE`. Each batch is retried up to three times if the embedding service returns transient errors. You can adjust this variable in your environment to optimize throughput based on your embedding provider's rate limits.