Embedding Service: How Open Notebook Vectorizes Content and Stores Embeddings in SurrealDB

Open Notebook converts text from sources, notes, and insights into vector embeddings using content-aware chunking and stores them in SurrealDB for semantic search.

The embedding service in lfnovo/open-notebook provides a robust, asynchronous pipeline that transforms raw text into searchable vectors. This system handles everything from intelligent content chunking to batch embedding generation and persistence in SurrealDB. Understanding how the embedding service vectorizes content and stores it in SurrealDB is essential for developers customizing semantic search capabilities or troubleshooting vector generation issues.

Content Chunking and Type Detection

Before generating vectors, the system analyzes and splits text into optimal chunks based on content type.

Automatic Content Type Detection

The detect_content_type function in open_notebook/utils/chunking.py examines the file extension and runs heuristics on the text to determine the appropriate splitting strategy. This ensures that HTML documents, Markdown files, and plain text are processed with specialized splitters that preserve semantic boundaries.

Splitting Strategies

The chunking module applies different strategies based on the detected content type:

• HTML → Uses HTMLHeaderTextSplitter to preserve document structure
• Markdown → Uses MarkdownHeaderTextSplitter to maintain header hierarchy
• Other formats → Falls back to a plain-text recursive splitter

The system respects a CHUNK_SIZE of approximately 400 tokens. When chunks exceed this limit, they are re-split using the plain-text splitter. Any chunk smaller than MIN_CHUNK_SIZE (default 5 tokens) is discarded to avoid noise.

Embedding Generation with Batching and Retries

Once chunked, text is converted to vectors using the configured embedding model via model_manager.get_embedding_model().

Batch Processing Configuration

The embedding utility in open_notebook/utils/embedding.py processes text in batches controlled by the OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE environment variable (default 50). Each batch is retried up to three times to handle transient failures from the embedding API.

Mean Pooling for Long Documents

For content that exceeds the token limit and is split into multiple chunks, the system computes embeddings for each chunk individually, then mean-pools the results. The vectors are unit-normed, averaged, and re-normed to produce a single representative embedding that retains the original length-scale of the document.

Storing Embeddings in SurrealDB

The system persists vectors differently depending on the content type, using the repository pattern to abstract SurrealDB operations.

Source Embeddings

The embed_source_command in commands/embedding_commands.py handles bulk embedding storage for sources:

1. Deletes previous source_embedding records for the source using repo_query
2. Generates embeddings for every chunk
3. Bulk-inserts rows into the source_embedding table using repo_insert

Each row contains source, order, content, and the resulting embedding vector.

Note and Insight Embeddings

• embed_note_command: Generates a single embedding (mean-pooled if necessary) for the note’s Markdown content and updates the record directly via repo_query using UPDATE … SET embedding = …
• embed_insight_command: Works identically to the note command but targets the source_insight table

Repository Abstraction

All database operations use generic repository helpers (repo_insert, repo_query) defined in open_notebook/database/repository.py. These wrappers handle SurrealDB’s async driver and ensure proper record ID formatting.

API Interface and Async Processing

The public HTTP endpoint POST /embed in api/routers/embedding.py provides the external interface for vectorization.

The endpoint validates that an embedding model is configured, then either queues an async command (embed_source or embed_note) or calls domain-model convenience methods (Source.vectorize(), Note.save()). It returns a command_id that callers can poll to track job status.

curl -X POST http://localhost:5055/embedding/embed \
  -H "Content-Type: application/json" \
  -d '{
        "item_id": "source:abcd1234",
        "item_type": "source",
        "async_processing": true
      }'

Response:

{
  "success": true,
  "message": "Embedding queued for background processing",
  "item_id": "source:abcd1234",
  "item_type": "source",
  "command_id": "cmd-2024-07-05-001"
}

Rebuilding the Embedding Corpus

When switching embedding models or reprocessing the entire corpus, the rebuild_embeddings_command in commands/embedding_commands.py orchestrates bulk re-vectorization.

The command collects IDs of existing items (or all items depending on filters), then submits individual embed_* commands for each one. This enables the system to re-vectorize the entire corpus asynchronously after a model change.

from commands.embedding_commands import rebuild_embeddings_command, RebuildEmbeddingsInput
import asyncio

async def rebuild_all():
    inp = RebuildEmbeddingsInput(mode="all", include_sources=True,
                                 include_notes=True, include_insights=True)
    result = await rebuild_embeddings_command(inp)
    print(f"Submitted {result.jobs_submitted} jobs (failed: {result.failed_submissions})")

asyncio.run(rebuild_all())

Manual Vectorization Example

For custom workflows, you can manually orchestrate the chunking and embedding pipeline:

from open_notebook.utils.embedding import generate_embedding
from open_notebook.utils.chunking import detect_content_type, chunk_text
from open_notebook.database.repository import repo_insert, repo_query
from open_notebook.domain.notebook import Source
import asyncio

async def vectorize_source(source_id: str):
    # Load the source record

    source = await Source.get(source_id)
    if not source or not source.full_text:
        raise ValueError("Source not found or empty")

    # Detect content type and chunk the text

    ctype = detect_content_type(source.full_text, source.asset.file_path if source.asset else None)
    chunks = chunk_text(source.full_text, content_type=ctype)

    # Generate embeddings for every chunk

    embeddings = await generate_embeddings(chunks)

    # Build rows for the source_embedding table

    rows = [
        {
            "source": source.id,
            "order": i,
            "content": chunk,
            "embedding": emb,
        }
        for i, (chunk, emb) in enumerate(zip(chunks, embeddings))
    ]

    # Delete any old embeddings and insert the new ones

    await repo_query("DELETE source_embedding WHERE source = $sid", {"sid": source.id})
    await repo_insert("source_embedding", rows)

# Run

asyncio.run(vectorize_source("source:abcd1234"))

Summary

• Content-aware chunking uses detect_content_type in open_notebook/utils/chunking.py to apply HTML, Markdown, or plain-text splitters with a default chunk size of ~400 tokens and minimum size of 5 tokens.
• Batch embedding generation respects OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE (default 50) with automatic retries, implementing mean pooling for multi-chunk documents.
• SurrealDB persistence uses repo_insert and repo_query wrappers to store vectors in the source_embedding, source_insight, or note tables depending on content type.
• Async command architecture allows the POST /embed endpoint to queue background jobs via embed_source_command, embed_note_command, or embed_insight_command.
• Bulk rebuilding is available through rebuild_embeddings_command to re-vectorize the entire corpus after model changes.

Frequently Asked Questions

How does Open Notebook handle different file types when chunking text?

The system uses detect_content_type in open_notebook/utils/chunking.py to inspect file extensions and run heuristics on the text. Based on this detection, it selects HTMLHeaderTextSplitter for HTML content, MarkdownHeaderTextSplitter for Markdown files, or a recursive plain-text splitter for other formats. This ensures that document structure is preserved during the chunking process.

What happens when text exceeds the embedding model's token limit?

When text exceeds the CHUNK_SIZE of approximately 400 tokens, the system splits it into multiple chunks. For long documents split this way, embeddings are generated for each chunk individually and then mean-pooled to create a single unit-normed vector that represents the entire document while retaining the original length-scale.

How can I rebuild all embeddings after changing the embedding model?

Use the rebuild_embeddings_command in commands/embedding_commands.py. This command accepts a RebuildEmbeddingsInput configuration allowing you to specify mode="all" and boolean flags for include_sources, include_notes, and include_insights. It collects the relevant IDs and submits individual embedding jobs for each item, returning the count of jobs submitted and any failed submissions.

What is the default batch size for embedding generation and can it be changed?

The default batch size is 50, controlled by the environment variable OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE. Each batch is retried up to three times if the embedding service returns transient errors. You can adjust this variable in your environment to optimize throughput based on your embedding provider's rate limits.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}