How Vector Embedding Generation and Storage Enable Semantic Search in Open Notebook

Open Notebook generates dense vector embeddings from notebook content, stores them directly in SurrealDB records, and queries them using cosine similarity to retrieve semantically relevant context for LLM-powered answers.

The open-source Open Notebook project implements a complete semantic search pipeline by converting textual content from notebooks, sources, and notes into vector representations. These embeddings are generated on-demand using provider-agnostic AI models and persisted alongside your data in SurrealDB, enabling real-time similarity searches without external vector databases. Understanding this architecture reveals how the system delivers context-aware responses grounded in your actual knowledge base.

Embedding Generation Pipeline

The core embedding logic resides in open_notebook/utils/embedding.py, which provides a unified interface for converting text into normalized vectors. The system handles everything from single-line queries to multi-document batches with automatic retry logic.

Batch Processing and Retries

The pipeline respects environment variables for production tuning. OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE controls how many texts are sent to the provider simultaneously, while EMBEDDING_MAX_RETRIES ensures transient failures are handled automatically. The generate_embeddings() function iterates over input lists in batches and calls the model's aembed() async method, implementing exponential backoff for resilience.

Chunking and Mean-Pooling for Large Documents

When content exceeds the CHUNK_SIZE token limit, the system splits text using chunk_text() from open_notebook/utils/chunking.py before embedding. Each chunk is embedded separately, then combined through mean-pooling to create a single representative vector:

from open_notebook.utils.embedding import generate_embedding

# Automatically handles chunking and pooling for long texts

embedding = await generate_embedding(long_document_text)

The mean_pool_embeddings() function normalizes all chunk vectors, averages them, and re-normalizes the result to ensure a unit-length vector suitable for cosine similarity comparisons.

Provider-Agnostic Model Resolution

The actual embedding model is resolved through open_notebook.ai.models.model_manager.get_embedding_model(), which abstracts over supported providers including OpenAI, Anthropic, and Ollama. This allows the same codebase to work with local or cloud-based embedding models without configuration changes.

Storing Embeddings in SurrealDB

Open Notebook leverages SurrealDB's native vector type to store embeddings directly on domain objects, eliminating the need for separate vector databases.

Domain Object Integration

The Source and Note domain objects in open_notebook/domain/source.py and open_notebook/domain/note.py include an optional embedding: List[float] field. When content is saved, the API layer calls generate_embedding() and writes the resulting vector back to the record:

# From open_notebook/domain/source.py (simplified flow)

source.embedding = await generate_embedding(text, content_type=ctype, file_path=fp)
await source_repo.save(source)  # Persists vector in SurrealDB

Bulk Rebuilding

When switching embedding models or updating existing content, the system provides a rebuild endpoint in commands/embedding_commands.py:

curl -X POST http://localhost:5055/api/embedding/rebuild

This walks every source and note, regenerates embeddings using the current model configuration, and updates the stored vectors atomically.

Semantic Search Implementation

The search layer in api/search_service.py and api/routers/search.py transforms user questions into vector queries and retrieves contextually relevant content.

Query Vectorization

When a search request arrives, the system first embeds the user query using the same generate_embedding() utility used for storage. This ensures query and document vectors exist in the same semantic space using identical model parameters.

Cosine Similarity Queries

The Search service constructs SurrealQL statements that compute cosine similarity between the query vector and stored embeddings:

SELECT *, vector::cosine(embedding, $query_vec) AS similarity
FROM source
WHERE similarity > 0.7
ORDER BY similarity DESC
LIMIT 10;

This query executes directly against SurrealDB's native vector operators, returning records ranked by semantic relevance without external indexing services.

Context Retrieval for LLM Responses

Matched sources and notes are returned to the ask graph, which feeds the relevant content chunks to the LLM. This retrieval-augmented generation (RAG) pattern ensures answers are grounded in the most similar material from your knowledge base, with citations linked to the original sources.

Practical Implementation Examples

Generate Embeddings for Arbitrary Text

from open_notebook.utils.embedding import generate_embedding

text = "The quick brown fox jumps over the lazy dog."
embedding = await generate_embedding(text)  # Returns List[float] (unit-length)

Perform Semantic Search via API

import httpx

async def semantic_search(query: str):
    async with httpx.AsyncClient() as client:
        resp = await client.post(
            "http://localhost:5055/search",
            json={"query": query, "top_k": 5}
        )
        return resp.json()  # Contains sources with similarity scores

Rebuild All Embeddings

Useful after changing models or updating chunking strategies:

curl -X POST http://localhost:5055/api/embedding/rebuild

Summary

• Unified pipeline: All embeddings flow through open_notebook/utils/embedding.py, with batching, retries, and provider abstraction handled automatically.
• Intelligent chunking: Large documents are split and mean-pooled to maintain semantic coherence while respecting token limits.
• Native storage: SurrealDB stores vectors directly on Source and Note records, enabling single-database architecture.
• Cosine similarity: Vector search uses SurrealDB's native vector::cosine() operator for efficient similarity calculations.
• RAG integration: Search results feed directly into LLM context windows, providing grounded, citation-ready answers.

Frequently Asked Questions

How does Open Notebook handle documents that exceed the embedding model's token limit?

When text exceeds CHUNK_SIZE, the system calls chunk_text() from open_notebook/utils/chunking.py to split the document into overlapping segments. Each chunk is embedded separately, then mean_pool_embeddings() averages the normalized vectors and re-normalizes the result. This produces a single unit-length vector that represents the entire document's semantics while respecting model constraints.

Can I use local embedding models instead of OpenAI or Anthropic?

Yes. The model_manager.get_embedding_model() function in open_notebook/ai/models.py abstracts over all supported providers, including Ollama for local execution. Configure your preferred provider through environment variables, and the embedding pipeline will automatically use the specified local or remote model without code changes.

Why are embeddings stored directly in SurrealDB rather than a dedicated vector database?

Open Notebook leverages SurrealDB's native vector type to keep data and embeddings co-located. This eliminates network overhead and synchronization complexity between separate systems. The database supports vector similarity operators like vector::cosine() directly, providing fast semantic search without external dependencies while maintaining ACID compliance for your knowledge base.

What happens if an embedding generation request fails?

The generate_embeddings() function implements automatic retry logic based on EMBEDDING_MAX_RETRIES. Transient errors from the embedding provider trigger exponential backoff retries. If all retries exhaust, the error propagates to the caller, ensuring the system doesn't persist partial or corrupted embedding data.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}