# How Vector Embedding Generation and Storage Enable Semantic Search in Open Notebook

> Discover how Open Notebook generates and stores vector embeddings in SurrealDB for powerful semantic search. Get semantically relevant context for LLM answers.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-15

---

**Open Notebook generates dense vector embeddings from notebook content, stores them directly in SurrealDB records, and queries them using cosine similarity to retrieve semantically relevant context for LLM-powered answers.**

The open-source Open Notebook project implements a complete **semantic search** pipeline by converting textual content from notebooks, sources, and notes into vector representations. These embeddings are generated on-demand using provider-agnostic AI models and persisted alongside your data in SurrealDB, enabling real-time similarity searches without external vector databases. Understanding this architecture reveals how the system delivers context-aware responses grounded in your actual knowledge base.

## Embedding Generation Pipeline

The core embedding logic resides in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py), which provides a unified interface for converting text into normalized vectors. The system handles everything from single-line queries to multi-document batches with automatic retry logic.

### Batch Processing and Retries

The pipeline respects environment variables for production tuning. `OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE` controls how many texts are sent to the provider simultaneously, while `EMBEDDING_MAX_RETRIES` ensures transient failures are handled automatically. The `generate_embeddings()` function iterates over input lists in batches and calls the model's `aembed()` async method, implementing exponential backoff for resilience.

### Chunking and Mean-Pooling for Large Documents

When content exceeds the `CHUNK_SIZE` token limit, the system splits text using `chunk_text()` from [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py) before embedding. Each chunk is embedded separately, then combined through **mean-pooling** to create a single representative vector:

```python
from open_notebook.utils.embedding import generate_embedding

# Automatically handles chunking and pooling for long texts

embedding = await generate_embedding(long_document_text)

```

The `mean_pool_embeddings()` function normalizes all chunk vectors, averages them, and re-normalizes the result to ensure a unit-length vector suitable for cosine similarity comparisons.

### Provider-Agnostic Model Resolution

The actual embedding model is resolved through `open_notebook.ai.models.model_manager.get_embedding_model()`, which abstracts over supported providers including OpenAI, Anthropic, and Ollama. This allows the same codebase to work with local or cloud-based embedding models without configuration changes.

## Storing Embeddings in SurrealDB

Open Notebook leverages SurrealDB's native **vector type** to store embeddings directly on domain objects, eliminating the need for separate vector databases.

### Domain Object Integration

The `Source` and `Note` domain objects in [`open_notebook/domain/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/source.py) and [`open_notebook/domain/note.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/note.py) include an optional `embedding: List[float]` field. When content is saved, the API layer calls `generate_embedding()` and writes the resulting vector back to the record:

```python

# From open_notebook/domain/source.py (simplified flow)

source.embedding = await generate_embedding(text, content_type=ctype, file_path=fp)
await source_repo.save(source)  # Persists vector in SurrealDB

```

### Bulk Rebuilding

When switching embedding models or updating existing content, the system provides a rebuild endpoint in [`commands/embedding_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/embedding_commands.py):

```bash
curl -X POST http://localhost:5055/api/embedding/rebuild

```

This walks every source and note, regenerates embeddings using the current model configuration, and updates the stored vectors atomically.

## Semantic Search Implementation

The search layer in [`api/search_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/search_service.py) and [`api/routers/search.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/search.py) transforms user questions into vector queries and retrieves contextually relevant content.

### Query Vectorization

When a search request arrives, the system first embeds the user query using the same `generate_embedding()` utility used for storage. This ensures query and document vectors exist in the same semantic space using identical model parameters.

### Cosine Similarity Queries

The Search service constructs SurrealQL statements that compute cosine similarity between the query vector and stored embeddings:

```sql
SELECT *, vector::cosine(embedding, $query_vec) AS similarity
FROM source
WHERE similarity > 0.7
ORDER BY similarity DESC
LIMIT 10;

```

This query executes directly against SurrealDB's native vector operators, returning records ranked by semantic relevance without external indexing services.

### Context Retrieval for LLM Responses

Matched sources and notes are returned to the `ask` graph, which feeds the relevant content chunks to the LLM. This **retrieval-augmented generation (RAG)** pattern ensures answers are grounded in the most similar material from your knowledge base, with citations linked to the original sources.

## Practical Implementation Examples

### Generate Embeddings for Arbitrary Text

```python
from open_notebook.utils.embedding import generate_embedding

text = "The quick brown fox jumps over the lazy dog."
embedding = await generate_embedding(text)  # Returns List[float] (unit-length)

```

### Perform Semantic Search via API

```python
import httpx

async def semantic_search(query: str):
    async with httpx.AsyncClient() as client:
        resp = await client.post(
            "http://localhost:5055/search",
            json={"query": query, "top_k": 5}
        )
        return resp.json()  # Contains sources with similarity scores

```

### Rebuild All Embeddings

Useful after changing models or updating chunking strategies:

```bash
curl -X POST http://localhost:5055/api/embedding/rebuild

```

## Summary

- **Unified pipeline**: All embeddings flow through [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py), with batching, retries, and provider abstraction handled automatically.
- **Intelligent chunking**: Large documents are split and mean-pooled to maintain semantic coherence while respecting token limits.
- **Native storage**: SurrealDB stores vectors directly on `Source` and `Note` records, enabling single-database architecture.
- **Cosine similarity**: Vector search uses SurrealDB's native `vector::cosine()` operator for efficient similarity calculations.
- **RAG integration**: Search results feed directly into LLM context windows, providing grounded, citation-ready answers.

## Frequently Asked Questions

### How does Open Notebook handle documents that exceed the embedding model's token limit?

When text exceeds `CHUNK_SIZE`, the system calls `chunk_text()` from [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py) to split the document into overlapping segments. Each chunk is embedded separately, then `mean_pool_embeddings()` averages the normalized vectors and re-normalizes the result. This produces a single unit-length vector that represents the entire document's semantics while respecting model constraints.

### Can I use local embedding models instead of OpenAI or Anthropic?

Yes. The `model_manager.get_embedding_model()` function in [`open_notebook/ai/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/models.py) abstracts over all supported providers, including Ollama for local execution. Configure your preferred provider through environment variables, and the embedding pipeline will automatically use the specified local or remote model without code changes.

### Why are embeddings stored directly in SurrealDB rather than a dedicated vector database?

Open Notebook leverages SurrealDB's native **vector type** to keep data and embeddings co-located. This eliminates network overhead and synchronization complexity between separate systems. The database supports vector similarity operators like `vector::cosine()` directly, providing fast semantic search without external dependencies while maintaining ACID compliance for your knowledge base.

### What happens if an embedding generation request fails?

The `generate_embeddings()` function implements automatic retry logic based on `EMBEDDING_MAX_RETRIES`. Transient errors from the embedding provider trigger exponential backoff retries. If all retries exhaust, the error propagates to the caller, ensuring the system doesn't persist partial or corrupted embedding data.