How SurrealDB's Graph Database Schema Supports Vector Embeddings and Semantic Search in Open Notebook

Open Notebook stores content as graph records in SurrealDB, using a source_embedding table with vector fields and the similarity() function to enable native semantic search without external vector databases.

Open Notebook, the knowledge management system in the lfnovo/open-notebook repository, uses SurrealDB as its primary data store to manage content as interconnected graph records. The implementation leverages SurrealDB's built-in vector support to perform semantic search directly within the graph database schema. This architecture eliminates the need for separate embedding stores by treating chunks of text and their vector representations as linked nodes in the graph.

Graph Schema Design for Vector Storage

The database schema centers on two interconnected tables that bridge content and its semantic representation.

The Source and SourceEmbedding Tables

Every piece of content resides in the source table, which holds metadata and full text. The source_embedding table stores the actual vector data, with each row representing a text chunk and its embedding vector.

In open_notebook/domain/notebook.py, the SourceEmbedding class inherits from ObjectModel and explicitly declares table_name = "source_embedding"【source:open_notebook/domain/notebook.py#L304-L311】. This table structure includes:

• id: Unique identifier for the chunk
• source: Graph reference pointing to the parent source record
• embedding: The vector representation stored as SurrealDB vector type
• chunk_text: The original text segment
• chunk_index: Positional index within the source

The graph relationship allows traversal from source to embeddings and back, enabling complex queries that combine semantic similarity with graph filtering.

Vector Embedding Generation

When content enters the system, Open Notebook automatically chunks text and generates embeddings through an async pipeline.

Chunking and Embedding Creation

The vectorize method in open_notebook/domain/notebook.py handles the transformation from raw text to vector records【source:open_notebook/domain/notebook.py#L485-L495】. The process splits source.full_text into manageable chunks, calls open_notebook.utils.embedding.generate_embedding to create vectors, and inserts rows into source_embedding with references back to the parent source.

This fire-and-forget approach ensures that indexing happens asynchronously without blocking the main ingestion flow.

Semantic Search Implementation

SurrealDB's native vector functions enable similarity calculations directly in the database layer, eliminating network hops to external services.

SurrealQL Vector Queries

The database provides the similarity() function for computing cosine similarity between stored vectors and query vectors. A typical semantic search query looks like:

SELECT
    source,
    chunk_index,
    chunk_text,
    embedding,
    similarity(embedding, $query) AS score
FROM source_embedding
WHERE score > 0.2
ORDER BY score DESC
LIMIT 10;

This query returns the source reference, chunk metadata, and similarity score for the closest matches.

The vector_search Method

Open Notebook encapsulates this logic in the Source.vector_search method within open_notebook/domain/notebook.py【source:open_notebook/domain/notebook.py#L722-L751】. The method accepts a notebook ID, query vector, minimum score threshold (default 0.2), and result limit. It constructs parameterized SurrealQL to filter by similarity and aggregates results into a structured response containing source IDs and chunk text.

The implementation uses repo_query from open_notebook/database/repository.py to execute the SurrealQL against the database.

End-to-End Workflow

The system processes content from ingestion through search using a unified graph-based pipeline.

Ingestion to Search Pipeline

1. Ingestion: Source.save() persists the raw content record
2. Embedding: Source.vectorize() generates chunks and vectors, writing to source_embedding
3. Search: The API receives a query, generates an embedding via generate_embedding, and calls Source.vector_search() to retrieve semantically similar chunks

All stages operate within SurrealDB, maintaining ACID compliance and graph relationships throughout the vector search process.

Code Examples

Creating Embeddings for a Source

from open_notebook.domain.notebook import Source

async def embed_source(source: Source) -> None:
    # Triggers chunking and async embedding generation

    await source.vectorize()

Performing Semantic Search

from open_notebook.domain.notebook import Source
from open_notebook.utils.embedding import generate_embedding

async def semantic_search(notebook_id: str, query: str):
    # Convert query text to vector

    query_vec = await generate_embedding(query)
    
    # Execute graph-based vector search

    results = await Source.vector_search(
        notebook_id=notebook_id,
        query_vector=query_vec,
        min_score=0.2,
        limit=5,
    )
    return results  # Returns source, chunk_text, score, etc.

Raw SurrealQL Vector Query

SELECT
    source,
    chunk_text,
    similarity(embedding, $vec) AS score
FROM source_embedding
WHERE score > 0.2
ORDER BY score DESC
LIMIT 5;

Summary

• Graph-native storage: Open Notebook stores embeddings in the source_embedding table with graph references to source records, eliminating the need for separate vector databases
• Native similarity search: SurrealDB's built-in similarity() function computes cosine similarity directly on stored vectors
• Automatic chunking: The vectorize method handles text segmentation and embedding generation asynchronously
• Unified architecture: All data—graph relationships, text content, and vector embeddings—resides in a single SurrealDB instance

Frequently Asked Questions

How does SurrealDB's graph model improve vector search compared to standalone vector databases?

Standalone vector stores isolate embeddings from contextual metadata. SurrealDB's graph schema links each source_embedding record to its parent source via graph edges, allowing queries to traverse relationships while filtering by vector similarity. This enables complex queries like "find similar chunks from sources created in the last week" without joining disparate systems.

What embedding model does Open Notebook use for generating vectors?

The generate_embedding function in open_notebook/utils/embedding.py provides a unified interface for embedding generation, supporting mean-pooling and batching optimizations. The specific model configuration depends on the deployment, but the architecture abstracts model details behind this utility, allowing seamless swaps between providers while maintaining the same SurrealDB storage schema.

Why store embeddings as separate graph records instead of array fields on the source table?

Storing each chunk as a distinct row in source_embedding enables granular similarity scoring and retrieval. SurrealDB can index vector fields for efficient nearest-neighbor search, and the one-to-many relationship mirrors the logical structure of chunked documents. This design also supports updating specific chunks without rewriting the entire source record.

Can the similarity threshold be adjusted for different search contexts?

Yes. The vector_search method accepts a min_score parameter (default 0.2) that filters results in the SurrealQL WHERE clause. Applications can tune this threshold per query—increasing it for stricter relevance or decreasing it to capture broader semantic matches—while leveraging the same underlying graph schema and similarity() function.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}