# Key Modules and Packages in the lfnovo/open-notebook Repository

> Explore key modules and packages in lfnovo/open-notebook. Discover its Next.js frontend, FastAPI backend, SurrealDB, API routing, AI provisioning, and LangGraph workflows.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: architecture
- Published: 2026-06-26

---

**The lfnovo/open-notebook repository implements a three-tier system architecture featuring a Next.js React frontend, FastAPI Python backend, and SurrealDB graph database, organized into specialized packages for API routing, AI provisioning, domain modeling, and LangGraph-based workflows.**

The open-notebook project is a privacy-first, multi-model research assistant designed to ingest PDFs, videos, and webpages while keeping data under user control. Understanding the key modules and packages in lfnovo/open-notebook reveals how the codebase maintains clean separation between the web API, business logic, and persistence layers. The Python backend is structured under the `open_notebook/` directory with specialized sub-packages for database operations, AI model management, and asynchronous workflow orchestration.

## Architecture Overview

Open Notebook divides functionality across three distinct tiers. The **frontend** delivers a React-based user interface built with Next.js, Zustand state management, and TanStack Query for data fetching. The **backend** consists of FastAPI endpoints that orchestrate domain logic through Python modules. The **database layer** uses SurrealDB as a graph database with asynchronous repository patterns for data access.

## Core Backend Packages

The Python implementation organizes code into logical sub-packages that handle specific concerns from HTTP routing to AI model provisioning.

### API Layer (`api/`)

The `api/` package serves as the FastAPI entry point and request routing layer. In [`api/main.py`](https://github.com/lfnovo/open-notebook/blob/main/api/main.py), the application initializes and registers all routers including CORS configuration. Individual router modules such as [`api/routers/sources.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/sources.py), [`api/routers/search.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/search.py), and [`api/routers/models.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/models.py) handle specific endpoint groups. The service layer, including files like [`api/sources_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/sources_service.py) and [`api/transformations_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/transformations_service.py), validates Pydantic schemas and forwards payloads to the domain layer.

### AI Provisioning (`open_notebook/ai/`)

The **AI provisioning system** manages LLM, embedding, speech-to-text, and text-to-speech models via the Esperanto library. The [`open_notebook/ai/provision.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/provision.py) file defines the `ModelManager` class and `Model` entities that handle model discovery, default selection, and credential management. This abstraction allows the system to switch between different AI providers without modifying business logic.

### Domain Models (`open_notebook/domain/`)

Domain entities in `open_notebook/domain/` use Pydantic to define data structures that map to SurrealDB records. The [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) module contains core classes like `Notebook`, `Source`, and `Asset`, along with vector search capabilities. Authentication and credential management reside in [`open_notebook/domain/credential.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/credential.py). These models enforce data validation and business rules before persistence.

### LangGraph Workflows (`open_notebook/graphs/`)

The **workflow engine** implements LangGraph state-machine graphs that drive content ingestion, chat, and transformations. The [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) module defines the source ingestion graph with nodes for content extraction, saving, and optional transformations. Additional workflow files include [`open_notebook/graphs/chat.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/chat.py) for conversational AI, [`open_notebook/graphs/ask.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/ask.py) for queries, and [`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py) for content processing pipelines.

### Database Access (`open_notebook/database/`)

Low-level SurrealDB access resides in `open_notebook/database/`. The [`open_notebook/database/repository.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/database/repository.py) module provides generic async CRUD helpers, while [`open_notebook/database/async_migrate.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/database/async_migrate.py) handles automatic schema migrations on startup. This layer abstracts the graph database interactions, allowing domain models to remain persistence-agnostic.

### Utilities (`open_notebook/utils/`)

Reusable helper functions for text processing and error handling live in `open_notebook/utils/`. Key modules include [`open_notebook/utils/text_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/text_utils.py) for text manipulation, [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py) for content segmentation, [`open_notebook/utils/encryption.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/encryption.py) for token security, and [`open_notebook/utils/error_classifier.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/error_classifier.py) for translating provider errors into user-friendly `OpenNotebookError` types.

### Podcast Generation (`open_notebook/podcasts/`)

The podcast feature uses [`open_notebook/podcasts/models.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/podcasts/models.py) to define `EpisodeProfile`, `SpeakerProfile`, and `PodcastEpisode` entities. This package supports the multi-speaker podcast generation feature with corresponding migration support.

### Command Interface (`commands/`)

Background job creation is encapsulated in the `commands/` directory. The [`commands/source_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/source_commands.py) module enqueues source processing tasks, while [`commands/podcast_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/podcast_commands.py) handles podcast generation jobs. These command objects provide a clean interface between the API layer and asynchronous task processing.

## Frontend and Testing Infrastructure

Beyond the Python backend, the repository includes complete frontend and testing packages.

### Next.js Frontend (`frontend/`)

The `frontend/` directory contains a React application built with Next.js, utilizing Zustand for state management and TanStack Query for server communication. Shadcn UI components provide the interface elements. Configuration files like [`frontend/next.config.ts`](https://github.com/lfnovo/open-notebook/blob/main/frontend/next.config.ts) handle the API proxy for development, while [`frontend/package.json`](https://github.com/lfnovo/open-notebook/blob/main/frontend/package.json) declares dependencies.

### Test Suite (`tests/`)

The `tests/` directory contains a comprehensive Pytest suite covering domain models, API endpoints, and graph workflows. Key files include [`tests/test_domain.py`](https://github.com/lfnovo/open-notebook/blob/main/tests/test_domain.py) for entity validation and [`tests/test_graphs.py`](https://github.com/lfnovo/open-notebook/blob/main/tests/test_graphs.py) for LangGraph workflow verification.

## Module Integration Flow

The packages interact through a defined orchestration pattern. The **API layer** receives HTTP requests and validates them with Pydantic schemas before forwarding to services. Services translate requests into **domain objects** and invoke **LangGraph workflows** from the graphs package. Each graph runs async nodes that call the **content-core** library for extraction, the **AI provision layer** for model selection, and the **database layer** for persistence. **Utility modules** supply common operations like token decryption and error classification throughout the pipeline.

## Practical Implementation Examples

### Executing the Source Ingestion Graph

To process content programmatically, invoke the source graph with a properly configured state:

```python
from open_notebook.graphs.source import source_graph
from open_notebook.domain.notebook import Source

# Assume an existing Source record with id='src-123'

state = {
    "content_state": {"url": "https://example.com/article", "document_engine": "auto"},
    "apply_transformations": [],
    "source_id": "src-123",
    "notebook_ids": ["nb-1"],
    "source": await Source.get("src-123"),
    "transformation": [],
    "embed": True,
}

result = await source_graph.ainvoke(state)
print(result["source"].full_text[:200])

```

This workflow executes content extraction, saving, and embedding generation as defined in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py).

### Selecting AI Models with ModelManager

Access the AI provisioning layer to retrieve configured models:

```python
from open_notebook.ai.provision import ModelManager

async def get_stt_model():
    manager = ModelManager()
    defaults = await manager.get_defaults()
    if defaults.default_speech_to_text_model:
        return await manager.get_model(defaults.default_speech_to_text_model)
    raise RuntimeError("No default STT model configured")

stt = await get_stt_model()
print(f"Using STT provider {stt.provider}, model {stt.name}")

```

The `ModelManager` class in [`open_notebook/ai/provision.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/provision.py) handles provider abstraction and credential resolution.

### Adding Transformations via Service Layer

Apply content transformations through the API service:

```python
from api.transformations_service import add_transformation_to_source

await add_transformation_to_source(
    source_id="src-123",
    transformation_name="Summarize",
)

```

This service layer function, defined in [`api/transformations_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/transformations_service.py), queues the transformation workflow while maintaining separation between API and domain logic.

### Querying Vector Similarity

Search for semantically similar sources using the domain model:

```python
from open_notebook.domain.notebook import Source

matches = await Source.search_vectors(
    query="quantum computing breakthroughs",
    limit=5,
)

for src in matches:
    print(src.title, src.id)

```

The `search_vectors` method in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) interfaces with SurrealDB's vector search capabilities.

## Summary

- **Three-tier architecture**: Next.js frontend, FastAPI backend, and SurrealDB database provide clear separation of concerns.
- **Specialized packages**: The `open_notebook/` directory contains focused modules for AI provisioning (`ai/`), domain models (`domain/`), LangGraph workflows (`graphs/`), database access (`database/`), and utilities (`utils/`).
- **API organization**: The `api/` package handles HTTP routing and service orchestration, while `commands/` manages background job creation.
- **Workflow automation**: LangGraph state machines in `open_notebook/graphs/` drive content ingestion, chat, and transformations.
- **Database abstraction**: Async repository patterns in `open_notebook/database/` provide CRUD operations and automatic migrations.
- **Frontend stack**: React/Next.js application in `frontend/` uses Zustand and TanStack Query for state and data management.

## Frequently Asked Questions

### What database does open-notebook use?

The system uses **SurrealDB** as its primary graph database. The `open_notebook/database/` package provides asynchronous access through [`open_notebook/database/repository.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/database/repository.py) for generic CRUD operations and [`open_notebook/database/async_migrate.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/database/async_migrate.py) for schema migrations. This graph database approach enables complex relationships between notebooks, sources, and notes while supporting vector similarity search for semantic queries.

### How does the AI provisioning system work?

The AI provisioning layer in [`open_notebook/ai/provision.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/provision.py) uses the **Esperanto library** to abstract multiple AI providers. The `ModelManager` class handles model discovery, default configuration storage, and credential resolution for LLMs, embedding models, and speech-to-text engines. This allows the system to switch providers without changing business logic, supporting privacy-focused deployments where users bring their own API keys.

### What is the role of LangGraph in open-notebook?

**LangGraph** powers the workflow engine located in `open_notebook/graphs/`. These state-machine graphs orchestrate complex multi-step processes like content ingestion ([`source.py`](https://github.com/lfnovo/open-notebook/blob/main/source.py)), which chains content extraction, saving, embedding generation, and optional transformations. The graph architecture ensures that each step (node) can be retried, logged, and executed asynchronously, providing robust error handling for long-running AI operations.

### How is the frontend structured?

The frontend is a **React/Next.js** application in the `frontend/` directory that communicates with the FastAPI backend via REST endpoints. It uses **Zustand** for client-side state management, **TanStack Query** for server state synchronization, and **Shadcn UI** for component styling. The [`frontend/next.config.ts`](https://github.com/lfnovo/open-notebook/blob/main/frontend/next.config.ts) configures the API proxy for development, while [`frontend/package.json`](https://github.com/lfnovo/open-notebook/blob/main/frontend/package.json) manages dependencies including the TypeScript toolchain and React framework.