How to Process Content from Files and URLs with content-core in Open Notebook

Open Notebook delegates all content extraction to the content-core library, using a LangGraph-based pipeline in open_notebook/graphs/source.py to convert raw files, URLs, and videos into searchable markdown text.

Open Notebook treats every incoming document—whether a local PDF, a remote web page, or a YouTube video—as raw content that must be transformed into embeddable text. The heavy lifting for this conversion is performed by the content-core library, which provides a uniform extract_content() API across dozens of media types. The orchestration happens inside a state machine defined in the source graph, which handles parsing, error detection, and optional speech-to-text transcription before persisting the results.

Where Content Extraction Happens

The entry point for ingestion lives in open_notebook/graphs/source.py. The graph is built with LangGraph and consists of three sequential nodes:

1. content_process – prepares the state and calls content_core.extract_content().
2. save_source – persists the extracted title, text, and metadata to the Source domain model.
3. transform_content – (optional) executes user-selected transformations on the text.

The content_process node is where the actual extraction occurs. It constructs a ProcessSourceState dictionary—containing the file path or URL plus configuration flags—and passes it to extract_content():

# open_notebook/graphs/source.py

from content_core import extract_content
from content_core.common import ProcessSourceState

async def content_process(state: SourceState) -> dict:
    content_state = state["content_state"]
    content_state["url_engine"] = "auto"
    content_state["document_engine"] = "auto"
    content_state["output_format"] = "markdown"

    # Optional: inject speech-to-text configuration

    try:
        model_manager = ModelManager()
        defaults = await model_manager.get_defaults()
        if defaults.default_speech_to_text_model:
            stt_model = await Model.get(defaults.default_speech_to_text_model)
            content_state["audio_provider"] = stt_model.provider
            content_state["audio_model"] = stt_model.name
    except Exception:
        pass

    # The actual extraction call

    processed_state = await extract_content(content_state)

    # Validate results

    if processed_state.title == "Error" and processed_state.content.startswith(
        "Failed to extract content:"
    ):
        raise ValueError("Could not extract content from this source")
    
    if not processed_state.content or not processed_state.content.strip():
        raise ValueError("Could not extract any text content from this source")

    return {"content_state": processed_state}

The await extract_content(content_state) line is the sole integration point with the library. Everything downstream—vector embedding, error handling, and transformation—operates on the processed_state object returned by content_core.

How the Data Flows

API Request to Graph Execution

Ingestion begins when the FastAPI endpoint in api/routers/sources.py receives a payload. The request is forwarded to SourcesService, which enqueues the source graph. The compiled workflow (source_graph = workflow.compile() defined around lines 184–200) receives an initial ProcessSourceState structure that mirrors the shape expected by content_core.

The extract_content() Call

Inside content_process, the content-core library automatically selects an extraction backend based on the input type:

• Document engine – for PDFs, Word documents, and other local files.
• URL engine – for web pages and remote resources.
• Audio/Video handler – for YouTube URLs, which can pull existing subtitles or stream audio to a configured STT provider.

The library returns a ProcessedState dataclass containing:

• title – the extracted document title or "Error" on soft failure.
• content – the plain-text body (Markdown by default).
• url / file_path – source identifiers for provenance.

Persistence and Transformation

The save_source node writes the cleaned text into the Source record (open_notebook/domain/notebook.py). If the request included embed=True, this node triggers source.vectorize() to create asynchronous vector embeddings. Finally, if the user requested any transformations (e.g., summarization), the transform_content node spawns the separate transformation graph to process the text.

Error Handling and Soft Failures

content-core never raises an exception for a missing file or unreachable URL. Instead, it returns a sentinel ProcessedState with title == "Error" and a descriptive content prefix. Open Notebook detects this pattern (lines 80–88 in source.py) and raises a ValueError, marking the job as failed and the source as retryable. This prevents incomplete or corrupted data from entering the notebook.

Configuring Extraction Engines

Default behavior is controlled by ContentSettings in open_notebook/domain/content_settings.py. These settings define fallback values for the document and URL engines used by content_process. Users can override selections—such as forcing pdfminer for PDF processing instead of the auto-selected engine—via the Settings UI, which persists values to the database model.

Speech-to-Text Integration

For video content lacking subtitles, Open Notebook can stream audio to an external STT provider. When a user configures a default speech-to-text model in the Settings panel, the source graph injects audio_provider and audio_model keys into the content_state dictionary (lines 62–71). The content-core library then hands these parameters to its internal audio extractor, enabling transcript generation during the extraction phase.

Code Examples

Directly Invoking the Source Graph

from open_notebook.graphs.source import source_graph
from content_core.common import ProcessSourceState

initial_state = {
    "content_state": ProcessSourceState(
        url="https://example.com/article.html",
        # file_path="/tmp/document.pdf"  # alternative to URL

    ),
    "apply_transformations": [],
    "source_id": "source:tmp-123",
    "notebook_ids": ["notebook:default"],
    "embed": True,
}

result = await source_graph.ainvoke(initial_state)
print(result["source"].full_text[:200])

Using the High-Level Service

from open_notebook.services import sources_service

new_source = sources_service.create_source(
    notebooks=["notebook:default"],
    url="https://arxiv.org/pdf/2301.01234.pdf",
    embed=True,
)

print(f"Created source {new_source.id} with title '{new_source.title}'")

Configuring Speech-to-Text for Video Processing

from open_notebook.ai.models import Model, ModelManager

model_manager = ModelManager()
defaults = await model_manager.get_defaults()

if defaults.default_speech_to_text_model:
    stt = await Model.get(defaults.default_speech_to_text_model)
    # source.py automatically uses these for audio extraction

    print(f"Using {stt.provider}/{stt.name} for transcription")

Summary

• Single entry point: The content_process node in open_notebook/graphs/source.py is the only place where content_core.extract_content() is invoked.
• Automatic engine selection: content-core chooses the appropriate parser for documents, URLs, or video based on the input type and configuration.
• Defensive error handling: Soft failures return title="Error", which Open Notebook converts to a ValueError to prevent storage of invalid data.
• Extensible via settings: Default engines and STT models are configurable through ContentSettings and injected into the extraction state.
• LangGraph orchestration: The entire pipeline—from download to embedding—is managed as a compiled LangGraph workflow.

Frequently Asked Questions

What file types does content-core support in Open Notebook?

The content-core library supports a wide variety of formats including PDFs, Word documents, plain text, HTML pages, and YouTube videos. According to the source graph implementation, it automatically selects a document engine for local files and a URL engine for web resources, returning everything as Markdown via the output_format parameter.

How does Open Notebook handle failed extractions?

Rather than crashing, content-core returns a ProcessedState with title == "Error" and a message starting with "Failed to extract content:". The content_process node in open_notebook/graphs/source.py detects this pattern and raises a ValueError, which marks the job as failed in the queue and allows the user to retry without corrupting the notebook.

Can I use a custom speech-to-text model for video processing?

Yes. If you configure a default speech-to-text model in the Settings UI, the source graph automatically injects audio_provider and audio_model into the content_state dictionary before calling extract_content(). This enables content-core to transcribe YouTube videos that lack subtitles using your chosen provider (e.g., OpenAI Whisper).

Where are the default extraction engines configured?

Default engine selections—such as default_content_processing_engine_doc—are stored in the ContentSettings domain model located at open_notebook/domain/content_settings.py. These values are read by the content_process node and merged into the ProcessSourceState before extraction begins.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}