How content-core Extracts Content from PDFs, Videos, Audio, and URLs in Open Notebook

The content-core library processes raw files and URLs by inspecting the ProcessSourceState configuration, automatically selecting specialized engines like pdfminer for documents, ffmpeg with speech-to-text models for media, and readability-lxml for web pages.

The content-core library serves as the extraction engine for the Open Notebook project, handling the transformation of unstructured data into clean markdown. When users upload PDFs, audio files, YouTube links, or web URLs, the system constructs a ProcessSourceState object that dictates which parsing pipeline to execute. This architecture allows Open Notebook to support over 50 file formats through a unified asynchronous interface.

The ProcessSourceState Abstraction

At the heart of the extraction workflow is the ProcessSourceState object defined in content_core.common. This dictionary-like structure tells content-core which extraction engine to use and supplies optional model configurations for speech-to-text processing.

In open_notebook/graphs/source.py, Open Notebook prepares this state and calls the extraction function:

from content_core import extract_content
from content_core.common import ProcessSourceState

# Build the state object based on user input

state: ProcessSourceState = {
    "document_engine": "auto",      # or "pdfminer", "docx2txt", etc.

    "url_engine": "auto",           # or "readability"

    "output_format": "markdown",
    "file_path": "/path/to/file.pdf",
    "url": None,
    "audio_provider": "openai",     # optional STT configuration

    "audio_model": "whisper-1",
}

# Execute extraction

processed = await extract_content(state)

The extract_content function inspects this state to determine whether the source is a local file or remote URL, then routes to the appropriate engine based on file extension or MIME type.

PDF and Office Document Extraction

For document files, content-core selects parsers based on the document_engine field. When set to "auto", the library automatically detects the format and chooses the appropriate backend.

• PDF files: Uses pdfminer.six to extract text and metadata (title, author)
• DOCX files: Routes through docx2txt to convert Word documents to plain text
• ODT/Office files: Utilizes odfpy for OpenDocument formats

The engine streams the file through the selected parser and returns structured content along with extracted metadata. This process is triggered when file_path points to a local document and document_engine specifies the parser or is set to "auto".

Audio and Video Processing

Media processing follows a two-stage pipeline involving audio extraction and speech-to-text transcription.

For audio files (MP3, WAV, etc.):

1. ffmpeg normalizes the audio to 16kHz mono PCM format
2. The configured speech-to-text provider transcribes the audio (default: OpenAI Whisper)
3. The resulting transcript populates the content field of the returned state

For video files (MP4, YouTube URLs):

1. The system first attempts to fetch closed captions via the YouTube Data API
2. If no captions exist, ffmpeg extracts the audio track from the downloaded video
3. The audio pipeline then transcribes the extracted audio using the configured model

The audio_provider and audio_model fields in ProcessSourceState allow customization of the transcription backend, supporting OpenAI's Whisper API or local models via Ollama.

Web Page and URL Extraction

For web sources, content-core employs a readability-focused approach to extract article content from HTML boilerplate.

When processing a URL:

1. The library downloads the raw HTML using httpx (with support for SOCKS proxies)
2. The HTML passes through readability-lxml to strip navigation, ads, and other non-content elements
3. The cleaned article converts to Markdown using html2text internally

The url_engine field controls this behavior—setting it to "auto" enables the readability pipeline, while specific engines can be selected for alternative parsing strategies.

Engine Selection and Auto-Detection

Open Notebook configures default engines in open_notebook/domain/content_settings.py. When fields are set to "auto", content-core implements the following fallback logic:

• PDF/DOCX/ODT → pdfminer, docx2txt, or odfpy based on file extension
• HTML/URLs → beautifulsoup4 + readability-lxml → Markdown conversion
• Audio → Selected speech-to-text model (OpenAI Whisper by default)
• Video → YouTube caption fetch → audio extraction → speech-to-text

This registry-based design means adding support for new file formats requires updating only the content-core library, without modifying Open Notebook's source code.

Code Examples

Extracting Text from a Local PDF

from content_core import extract_content
from content_core.common import ProcessSourceState

state: ProcessSourceState = {
    "document_engine": "auto",
    "output_format": "markdown",
    "file_path": "/tmp/research_paper.pdf",
    "url": None,
}

processed = await extract_content(state)
print(processed.content)   # Extracted markdown text

print(processed.title)   # Title from PDF metadata

Processing a YouTube Video with Caption Fallback

state = {
    "url_engine": "auto",
    "document_engine": "auto",
    "output_format": "markdown",
    "url": "https://www.youtube.com/watch?v=abc123",
    "audio_provider": "openai",
    "audio_model": "whisper-1",
}

processed = await extract_content(state)
print(processed.content)   # Transcript from captions or transcribed audio

Transcribing an Audio File with a Local Model

state = {
    "document_engine": "auto",
    "output_format": "markdown",
    "file_path": "/tmp/interview.mp3",
    "audio_provider": "ollama",
    "audio_model": "whisper.cpp",
}

processed = await extract_content(state)
print(processed.content)   # Transcribed speech content

Extracting Article Content from a Web URL

state = {
    "url_engine": "auto",
    "document_engine": "auto",
    "output_format": "markdown",
    "url": "https://example.com/blog/article",
}

processed = await extract_content(state)
print(processed.content)   # Clean article body in markdown format

Summary

• content-core handles all content extraction in Open Notebook through a unified extract_content function
• ProcessSourceState objects configure the extraction pipeline, specifying engines for documents, URLs, and audio processing
• Document engines like pdfminer and docx2txt handle PDFs and Office files, while readability-lxml processes web pages
• Media processing uses ffmpeg for audio extraction and supports configurable speech-to-text models including OpenAI Whisper
• Auto-detection allows the system to select appropriate parsers based on file extensions and MIME types without manual configuration

Frequently Asked Questions

What is ProcessSourceState and where is it defined?

ProcessSourceState is a typed dictionary defined in content_core.common that serves as the configuration contract between Open Notebook and the extraction library. It contains fields like document_engine, url_engine, file_path, and audio_model that tell content-core which parsing pipeline to execute and which specific models to use for transcription tasks.

How does content-core handle YouTube videos without captions?

When processing YouTube URLs, content-core first attempts to retrieve official closed captions via the YouTube Data API. If no captions are available, the library downloads the video, extracts the audio track using ffmpeg, and passes that audio to the configured speech-to-text model (such as OpenAI's Whisper) to generate a transcript.

Can I use a custom speech-to-text model instead of OpenAI Whisper?

Yes. The audio_provider and audio_model fields in ProcessSourceState allow you to specify alternative providers. For example, setting "audio_provider": "ollama" and "audio_model": "whisper.cpp" routes transcription through a locally-hosted model rather than the OpenAI API, enabling offline processing and privacy-sensitive workflows.

Which engines are used for extracting text from PDFs?

For PDF documents, content-core defaults to pdfminer.six, a robust Python library that extracts text and metadata while preserving document structure. When document_engine is set to "auto", the system automatically selects pdfminer for .pdf files, docx2txt for .docx files, and odfpy for OpenDocument formats based on file extension detection.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}