How content-core Extracts Text from Files and URLs in Open Notebook

The content-core library extracts text from files and URLs using the extract_content(state) function, which auto-detects file types via MIME analysis and routes content through specialized parsers or URL engines to produce clean markdown output.

Open Notebook relies on the content-core library to transform raw documents and web pages into structured markdown. The extraction pipeline processes local files and remote URLs through a unified interface defined by the ProcessSourceState dictionary. Understanding this workflow helps developers integrate robust text extraction into their own AI applications.

The Extraction Workflow

The extract_content function orchestrates a six-step pipeline that normalizes diverse input formats into standardized text. According to the Open Notebook source code, this process handles everything from PDF documents to YouTube videos through modular engines.

State Preparation and Configuration

Extraction begins in open_notebook/graphs/source.py (lines 34-60), where the system constructs a ProcessSourceState dictionary. This state object configures the url_engine, document_engine, and output_format parameters, defaulting to "auto" for automatic detection.

Engine Selection and MIME Detection

The content_core/engine_selector.py module examines the input to determine the appropriate processing path. When url_engine or document_engine is set to "auto", the library uses python-magic or mimetypes for MIME type detection. URLs route to content_core.url_engine.* modules, while file paths trigger content_core.doc_engine.* handlers.

File-Type Specific Parsers

Document parsing relies on specialized handlers in content_core/parsers/:

• PDF: Uses pdfminer.six or PyMuPDF to extract ordered text while preserving layout information
• DOCX / PPTX: Leverages python-docx and python-pptx to read paragraphs and slide content
• HTML / Markdown: Employs beautifulsoup4 or native markdown parsers to strip tags while retaining headings
• Images: Invokes pytesseract OCR when the engine is set to "ocr" or as a fallback for binary files without text layers

All parsers return a plain-text string alongside a detected title when available.

URL Fetching and Article Extraction

For remote content, the URL engine uses httpx (with optional proxy support) to fetch pages. The extraction strategy depends on content type:

1. Readability extraction: Uses readability-lxml to identify the main article block
2. HTML conversion: Falls back to html2text for general HTML-to-markdown conversion
3. YouTube handling: Checks the YouTube Data API for existing transcripts; if unavailable and an audio model is configured via ModelManager, streams video audio to speech-to-text models

Post-Processing and Output Normalization

The content_core/postprocess.py module finalizes the text by converting inline images to markdown tags, sanitizing headings, and removing excess whitespace. The function returns a ProcessSourceState-derived object containing title, content, url, file_path, metadata, and optional audio configuration fields.

Error Handling and Extensibility

When extraction fails, content-core returns a sentinel value with title="Error" and a message prefixed with "Failed to extract content:". Open Notebook catches this in source.py (lines 80-88) and raises an appropriate exception.

The architecture supports extensibility through registration in content_core/parsers/__init__.py. Developers can add new file-type handlers by implementing the standard parser interface and registering the module.

Implementation Example

The following example demonstrates extracting text from a local PDF using the async API:

from content_core import extract_content
from content_core.common import ProcessSourceState

# Configure state for automatic detection

state: ProcessSourceState = {
    "file_path": "/tmp/report.pdf",
    "url_engine": "auto",
    "document_engine": "auto",
    "output_format": "markdown",
    # Optional: specify audio model for video URLs

    # "audio_provider": "openai",

    # "audio_model": "whisper-1",

}

# Execute extraction

result = await extract_content(state)

print(f"Title: {result.title}")
print(f"Content preview: {result.content[:500]}...")

Summary

• extract_content(state) serves as the primary entry point for all text extraction operations in Open Notebook
• Automatic MIME detection via python-magic eliminates manual file-type configuration when using "auto" engine settings
• Modular parsers in content_core/parsers/ handle PDF, Office documents, HTML, and images (via OCR)
• URL engine fetches web content using httpx and extracts articles via readability-lxml or html2text, with special handling for YouTube transcripts
• Post-processing in content_core/postprocess.py normalizes output to markdown format with sanitized formatting

Frequently Asked Questions

How does content-core handle password-protected PDFs?

The library does not natively handle encrypted PDFs. When pdfminer.six or PyMuPDF encounters a password-protected file, extraction fails and returns the standard error sentinel. Users must decrypt files externally before processing.

Can content-core extract text from scanned documents?

Yes. When the document engine detects image-based PDFs or image files (PNG, JPG) and text extraction returns empty results, setting document_engine to "ocr" invokes pytesseract to perform optical character recognition. This requires the Tesseract OCR engine installed on the host system.

What happens when a YouTube video lacks transcripts?

If the YouTube Data API returns no transcripts and no audio model is configured, the extraction returns an error. However, when audio_provider and audio_model are specified in the ProcessSourceState (e.g., "openai" and "whisper-1"), the engine downloads the audio stream and performs speech-to-text conversion automatically.

Is the extraction process synchronous or asynchronous?

The extract_content function is asynchronous and must be awaited. This design allows concurrent processing of multiple files and non-blocking HTTP requests when fetching URLs, as implemented in open_notebook/graphs/source.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}