# How content-core Extracts Text from Files and URLs in Open Notebook

> Learn how content-core extracts text from files and URLs in Open Notebook. Discover its MIME analysis and specialized parsers for clean markdown.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-24

---

**The `content-core` library extracts text from files and URLs using the `extract_content(state)` function, which auto-detects file types via MIME analysis and routes content through specialized parsers or URL engines to produce clean markdown output.**

Open Notebook relies on the `content-core` library to transform raw documents and web pages into structured markdown. The extraction pipeline processes local files and remote URLs through a unified interface defined by the `ProcessSourceState` dictionary. Understanding this workflow helps developers integrate robust text extraction into their own AI applications.

## The Extraction Workflow

The `extract_content` function orchestrates a six-step pipeline that normalizes diverse input formats into standardized text. According to the Open Notebook source code, this process handles everything from PDF documents to YouTube videos through modular engines.

### State Preparation and Configuration

Extraction begins in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) (lines 34-60), where the system constructs a `ProcessSourceState` dictionary. This state object configures the `url_engine`, `document_engine`, and `output_format` parameters, defaulting to `"auto"` for automatic detection.

### Engine Selection and MIME Detection

The [`content_core/engine_selector.py`](https://github.com/lfnovo/open-notebook/blob/main/content_core/engine_selector.py) module examines the input to determine the appropriate processing path. When `url_engine` or `document_engine` is set to `"auto"`, the library uses `python-magic` or `mimetypes` for MIME type detection. URLs route to `content_core.url_engine.*` modules, while file paths trigger `content_core.doc_engine.*` handlers.

### File-Type Specific Parsers

Document parsing relies on specialized handlers in `content_core/parsers/`:

- **PDF**: Uses `pdfminer.six` or `PyMuPDF` to extract ordered text while preserving layout information
- **DOCX / PPTX**: Leverages `python-docx` and `python-pptx` to read paragraphs and slide content
- **HTML / Markdown**: Employs `beautifulsoup4` or native markdown parsers to strip tags while retaining headings
- **Images**: Invokes `pytesseract` OCR when the engine is set to `"ocr"` or as a fallback for binary files without text layers

All parsers return a plain-text string alongside a detected title when available.

### URL Fetching and Article Extraction

For remote content, the URL engine uses `httpx` (with optional proxy support) to fetch pages. The extraction strategy depends on content type:

1. **Readability extraction**: Uses `readability-lxml` to identify the main article block
2. **HTML conversion**: Falls back to `html2text` for general HTML-to-markdown conversion
3. **YouTube handling**: Checks the YouTube Data API for existing transcripts; if unavailable and an audio model is configured via `ModelManager`, streams video audio to speech-to-text models

### Post-Processing and Output Normalization

The [`content_core/postprocess.py`](https://github.com/lfnovo/open-notebook/blob/main/content_core/postprocess.py) module finalizes the text by converting inline images to markdown tags, sanitizing headings, and removing excess whitespace. The function returns a `ProcessSourceState`-derived object containing `title`, `content`, `url`, `file_path`, `metadata`, and optional audio configuration fields.

## Error Handling and Extensibility

When extraction fails, `content-core` returns a sentinel value with `title="Error"` and a message prefixed with `"Failed to extract content:"`. Open Notebook catches this in [`source.py`](https://github.com/lfnovo/open-notebook/blob/main/source.py) (lines 80-88) and raises an appropriate exception.

The architecture supports extensibility through registration in [`content_core/parsers/__init__.py`](https://github.com/lfnovo/open-notebook/blob/main/content_core/parsers/__init__.py). Developers can add new file-type handlers by implementing the standard parser interface and registering the module.

## Implementation Example

The following example demonstrates extracting text from a local PDF using the async API:

```python
from content_core import extract_content
from content_core.common import ProcessSourceState

# Configure state for automatic detection

state: ProcessSourceState = {
    "file_path": "/tmp/report.pdf",
    "url_engine": "auto",
    "document_engine": "auto",
    "output_format": "markdown",
    # Optional: specify audio model for video URLs

    # "audio_provider": "openai",

    # "audio_model": "whisper-1",

}

# Execute extraction

result = await extract_content(state)

print(f"Title: {result.title}")
print(f"Content preview: {result.content[:500]}...")

```

## Summary

- **`extract_content(state)`** serves as the primary entry point for all text extraction operations in Open Notebook
- **Automatic MIME detection** via `python-magic` eliminates manual file-type configuration when using `"auto"` engine settings
- **Modular parsers** in `content_core/parsers/` handle PDF, Office documents, HTML, and images (via OCR)
- **URL engine** fetches web content using `httpx` and extracts articles via `readability-lxml` or `html2text`, with special handling for YouTube transcripts
- **Post-processing** in [`content_core/postprocess.py`](https://github.com/lfnovo/open-notebook/blob/main/content_core/postprocess.py) normalizes output to markdown format with sanitized formatting

## Frequently Asked Questions

### How does content-core handle password-protected PDFs?

The library does not natively handle encrypted PDFs. When `pdfminer.six` or `PyMuPDF` encounters a password-protected file, extraction fails and returns the standard error sentinel. Users must decrypt files externally before processing.

### Can content-core extract text from scanned documents?

Yes. When the document engine detects image-based PDFs or image files (PNG, JPG) and text extraction returns empty results, setting `document_engine` to `"ocr"` invokes `pytesseract` to perform optical character recognition. This requires the Tesseract OCR engine installed on the host system.

### What happens when a YouTube video lacks transcripts?

If the YouTube Data API returns no transcripts and no audio model is configured, the extraction returns an error. However, when `audio_provider` and `audio_model` are specified in the `ProcessSourceState` (e.g., `"openai"` and `"whisper-1"`), the engine downloads the audio stream and performs speech-to-text conversion automatically.

### Is the extraction process synchronous or asynchronous?

The `extract_content` function is asynchronous and must be awaited. This design allows concurrent processing of multiple files and non-blocking HTTP requests when fetching URLs, as implemented in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py).