# What Are the Dependencies for MediaCrawler? Complete Requirements Guide

> MediaCrawler needs Python 3.x and over 30 runtime packages like httpx playwright and sqlalchemy. Learn about all MediaCrawler dependencies to get started.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-28

---

**MediaCrawler requires Python 3.x and installs 30+ runtime packages—including `httpx`, `playwright`, `sqlalchemy`, and `motor`—defined in [`requirements.txt`](https://github.com/NanmiCoder/MediaCrawler/blob/main/requirements.txt), plus optional Node.js tooling in [`package.json`](https://github.com/NanmiCoder/MediaCrawler/blob/main/package.json) for documentation.**

The NanmiCoder/MediaCrawler repository is an async-first Python web crawler with a VitePress documentation site. Understanding its dependencies is essential for deployment, troubleshooting, and extending the platform. The project splits its requirements between [`requirements.txt`](https://github.com/NanmiCoder/MediaCrawler/blob/main/requirements.txt) for runtime Python libraries and [`package.json`](https://github.com/NanmiCoder/MediaCrawler/blob/main/package.json) for build-time documentation tools.

## Python Runtime Dependencies

All production code dependencies are pinned or lower-bounded in `[/requirements.txt](https://github.com/NanmiCoder/MediaCrawler/blob/main/requirements.txt)`. The codebase follows an async-first architecture, so the majority of I/O libraries are compatible with `asyncio`.

### Network and Browser Automation

**`httpx==0.28.1`** serves as the primary async HTTP client for fetching web pages and API endpoints (line 1). For headless browser automation, **`playwright==1.45.0`** powers the CDP-based browser module to handle JavaScript-heavy sites (line 3). The project also includes **`requests==2.32.3`** as a synchronous fallback for legacy helpers (line 17) and **`parsel==1.9.1`** for CSS/XPath parsing built on lxml (line 18).

### Data Processing and Visualization

**`pandas==2.2.3`** handles tabular data export to CSV/Excel (line 20), while **`openpyxl>=3.1.2`** provides the underlying Excel read/write capabilities (line 28). For image processing, the codebase uses **`Pillow==12.1.0`** for thumbnails and captcha screenshots (line 2) and **`opencv-python`** for computer vision tasks like QR detection (line 6).

Visualization dependencies include **`matplotlib==3.9.0`** for charting (line 16), **`wordcloud==1.9.3`** for generating visual word clouds (line 15), and **`jieba==0.42.1`** for Chinese text segmentation in the Baidu Tieba scraper (line 14).

### Database and Caching

The storage layer abstracts multiple backends through **`sqlalchemy>=2.0.43`** (line 26) with migration support from **`alembic>=1.16.5`** (line 24). For NoSQL, **`motor>=3.3.0`** provides the async MongoDB driver used in `[/database/db.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/db.py)` (line 27).

Relational database support includes **`aiomysql==0.2.0`** (line 7) and **`asyncmy>=0.2.10`** (line 25) for MySQL, plus **`aiosqlite==0.21.0`** for lightweight local SQLite storage (line 21). Caching is handled by **`redis~=4.6.0`**, implemented in `[/cache/redis_cache.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/cache/redis_cache.py)` (line 8).

### Web Framework and Server

**`fastapi==0.110.2`** (line 11) provides an optional HTTP API layer to expose crawler results as a service, served by **`uvicorn==0.29.0`** (line 12).

### CLI and Configuration

The command-line interface relies on **`typer>=0.12.3`** (line 5). Data validation and settings management use **`pydantic==2.5.2`** (line 9), while **`python-dotenv==1.0.1`** loads environment variables for API keys and credentials (line 13). The **`pyhumps==3.8.0`** library converts between snake_case and camelCase when interacting with external APIs (line 22).

### Testing and Utilities

**`pytest>=7.4.0`** and **`pytest-asyncio>=0.21.0`** handle unit and integration testing for async code paths (lines 29-30). Resilience comes from **`tenacity==8.2.2`** for retry logic (line 4), while **`aiofiles~=23.2.1`** enables non-blocking file I/O for large media assets (line 10).

Security and execution utilities include **`cryptography>=45.0.7`** for TLS and encryption (line 23), **`pyexecjs==1.5.1`** for executing JavaScript snippets during obfuscation decoding (line 19), and **`xhshow>=0.1.9`** for debugging pretty-printing (line 31).

## Documentation Dependencies

The VitePress documentation site requires Node.js tooling defined in `[/package.json](https://github.com/NanmiCoder/MediaCrawler/blob/main/package.json)`. These are **development-only** and not required for crawler execution.

- **`vitepress@^1.3.4`** – Static site generator for the docs folder.
- **`mermaid@^11.12.2`** – Diagramming language for flowcharts.
- **`vitepress-plugin-mermaid@^2.0.17`** – Integration layer for Mermaid diagrams in VitePress pages.

## How to Install MediaCrawler Dependencies

Install the Python runtime dependencies before executing any crawlers:

```bash

# Create isolated environment

python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate

# Install pinned runtime dependencies

pip install -r requirements.txt

```

For local documentation development:

```bash
npm install
npm run docs:dev  # Starts VitePress at http://localhost:5173

```

## Key Implementation Examples

The dependencies are integrated throughout the source code. In `[/tools/httpx_util.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/httpx_util.py)`, the `httpx` client is wrapped for reusable session handling. The `[/tools/cdp_browser.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/cdp_browser.py)` module demonstrates Playwright-based browser automation.

Below is a minimal example showing the interplay between `httpx`, `motor`, and the project's base classes:

```python
import asyncio
from tools.httpx_util import HttpxClient
from database.db import get_mongo_client
from base.base_crawler import BaseCrawler

class DemoCrawler(BaseCrawler):
    async def fetch(self, url: str):
        async with HttpxClient() as client:  # Uses httpx

            resp = await client.get(url)
            return resp.text

async def main():
    crawler = DemoCrawler()
    html = await crawler.fetch("https://example.com")
    
    # Persist to MongoDB using motor

    client = await get_mongo_client()
    await client.media.crawled.insert_one({
        "url": "https://example.com", 
        "html": html
    })

if __name__ == "__main__":
    asyncio.run(main())

```

## Summary

- **Runtime requirements** are defined in [`requirements.txt`](https://github.com/NanmiCoder/MediaCrawler/blob/main/requirements.txt) with 30+ packages covering HTTP, browser automation, databases, and data processing.
- **Core libraries** include `httpx` for async HTTP, `playwright` for browser automation, `sqlalchemy`/`motor` for database abstraction, and `typer` for the CLI.
- **Documentation tools** in [`package.json`](https://github.com/NanmiCoder/MediaCrawler/blob/main/package.json) (VitePress, Mermaid) are optional and only needed for building the docs site.
- **Version pinning** in [`requirements.txt`](https://github.com/NanmiCoder/MediaCrawler/blob/main/requirements.txt) ensures reproducible builds across environments.

## Frequently Asked Questions

### Do I need Node.js to run MediaCrawler?

No. Node.js and the [`package.json`](https://github.com/NanmiCoder/MediaCrawler/blob/main/package.json) dependencies are only required if you want to build or serve the VitePress documentation site locally. The Python crawler runs entirely on the dependencies listed in [`requirements.txt`](https://github.com/NanmiCoder/MediaCrawler/blob/main/requirements.txt).

### Which database drivers are supported out of the box?

MediaCrawler supports MongoDB via `motor`, MySQL via `aiomysql` or `asyncmy`, and SQLite via `aiosqlite`. The SQLAlchemy ORM layer in `[/database/db.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/db.py)` abstracts these connections, while Redis is available for caching through `redis~=4.6.0`.

### Why are exact versions pinned in requirements.txt?

Exact versions (e.g., `httpx==0.28.1`, `playwright==1.45.0`) ensure reproducible builds and prevent breaking changes from upstream updates. This is critical for browser automation tools like Playwright, where API behavior can change between versions.

### Can I install only specific database dependencies?

Yes, though the project is designed to use all listed packages. You can edit [`requirements.txt`](https://github.com/NanmiCoder/MediaCrawler/blob/main/requirements.txt) to remove unused database drivers (e.g., delete `aiomysql` if you only use MongoDB), but you must ensure `sqlalchemy` remains if using any relational database features.