What Are the Dependencies for MediaCrawler? Complete Requirements Guide

MediaCrawler requires Python 3.x and installs 30+ runtime packages—including httpx, playwright, sqlalchemy, and motor—defined in requirements.txt, plus optional Node.js tooling in package.json for documentation.

The NanmiCoder/MediaCrawler repository is an async-first Python web crawler with a VitePress documentation site. Understanding its dependencies is essential for deployment, troubleshooting, and extending the platform. The project splits its requirements between requirements.txt for runtime Python libraries and package.json for build-time documentation tools.

Python Runtime Dependencies

All production code dependencies are pinned or lower-bounded in [/requirements.txt](https://github.com/NanmiCoder/MediaCrawler/blob/main/requirements.txt). The codebase follows an async-first architecture, so the majority of I/O libraries are compatible with asyncio.

Network and Browser Automation

httpx==0.28.1 serves as the primary async HTTP client for fetching web pages and API endpoints (line 1). For headless browser automation, playwright==1.45.0 powers the CDP-based browser module to handle JavaScript-heavy sites (line 3). The project also includes requests==2.32.3 as a synchronous fallback for legacy helpers (line 17) and parsel==1.9.1 for CSS/XPath parsing built on lxml (line 18).

Data Processing and Visualization

pandas==2.2.3 handles tabular data export to CSV/Excel (line 20), while openpyxl>=3.1.2 provides the underlying Excel read/write capabilities (line 28). For image processing, the codebase uses Pillow==12.1.0 for thumbnails and captcha screenshots (line 2) and opencv-python for computer vision tasks like QR detection (line 6).

Visualization dependencies include matplotlib==3.9.0 for charting (line 16), wordcloud==1.9.3 for generating visual word clouds (line 15), and jieba==0.42.1 for Chinese text segmentation in the Baidu Tieba scraper (line 14).

Database and Caching

The storage layer abstracts multiple backends through sqlalchemy>=2.0.43 (line 26) with migration support from alembic>=1.16.5 (line 24). For NoSQL, motor>=3.3.0 provides the async MongoDB driver used in [/database/db.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/db.py) (line 27).

Relational database support includes aiomysql==0.2.0 (line 7) and asyncmy>=0.2.10 (line 25) for MySQL, plus aiosqlite==0.21.0 for lightweight local SQLite storage (line 21). Caching is handled by redis~=4.6.0, implemented in [/cache/redis_cache.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/cache/redis_cache.py) (line 8).

Web Framework and Server

fastapi==0.110.2 (line 11) provides an optional HTTP API layer to expose crawler results as a service, served by uvicorn==0.29.0 (line 12).

CLI and Configuration

The command-line interface relies on typer>=0.12.3 (line 5). Data validation and settings management use pydantic==2.5.2 (line 9), while python-dotenv==1.0.1 loads environment variables for API keys and credentials (line 13). The pyhumps==3.8.0 library converts between snake_case and camelCase when interacting with external APIs (line 22).

Testing and Utilities

pytest>=7.4.0 and pytest-asyncio>=0.21.0 handle unit and integration testing for async code paths (lines 29-30). Resilience comes from tenacity==8.2.2 for retry logic (line 4), while aiofiles~=23.2.1 enables non-blocking file I/O for large media assets (line 10).

Security and execution utilities include cryptography>=45.0.7 for TLS and encryption (line 23), pyexecjs==1.5.1 for executing JavaScript snippets during obfuscation decoding (line 19), and xhshow>=0.1.9 for debugging pretty-printing (line 31).

Documentation Dependencies

The VitePress documentation site requires Node.js tooling defined in [/package.json](https://github.com/NanmiCoder/MediaCrawler/blob/main/package.json). These are development-only and not required for crawler execution.

• vitepress@^1.3.4 – Static site generator for the docs folder.
• mermaid@^11.12.2 – Diagramming language for flowcharts.
• vitepress-plugin-mermaid@^2.0.17 – Integration layer for Mermaid diagrams in VitePress pages.

How to Install MediaCrawler Dependencies

Install the Python runtime dependencies before executing any crawlers:

# Create isolated environment

python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate

# Install pinned runtime dependencies

pip install -r requirements.txt

For local documentation development:

npm install
npm run docs:dev  # Starts VitePress at http://localhost:5173

Key Implementation Examples

The dependencies are integrated throughout the source code. In [/tools/httpx_util.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/httpx_util.py), the httpx client is wrapped for reusable session handling. The [/tools/cdp_browser.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/cdp_browser.py) module demonstrates Playwright-based browser automation.

Below is a minimal example showing the interplay between httpx, motor, and the project's base classes:

import asyncio
from tools.httpx_util import HttpxClient
from database.db import get_mongo_client
from base.base_crawler import BaseCrawler

class DemoCrawler(BaseCrawler):
    async def fetch(self, url: str):
        async with HttpxClient() as client:  # Uses httpx

            resp = await client.get(url)
            return resp.text

async def main():
    crawler = DemoCrawler()
    html = await crawler.fetch("https://example.com")
    
    # Persist to MongoDB using motor

    client = await get_mongo_client()
    await client.media.crawled.insert_one({
        "url": "https://example.com", 
        "html": html
    })

if __name__ == "__main__":
    asyncio.run(main())

Summary

• Runtime requirements are defined in requirements.txt with 30+ packages covering HTTP, browser automation, databases, and data processing.
• Core libraries include httpx for async HTTP, playwright for browser automation, sqlalchemy/motor for database abstraction, and typer for the CLI.
• Documentation tools in package.json (VitePress, Mermaid) are optional and only needed for building the docs site.
• Version pinning in requirements.txt ensures reproducible builds across environments.

Frequently Asked Questions

Do I need Node.js to run MediaCrawler?

No. Node.js and the package.json dependencies are only required if you want to build or serve the VitePress documentation site locally. The Python crawler runs entirely on the dependencies listed in requirements.txt.

Which database drivers are supported out of the box?

MediaCrawler supports MongoDB via motor, MySQL via aiomysql or asyncmy, and SQLite via aiosqlite. The SQLAlchemy ORM layer in [/database/db.py](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/db.py) abstracts these connections, while Redis is available for caching through redis~=4.6.0.

Why are exact versions pinned in requirements.txt?

Exact versions (e.g., httpx==0.28.1, playwright==1.45.0) ensure reproducible builds and prevent breaking changes from upstream updates. This is critical for browser automation tools like Playwright, where API behavior can change between versions.

Can I install only specific database dependencies?

Yes, though the project is designed to use all listed packages. You can edit requirements.txt to remove unused database drivers (e.g., delete aiomysql if you only use MongoDB), but you must ensure sqlalchemy remains if using any relational database features.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}