How to Set Up and Use the MediaCrawler WebUI API Server

The MediaCrawler WebUI API server is a FastAPI-based REST service defined in api/main.py that exposes endpoints for controlling crawler processes, accessing data files, and streaming real-time logs via WebSocket, which you can start with uvicorn api.main:app --port 8080 --reload.

The MediaCrawler WebUI API server provides a programmatic interface for the NanmiCoder/MediaCrawler repository, allowing you to manage crawlers and retrieve data without interacting with the command line directly. This server powers the optional web interface while serving as a standalone API for automation and integration. The application uses FastAPI to define routes in modular routers and includes built-in CORS support for local development.

Prerequisites and Installation

Before starting the MediaCrawler WebUI API server, ensure you have installed the project dependencies. The server requires the same Python environment as the main crawler, specified in the repository's requirements.txt.

Install dependencies using your preferred package manager:

pip install -r requirements.txt

# or

uv pip install -r requirements.txt

If you intend to use the bundled WebUI frontend, navigate to the api/webui directory and build the static assets:

cd api/webui
npm install
npm run build

Architecture Overview

The server architecture follows FastAPI best practices by separating concerns into distinct router modules. In api/main.py (lines 36–62), the application initializes the FastAPI instance and registers three primary routers under the /api prefix.

The codebase organizes functionality into these components:

• api/routers/crawler.py – Handles crawler lifecycle management via endpoints for starting, stopping, and monitoring processes
• api/routers/data.py – Manages data files generated by crawlers, providing listing, preview, and download capabilities
• api/routers/websocket.py – Enables real-time communication for log streaming and status updates through WebSocket connections
• api/services/crawler_manager.py – Contains the core logic for spawning crawler subprocesses, tracking PIDs, and buffering log records

Starting the Server

You can launch the MediaCrawler WebUI API server using either the direct uvicorn command or the module execution method. The entry point is defined in api/main.py with the FastAPI application instance named app.

Use one of the following commands from the repository root:

# Development mode with auto-reload

uvicorn api.main:app --port 8080 --reload

# Alternative using Python module execution

python -m api.main

The docstring in api/main.py (lines 20–23) explicitly documents the recommended start command as uvicorn api.main:app --port 8080 --reload.

Core API Endpoints

The API exposes RESTful endpoints grouped by functionality under the /api prefix. Each endpoint returns JSON responses and accepts standard HTTP methods.

System Health and Environment Verification

Verify server availability and environment configuration before executing crawler operations.

Health Check: Send a GET request to /api/health to confirm the server is running:

curl http://localhost:8080/api/health

This returns {"status":"ok"} with HTTP 200 when the service is operational.

Environment Check: Validate that the MediaCrawler command-line tool is executable by querying /api/env/check. This endpoint (implemented in api/main.py lines 79–143) executes uv run main.py --help to verify the underlying crawler environment:

curl http://localhost:8080/api/env/check

A successful response includes {"success":true,"message":"MediaCrawler environment configured correctly"}.

Crawler Control Operations

The crawler router in api/routers/crawler.py provides full process management through the following endpoints.

Start a Crawler: POST to /api/crawler/start with a JSON body matching the CrawlerStartRequest schema defined in api/schemas.py:

curl -X POST http://localhost:8080/api/crawler/start \
     -H "Content-Type: application/json" \
     -d '{
           "platform": "dy",
           "login_type": "qrcode",
           "search_keyword": "funny cats",
           "save_option": "jsonl"
         }'

This initiates the crawler subprocess through api/services/crawler_manager.py and returns {"status":"ok","message":"Crawler started successfully"}.

Stop the Crawler: Terminate the running process:

curl -X POST http://localhost:8080/api/crawler/stop

Check Status: Retrieve current process state including PID and start time:

curl http://localhost:8080/api/crawler/status

Response format: {"running":false,"pid":null,"started_at":null}.

Retrieve Logs: Fetch recent log entries with optional line limiting (default 100 lines):

curl "http://localhost:8080/api/crawler/logs?limit=50"

Returns a JSON array of log objects with time, level, and msg fields.

Data File Management

The data router in api/routers/data.py handles files generated by crawler executions.

List Files: GET /api/data/files with optional query parameters for filtering:

curl "http://localhost:8080/api/data/files?platform=dy&file_type=json"

Returns metadata including file names, sizes, modification timestamps, and record counts.

Preview File Content: View the first N records of a JSON file without downloading:

curl "http://localhost:8080/api/data/files/dy_2025-01-01.json?preview=true&limit=10"

Download Files: Retrieve the complete file using the download endpoint:

curl -OJ http://localhost:8080/api/data/download/dy_2025-01-01.json

Real-Time Log Streaming via WebSocket

For real-time monitoring, connect to the WebSocket endpoints defined in api/routers/websocket.py. The server exposes two WebSocket routes: /ws/logs for log streaming and /ws/status for status updates.

Connect using a WebSocket client like wscat:

wscat -c ws://localhost:8080/api/ws/logs

Messages arrive as JSON objects whenever the crawler writes output, enabling live log visualization in the WebUI.

Frontend Integration and CORS

The MediaCrawler WebUI API server includes configurations for serving the optional frontend and handling cross-origin requests.

CORS Configuration: Lines 45–56 in api/main.py configure CORS middleware to allow requests from development servers running on ports 5173 and 3000 (typical Vite development ports). This enables the WebUI to communicate with the API during development.

Static File Serving: When the api/webui directory exists (containing the built frontend), the server mounts static routes at lines 86–96 in api/main.py. It serves /assets, /logos, and provides a catch-all /static route to support the single-page application (SPA) directly from the same host.

Configuration Endpoints

The API provides static configuration data through dedicated endpoints defined in api/main.py (lines 45–84).

Available Platforms: Retrieve supported crawler platforms:

curl http://localhost:8080/api/config/platforms

Crawler Options: Get available UI options and parameters:

curl http://localhost:8080/api/config/options

These endpoints return JSON describing valid platform codes and configuration parameters for constructing start requests.

Summary

• Entry Point: The MediaCrawler WebUI API server initializes in api/main.py using FastAPI and registers routers for crawler control, data management, and WebSocket streaming.
• Startup: Execute uvicorn api.main:app --port 8080 --reload for development or python -m api.main for alternative deployment.
• Core Routers: api/routers/crawler.py manages process lifecycle, api/routers/data.py handles file operations, and api/routers/websocket.py enables real-time updates.
• Process Management: The api/services/crawler_manager.py module spawns crawler subprocesses and tracks PID/status information.
• Development Support: Built-in CORS configuration supports frontend development on ports 5173/3000, while static file mounting serves the compiled WebUI from api/webui.

Frequently Asked Questions

How do I troubleshoot "MediaCrawler environment not configured" errors?

Check the environment verification endpoint at /api/env/check, which attempts to run uv run main.py --help to validate that the underlying crawler CLI is accessible. Ensure you have installed all dependencies from requirements.txt and that the main.py crawler script exists in the repository root.

Can I run the API server without the WebUI frontend?

Yes, the MediaCrawler WebUI API server functions independently as a REST API. The frontend in api/webui is optional; if the directory does not exist, the server simply skips mounting static files (lines 86–96 in api/main.py) and continues operating as a headless API server for crawler management.

What authentication is required for the API endpoints?

The current implementation in the NanmiCoder/MediaCrawler repository does not include authentication middleware. All endpoints are publicly accessible when the server runs. For production deployments, implement authentication through FastAPI dependencies or reverse proxy configuration.

How do I access real-time logs from my own client application?

Connect to the WebSocket endpoint at /api/ws/logs using any WebSocket client library. The server streams JSON-formatted log messages as the crawler produces output. Ensure your client handles the WebSocket protocol and can reconnect if the connection drops during long-running crawl operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}