# How to Set Up and Use the MediaCrawler WebUI API Server

> Learn how to set up and use the MediaCrawler WebUI API server. Control crawler processes, access data, and stream logs with this FastAPI REST service. Start it with uvicorn.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-27

---

**The MediaCrawler WebUI API server is a FastAPI-based REST service defined in [`api/main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/main.py) that exposes endpoints for controlling crawler processes, accessing data files, and streaming real-time logs via WebSocket, which you can start with `uvicorn api.main:app --port 8080 --reload`.**

The MediaCrawler WebUI API server provides a programmatic interface for the NanmiCoder/MediaCrawler repository, allowing you to manage crawlers and retrieve data without interacting with the command line directly. This server powers the optional web interface while serving as a standalone API for automation and integration. The application uses FastAPI to define routes in modular routers and includes built-in CORS support for local development.

## Prerequisites and Installation

Before starting the MediaCrawler WebUI API server, ensure you have installed the project dependencies. The server requires the same Python environment as the main crawler, specified in the repository's [`requirements.txt`](https://github.com/NanmiCoder/MediaCrawler/blob/main/requirements.txt).

Install dependencies using your preferred package manager:

```bash
pip install -r requirements.txt

# or

uv pip install -r requirements.txt

```

If you intend to use the bundled WebUI frontend, navigate to the `api/webui` directory and build the static assets:

```bash
cd api/webui
npm install
npm run build

```

## Architecture Overview

The server architecture follows FastAPI best practices by separating concerns into distinct router modules. In [`api/main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/main.py) (lines 36–62), the application initializes the FastAPI instance and registers three primary routers under the `/api` prefix.

The codebase organizes functionality into these components:

- **[`api/routers/crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/crawler.py)** – Handles crawler lifecycle management via endpoints for starting, stopping, and monitoring processes
- **[`api/routers/data.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/data.py)** – Manages data files generated by crawlers, providing listing, preview, and download capabilities  
- **[`api/routers/websocket.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/websocket.py)** – Enables real-time communication for log streaming and status updates through WebSocket connections
- **[`api/services/crawler_manager.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/services/crawler_manager.py)** – Contains the core logic for spawning crawler subprocesses, tracking PIDs, and buffering log records

## Starting the Server

You can launch the MediaCrawler WebUI API server using either the direct uvicorn command or the module execution method. The entry point is defined in [`api/main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/main.py) with the FastAPI application instance named `app`.

Use one of the following commands from the repository root:

```bash

# Development mode with auto-reload

uvicorn api.main:app --port 8080 --reload

# Alternative using Python module execution

python -m api.main

```

The docstring in [`api/main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/main.py) (lines 20–23) explicitly documents the recommended start command as `uvicorn api.main:app --port 8080 --reload`.

## Core API Endpoints

The API exposes RESTful endpoints grouped by functionality under the `/api` prefix. Each endpoint returns JSON responses and accepts standard HTTP methods.

### System Health and Environment Verification

Verify server availability and environment configuration before executing crawler operations.

**Health Check:**
Send a GET request to `/api/health` to confirm the server is running:

```bash
curl http://localhost:8080/api/health

```

This returns `{"status":"ok"}` with HTTP 200 when the service is operational.

**Environment Check:**
Validate that the MediaCrawler command-line tool is executable by querying `/api/env/check`. This endpoint (implemented in [`api/main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/main.py) lines 79–143) executes `uv run main.py --help` to verify the underlying crawler environment:

```bash
curl http://localhost:8080/api/env/check

```

A successful response includes `{"success":true,"message":"MediaCrawler environment configured correctly"}`.

### Crawler Control Operations

The crawler router in [`api/routers/crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/crawler.py) provides full process management through the following endpoints.

**Start a Crawler:**
POST to `/api/crawler/start` with a JSON body matching the `CrawlerStartRequest` schema defined in [`api/schemas.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/schemas.py):

```bash
curl -X POST http://localhost:8080/api/crawler/start \
     -H "Content-Type: application/json" \
     -d '{
           "platform": "dy",
           "login_type": "qrcode",
           "search_keyword": "funny cats",
           "save_option": "jsonl"
         }'

```

This initiates the crawler subprocess through [`api/services/crawler_manager.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/services/crawler_manager.py) and returns `{"status":"ok","message":"Crawler started successfully"}`.

**Stop the Crawler:**
Terminate the running process:

```bash
curl -X POST http://localhost:8080/api/crawler/stop

```

**Check Status:**
Retrieve current process state including PID and start time:

```bash
curl http://localhost:8080/api/crawler/status

```

Response format: `{"running":false,"pid":null,"started_at":null}`.

**Retrieve Logs:**
Fetch recent log entries with optional line limiting (default 100 lines):

```bash
curl "http://localhost:8080/api/crawler/logs?limit=50"

```

Returns a JSON array of log objects with `time`, `level`, and `msg` fields.

### Data File Management

The data router in [`api/routers/data.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/data.py) handles files generated by crawler executions.

**List Files:**
GET `/api/data/files` with optional query parameters for filtering:

```bash
curl "http://localhost:8080/api/data/files?platform=dy&file_type=json"

```

Returns metadata including file names, sizes, modification timestamps, and record counts.

**Preview File Content:**
View the first N records of a JSON file without downloading:

```bash
curl "http://localhost:8080/api/data/files/dy_2025-01-01.json?preview=true&limit=10"

```

**Download Files:**
Retrieve the complete file using the download endpoint:

```bash
curl -OJ http://localhost:8080/api/data/download/dy_2025-01-01.json

```

### Real-Time Log Streaming via WebSocket

For real-time monitoring, connect to the WebSocket endpoints defined in [`api/routers/websocket.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/websocket.py). The server exposes two WebSocket routes: `/ws/logs` for log streaming and `/ws/status` for status updates.

Connect using a WebSocket client like `wscat`:

```bash
wscat -c ws://localhost:8080/api/ws/logs

```

Messages arrive as JSON objects whenever the crawler writes output, enabling live log visualization in the WebUI.

## Frontend Integration and CORS

The MediaCrawler WebUI API server includes configurations for serving the optional frontend and handling cross-origin requests.

**CORS Configuration:**
Lines 45–56 in [`api/main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/main.py) configure CORS middleware to allow requests from development servers running on ports 5173 and 3000 (typical Vite development ports). This enables the WebUI to communicate with the API during development.

**Static File Serving:**
When the `api/webui` directory exists (containing the built frontend), the server mounts static routes at lines 86–96 in [`api/main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/main.py). It serves `/assets`, `/logos`, and provides a catch-all `/static` route to support the single-page application (SPA) directly from the same host.

## Configuration Endpoints

The API provides static configuration data through dedicated endpoints defined in [`api/main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/main.py) (lines 45–84).

**Available Platforms:**
Retrieve supported crawler platforms:

```bash
curl http://localhost:8080/api/config/platforms

```

**Crawler Options:**
Get available UI options and parameters:

```bash
curl http://localhost:8080/api/config/options

```

These endpoints return JSON describing valid platform codes and configuration parameters for constructing start requests.

## Summary

- **Entry Point:** The MediaCrawler WebUI API server initializes in [`api/main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/main.py) using FastAPI and registers routers for crawler control, data management, and WebSocket streaming.
- **Startup:** Execute `uvicorn api.main:app --port 8080 --reload` for development or `python -m api.main` for alternative deployment.
- **Core Routers:** [`api/routers/crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/crawler.py) manages process lifecycle, [`api/routers/data.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/data.py) handles file operations, and [`api/routers/websocket.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/routers/websocket.py) enables real-time updates.
- **Process Management:** The [`api/services/crawler_manager.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/services/crawler_manager.py) module spawns crawler subprocesses and tracks PID/status information.
- **Development Support:** Built-in CORS configuration supports frontend development on ports 5173/3000, while static file mounting serves the compiled WebUI from `api/webui`.

## Frequently Asked Questions

### How do I troubleshoot "MediaCrawler environment not configured" errors?

Check the environment verification endpoint at `/api/env/check`, which attempts to run `uv run main.py --help` to validate that the underlying crawler CLI is accessible. Ensure you have installed all dependencies from [`requirements.txt`](https://github.com/NanmiCoder/MediaCrawler/blob/main/requirements.txt) and that the [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) crawler script exists in the repository root.

### Can I run the API server without the WebUI frontend?

Yes, the MediaCrawler WebUI API server functions independently as a REST API. The frontend in `api/webui` is optional; if the directory does not exist, the server simply skips mounting static files (lines 86–96 in [`api/main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/api/main.py)) and continues operating as a headless API server for crawler management.

### What authentication is required for the API endpoints?

The current implementation in the NanmiCoder/MediaCrawler repository does not include authentication middleware. All endpoints are publicly accessible when the server runs. For production deployments, implement authentication through FastAPI dependencies or reverse proxy configuration.

### How do I access real-time logs from my own client application?

Connect to the WebSocket endpoint at `/api/ws/logs` using any WebSocket client library. The server streams JSON-formatted log messages as the crawler produces output. Ensure your client handles the WebSocket protocol and can reconnect if the connection drops during long-running crawl operations.