# How to Use MediaCrawler for Bilibili Data: Complete Guide to Video and Comment Scraping

> Learn to scrape Bilibili videos and comments with MediaCrawler's CLI or Python API. This complete guide covers data extraction, storage, and headless browser automation for NanmiCoder users.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-26

---

**Use MediaCrawler's command-line interface or Python API to scrape Bilibili videos, comments, and creator profiles by specifying the platform, crawling type, and storage backend, with support for headless browser automation and multiple export formats.**

The MediaCrawler open-source project provides a robust asynchronous framework for extracting Bilibili content at scale. This guide explains how to configure and execute crawls for video metadata, user comments, and creator statistics using both CLI commands and programmatic Python APIs.

## Understanding the Bilibili Crawling Architecture

MediaCrawler follows a modular pipeline defined in the `media_platform/bilibili/` directory. The architecture separates concerns between browser automation, HTTP API clients, and storage implementations.

| Component | Source File | Key Function |
|-----------|-------------|--------------|
| **Command-line parser** | [`cmd_arg/arg.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/cmd_arg/arg.py) | `parse_cmd` converts flags like `--platform bili` into configuration objects |
| **Configuration** | [`config/bilibili_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/bilibili_config.py) | Stores `BILI_SPECIFIED_ID_LIST` and Bilibili-specific defaults |
| **Crawler core** | [`media_platform/bilibili/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/bilibili/core.py) | `BilibiliCrawler.start` orchestrates browser lifecycle and execution flow |
| **HTTP client** | [`media_platform/bilibili/client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/bilibili/client.py) | `BilibiliClient.search_video_by_keyword` and video info methods |
| **Storage layer** | [`store/bilibili/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/bilibili/_store_impl.py) | `BiliCsvStoreImplement`, `BiliMongoStoreImplement`, and database adapters |

The workflow begins with `parse_cmd` processing CLI arguments, then `BilibiliCrawler.start()` launches a Playwright browser, handles authentication via [`media_platform/bilibili/login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/bilibili/login.py), and dispatches to the appropriate crawling mode based on `config.CRAWLER_TYPE`.

## Installation and Prerequisites

Install the required dependencies and browser binaries before executing Bilibili crawls.

```bash
pip install -r requirements.txt
playwright install chromium

```

Configure authentication by setting cookies in [`config/bilibili_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/bilibili_config.py) or enable interactive QR-code login. The crawler respects `config.MAX_CONCURRENCY_NUM` (default 10) to limit simultaneous requests.

## Command-Line Usage Patterns

MediaCrawler supports three primary crawling modes controlled by the `--type` flag: **search**, **detail**, and **creator**.

### Search Mode

Search mode queries Bilibili's search API for keywords and stores video metadata with comments.

```bash
python -m MediaCrawler \
  --platform bili \
  --type search \
  --keywords "gaming,anime" \
  --save_data_option jsonl \
  --headless true

```

Internally, `BilibiliCrawler.start()` calls `search_by_keywords()`, which uses `BilibiliClient.search_video_by_keyword` while respecting `config.BILI_SEARCH_MODE` settings.

### Detail Mode

Detail mode fetches specific videos by URL or BV ID, extracting full metadata and comment threads.

```bash
python -m MediaCrawler \
  --platform bili \
  --type detail \
  --specified_id "BV1dwuKzmE26,BV14Q4y1n7jz" \
  --save_data_option mongodb

```

This parses IDs via `parse_video_info_from_url`, retrieves data via `BilibiliClient.get_video_info`, and persists results using `BiliMongoStoreImplement.store_content`. Comments are fetched via `BilibiliClient.get_video_all_comments`.

### Creator Mode

Creator mode harvests profile statistics, fan lists, and dynamic posts for specific user IDs.

```bash
python -m MediaCrawler \
  --platform bili \
  --type creator \
  --creator_id "20813884,434377496" \
  --save_data_option csv

```

The crawler calls `get_all_creator_details()` and uses `BilibiliClient.get_creator_all_fans` to enumerate relationships, storing results via `bilibili_store.batch_update_bilibili_creator_fans`.

### Storage Configuration

The `--save_data_option` flag accepts `csv`, `db`, `json`, `jsonl`, `sqlite`, `mongodb`, or `excel`. Each maps to implementations in [`store/bilibili/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/bilibili/_store_impl.py), such as `BiliDbStoreImplement` for SQL databases. Enable video downloads via configuration flags handled by `BilibiliVideo.store_video` in [`store/bilibili/bilibilli_store_media.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/bilibili/bilibilli_store_media.py).

## Programmatic API Usage

Integrate MediaCrawler into existing Python applications by importing core classes directly.

### Running the Crawler from Python

```python
import asyncio
from cmd_arg.arg import parse_cmd
from media_platform.bilibili.core import BilibiliCrawler

async def run():
    args = await parse_cmd([
        "--platform", "bili",
        "--type", "search",
        "--keywords", "anime,music",
        "--save_data_option", "jsonl",
        "--headless", "true"
    ])
    crawler = BilibiliCrawler()
    await crawler.start()
    await crawler.close()

asyncio.run(run())

```

This mimics CLI behavior by populating global `config` values before invoking `BilibiliCrawler.start()`.

### Implementing Custom Storage

Extend the `AbstractStore` base class from [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py) to add PostgreSQL or other backends.

```python
from base.base_crawler import AbstractStore

class PostgresStore(AbstractStore):
    async def store_content(self, content_item: dict):
        # Insert video metadata into PostgreSQL

        pass
    
    async def store_comment(self, comment_item: dict):
        # Insert comment data

        pass
    
    async def store_creator(self, creator: dict):
        # Insert creator profile

        pass

```

Register the implementation in [`media_platform/bilibili/__init__.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/bilibili/__init__.py) to enable it via `--save_data_option postgres`.

### Extracting Creator Fan Data

Access creator relationships programmatically with semaphore-controlled concurrency.

```python
from media_platform.bilibili.core import BilibiliCrawler
import asyncio

async def fetch_fans(creator_id):
    crawler = BilibiliCrawler()
    await crawler.start()
    await crawler.get_fans(
        creator_info={"id": creator_id},
        semaphore=asyncio.Semaphore(5)
    )
    await crawler.close()

asyncio.run(fetch_fans(20813884))

```

This leverages `BilibiliClient.get_creator_all_fans` and stores results via `batch_update_bilibili_creator_fans` defined in [`store/bilibili/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/bilibili/_store_impl.py).

## Key Implementation Files

| File | Purpose |
|------|---------|
| [`media_platform/bilibili/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/bilibili/core.py) | Main `BilibiliCrawler` class with `start()` and crawling logic |
| [`media_platform/bilibili/client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/bilibili/client.py) | API wrapper with `search_video_by_keyword`, `get_video_info`, `get_video_all_comments` |
| [`media_platform/bilibili/login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/bilibili/login.py) | QR-code, phone, and cookie authentication flows |
| [`store/bilibili/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/bilibili/_store_impl.py) | Storage backends including `BiliCsvStoreImplement` and `BiliMongoStoreImplement` |
| [`store/bilibili/bilibilli_store_media.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/bilibili/bilibilli_store_media.py) | Video file downloader |
| [`config/bilibili_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/bilibili_config.py) | Default constants and BV ID lists |
| [`tools/utils.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/utils.py) | Helper functions for logging, user-agent generation, and proxy formatting |
| [`tools/async_file_writer.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/async_file_writer.py) | Async file I/O utilities |

## Summary

- MediaCrawler uses **Playwright** for browser automation and **async HTTP clients** for Bilibili API interaction.
- Three crawling modes exist: **search** (by keyword), **detail** (by BV ID), and **creator** (by user ID).
- Storage options include **CSV, JSON, JSONL, SQLite, MongoDB, and Excel** via implementations in [`store/bilibili/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/bilibili/_store_impl.py).
- The **programmatic API** allows integration into existing Python workflows by importing `BilibiliCrawler` from `media_platform.bilibili.core`.

## Frequently Asked Questions

### How does MediaCrawler handle Bilibili authentication?

The crawler supports three methods defined in [`media_platform/bilibili/login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/bilibili/login.py): QR-code scanning, phone number login, and cookie-based authentication. Set cookies in [`config/bilibili_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/bilibili_config.py) or run with `--headless false` to scan QR codes manually.

### Can I download Bilibili videos using MediaCrawler?

Yes. Enable video downloading via configuration flags. The `BilibiliVideo.store_video` method in [`store/bilibili/bilibilli_store_media.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/bilibili/bilibilli_store_media.py) handles binary downloads and local file storage.

### What is the difference between search mode and detail mode?

Search mode (`--type search`) queries Bilibili's search endpoint for keywords and discovers videos, while detail mode (`--type detail`) processes specific video URLs or BV IDs from `BILI_SPECIFIED_ID_LIST` to extract comprehensive metadata and comments.

### How do I limit concurrent requests to avoid rate limiting?

Control concurrency using the `--max_concurrency_num` flag or by passing a custom `asyncio.Semaphore` to methods like `get_fans()`. The default limit is defined in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) and defaults to 10 simultaneous requests.