How to Use MediaCrawler for Bilibili Data: Complete Guide to Video and Comment Scraping

Use MediaCrawler's command-line interface or Python API to scrape Bilibili videos, comments, and creator profiles by specifying the platform, crawling type, and storage backend, with support for headless browser automation and multiple export formats.

The MediaCrawler open-source project provides a robust asynchronous framework for extracting Bilibili content at scale. This guide explains how to configure and execute crawls for video metadata, user comments, and creator statistics using both CLI commands and programmatic Python APIs.

Understanding the Bilibili Crawling Architecture

MediaCrawler follows a modular pipeline defined in the media_platform/bilibili/ directory. The architecture separates concerns between browser automation, HTTP API clients, and storage implementations.

Component	Source File	Key Function
Command-line parser	cmd_arg/arg.py	parse_cmd converts flags like --platform bili into configuration objects
Configuration	config/bilibili_config.py	Stores BILI_SPECIFIED_ID_LIST and Bilibili-specific defaults
Crawler core	media_platform/bilibili/core.py	BilibiliCrawler.start orchestrates browser lifecycle and execution flow
HTTP client	media_platform/bilibili/client.py	BilibiliClient.search_video_by_keyword and video info methods
Storage layer	store/bilibili/_store_impl.py	BiliCsvStoreImplement, BiliMongoStoreImplement, and database adapters

The workflow begins with parse_cmd processing CLI arguments, then BilibiliCrawler.start() launches a Playwright browser, handles authentication via media_platform/bilibili/login.py, and dispatches to the appropriate crawling mode based on config.CRAWLER_TYPE.

Installation and Prerequisites

Install the required dependencies and browser binaries before executing Bilibili crawls.

pip install -r requirements.txt
playwright install chromium

Configure authentication by setting cookies in config/bilibili_config.py or enable interactive QR-code login. The crawler respects config.MAX_CONCURRENCY_NUM (default 10) to limit simultaneous requests.

Command-Line Usage Patterns

MediaCrawler supports three primary crawling modes controlled by the --type flag: search, detail, and creator.

Search Mode

Search mode queries Bilibili's search API for keywords and stores video metadata with comments.

python -m MediaCrawler \
  --platform bili \
  --type search \
  --keywords "gaming,anime" \
  --save_data_option jsonl \
  --headless true

Internally, BilibiliCrawler.start() calls search_by_keywords(), which uses BilibiliClient.search_video_by_keyword while respecting config.BILI_SEARCH_MODE settings.

Detail Mode

Detail mode fetches specific videos by URL or BV ID, extracting full metadata and comment threads.

python -m MediaCrawler \
  --platform bili \
  --type detail \
  --specified_id "BV1dwuKzmE26,BV14Q4y1n7jz" \
  --save_data_option mongodb

This parses IDs via parse_video_info_from_url, retrieves data via BilibiliClient.get_video_info, and persists results using BiliMongoStoreImplement.store_content. Comments are fetched via BilibiliClient.get_video_all_comments.

Creator Mode

Creator mode harvests profile statistics, fan lists, and dynamic posts for specific user IDs.

python -m MediaCrawler \
  --platform bili \
  --type creator \
  --creator_id "20813884,434377496" \
  --save_data_option csv

The crawler calls get_all_creator_details() and uses BilibiliClient.get_creator_all_fans to enumerate relationships, storing results via bilibili_store.batch_update_bilibili_creator_fans.

Storage Configuration

The --save_data_option flag accepts csv, db, json, jsonl, sqlite, mongodb, or excel. Each maps to implementations in store/bilibili/_store_impl.py, such as BiliDbStoreImplement for SQL databases. Enable video downloads via configuration flags handled by BilibiliVideo.store_video in store/bilibili/bilibilli_store_media.py.

Programmatic API Usage

Integrate MediaCrawler into existing Python applications by importing core classes directly.

Running the Crawler from Python

import asyncio
from cmd_arg.arg import parse_cmd
from media_platform.bilibili.core import BilibiliCrawler

async def run():
    args = await parse_cmd([
        "--platform", "bili",
        "--type", "search",
        "--keywords", "anime,music",
        "--save_data_option", "jsonl",
        "--headless", "true"
    ])
    crawler = BilibiliCrawler()
    await crawler.start()
    await crawler.close()

asyncio.run(run())

This mimics CLI behavior by populating global config values before invoking BilibiliCrawler.start().

Implementing Custom Storage

Extend the AbstractStore base class from base/base_crawler.py to add PostgreSQL or other backends.

from base.base_crawler import AbstractStore

class PostgresStore(AbstractStore):
    async def store_content(self, content_item: dict):
        # Insert video metadata into PostgreSQL

        pass
    
    async def store_comment(self, comment_item: dict):
        # Insert comment data

        pass
    
    async def store_creator(self, creator: dict):
        # Insert creator profile

        pass

Register the implementation in media_platform/bilibili/__init__.py to enable it via --save_data_option postgres.

Extracting Creator Fan Data

Access creator relationships programmatically with semaphore-controlled concurrency.

from media_platform.bilibili.core import BilibiliCrawler
import asyncio

async def fetch_fans(creator_id):
    crawler = BilibiliCrawler()
    await crawler.start()
    await crawler.get_fans(
        creator_info={"id": creator_id},
        semaphore=asyncio.Semaphore(5)
    )
    await crawler.close()

asyncio.run(fetch_fans(20813884))

This leverages BilibiliClient.get_creator_all_fans and stores results via batch_update_bilibili_creator_fans defined in store/bilibili/_store_impl.py.

Key Implementation Files

File	Purpose
media_platform/bilibili/core.py	Main BilibiliCrawler class with start() and crawling logic
media_platform/bilibili/client.py	API wrapper with search_video_by_keyword, get_video_info, get_video_all_comments
media_platform/bilibili/login.py	QR-code, phone, and cookie authentication flows
store/bilibili/_store_impl.py	Storage backends including BiliCsvStoreImplement and BiliMongoStoreImplement
store/bilibili/bilibilli_store_media.py	Video file downloader
config/bilibili_config.py	Default constants and BV ID lists
tools/utils.py	Helper functions for logging, user-agent generation, and proxy formatting
tools/async_file_writer.py	Async file I/O utilities

Summary

• MediaCrawler uses Playwright for browser automation and async HTTP clients for Bilibili API interaction.
• Three crawling modes exist: search (by keyword), detail (by BV ID), and creator (by user ID).
• Storage options include CSV, JSON, JSONL, SQLite, MongoDB, and Excel via implementations in store/bilibili/_store_impl.py.
• The programmatic API allows integration into existing Python workflows by importing BilibiliCrawler from media_platform.bilibili.core.

Frequently Asked Questions

How does MediaCrawler handle Bilibili authentication?

The crawler supports three methods defined in media_platform/bilibili/login.py: QR-code scanning, phone number login, and cookie-based authentication. Set cookies in config/bilibili_config.py or run with --headless false to scan QR codes manually.

Can I download Bilibili videos using MediaCrawler?

Yes. Enable video downloading via configuration flags. The BilibiliVideo.store_video method in store/bilibili/bilibilli_store_media.py handles binary downloads and local file storage.

What is the difference between search mode and detail mode?

Search mode (--type search) queries Bilibili's search endpoint for keywords and discovers videos, while detail mode (--type detail) processes specific video URLs or BV IDs from BILI_SPECIFIED_ID_LIST to extract comprehensive metadata and comments.

How do I limit concurrent requests to avoid rate limiting?

Control concurrency using the --max_concurrency_num flag or by passing a custom asyncio.Semaphore to methods like get_fans(). The default limit is defined in config/base_config.py and defaults to 10 simultaneous requests.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}