How to Use MediaCrawler for Weibo Data: A Complete Guide to Scraping Weibo Posts

MediaCrawler is a modular Python framework that scrapes Weibo data through a three-layer architecture: the crawler core in media_platform/weibo/core.py, an HTTP client in client.py, and a persistence layer in store/weibo/_store_impl.py, configured via config/weibo_config.py and executed with python main.py --platform weibo.

The NanmiCoder/MediaCrawler repository provides a comprehensive open-source solution for extracting social media content from Chinese platforms. When you use MediaCrawler for Weibo data, you leverage a specialized pipeline designed to handle authentication, API communication, and structured storage of posts, comments, and media files.

Understanding the Weibo Crawler Architecture

The Weibo implementation follows a clean separation of concerns across three distinct layers. This design allows you to customize behavior at the network, processing, or storage level without affecting other components.

Crawler Core Layer

The orchestration logic resides in media_platform/weibo/core.py. The WeiboCrawler class manages the complete lifecycle: initializing the HTTP client, executing login flows, iterating through search results, and dispatching data to the store layer. The start() method implements the main control flow that coordinates all subsequent operations.

HTTP Client Layer

Located in media_platform/weibo/client.py, the WeiboClient class wraps the mobile Weibo API (https://m.weibo.cn). It maintains session state through cookies, headers, and token handling, exposing methods like get_detail(mid_id) to fetch specific post metadata and media URLs.

Data Store Layer

The persistence implementation in store/weibo/_store_impl.py handles writing records to MongoDB and local filesystems. It defines functions such as update_weibo_note() and update_weibo_note_image() to save structured data according to the schema defined in database/models.py.

Configuration Settings for Weibo Scraping

All platform-specific parameters live in config/weibo_config.py. The framework loads these defaults and allows override via environment variables using os.getenv.

Key configuration options include:

# config/weibo_config.py

WEIBO_SEARCH_TYPE = "keyword"          # Options: "keyword" or "user"

WEIBO_KEYWORDS = ["python", "AI"]      # List of search terms

CRAWLER_MAX_NOTES_COUNT = 200          # Maximum posts to fetch per run

SAVE_DATA_PATH = "data/weibo"           # Directory for images and JSON exports

You can modify these values directly or inject them at runtime through environment variables to avoid editing source files.

Running the Weibo Crawler

The entry point for all platforms is main.py, which parses command-line arguments and instantiates the appropriate crawler class.

Execute the Weibo pipeline with:

python main.py --platform weibo

Behind the scenes, main.py imports and initializes the crawler:

# main.py (excerpt)

from media_platform.weibo import WeiboCrawler

crawler = WeiboCrawler()
await crawler.start()

The WeiboCrawler.start() method executes the following sequence as implemented in media_platform/weibo/core.py:

1. Login – Checks for existing cookie files; if absent, triggers the QR-code flow defined in media_platform/weibo/login.py.
2. Search – Iterates over WEIBO_KEYWORDS, constructs request URLs (/search?type=...&key=...), and paginates through results until reaching CRAWLER_MAX_NOTES_COUNT.
3. Detail Extraction – For each post ID, calls client.get_detail(mid_id) to retrieve JSON payloads containing content, images, and engagement metrics.
4. Persistence – Invokes the store layer to write WeiboNote documents to MongoDB and save image files locally.

Customizing Your Weibo Data Extraction

You can tailor the crawler behavior through several configuration vectors:

• Search Type – Set WEIBO_SEARCH_TYPE to "user" to crawl a specific user’s timeline instead of keyword-based discovery.
• Pagination Limits – Adjust CRAWLER_MAX_NOTES_COUNT or modify the internal weibo_limit_count (hard-coded to 10 requests per page) to control data volume.
• Proxy Support – Configure HTTP proxies via the proxy package by setting HTTPX_PROXY in your environment or editing config/base_config.py to route requests through intermediary servers.

Accessing Extracted Weibo Data

The store layer writes to MongoDB collections defined in database/models.py:

• WeiboNote – Contains post content, publish_time, like_count, and metadata.
• WeiboNoteComment – Nested comments linked to parent posts via note_id.
• WeiboCreator – User profiles including uid, screen_name, and followers_count.

For flat-file output, the crawler writes JSON documents to the path specified by SAVE_DATA_PATH (default: data/weibo/). Images are downloaded to data/weibo/images/ with filenames derived from the original URLs.

Practical Code Examples

Running the Full Crawler Pipeline

This script initializes the crawler and retrieves the first stored note from MongoDB:

import asyncio
from media_platform.weibo import WeiboCrawler

async def main():
    crawler = WeiboCrawler()
    await crawler.start()  # Executes login, search, and storage

    
    # Retrieve first note from database

    from database.db_session import SessionLocal
    from database.models import WeiboNote

    async with SessionLocal() as session:
        result = await session.execute(
            WeiboNote.select().limit(1)
        )
        note = result.scalar_one()
        print("First fetched note:", note.content[:100], "...")

asyncio.run(main())

Using the Low-Level Client for Debugging

For targeted testing or single-post extraction, use the WeiboClient directly:

import asyncio
from media_platform.weibo.client import WeiboClient

async def fetch_one(mid):
    client = WeiboClient()
    await client.login()  # Handles QR code or cookie restoration

    detail = await client.get_detail(mid)
    
    print(detail["data"]["text"])
    for pic in detail["data"]["pics"]:
        print("Image URL:", pic["large"]["url"])

asyncio.run(fetch_one("1234567890"))

Overriding Configuration at Runtime

You can inject configuration values without modifying files:

import os
os.environ["WEIBO_KEYWORDS"] = "['data science', 'machine learning']"
os.environ["CRAWLER_MAX_NOTES_COUNT"] = "500"

import asyncio
from media_platform.weibo import WeiboCrawler

asyncio.run(WeiboCrawler().start())

Key Source Files Reference

• main.py – CLI entry point for platform selection.
• media_platform/weibo/core.py – High-level crawling orchestration.
• media_platform/weibo/client.py – HTTP client for m.weibo.cn API.
• media_platform/weibo/login.py – QR-code authentication and cookie management.
• store/weibo/_store_impl.py – MongoDB and local file persistence.
• config/weibo_config.py – Platform-specific settings.
• model/m_weibo.py – Pydantic models for JSON schema validation.
• database/models.py – ORM definitions for WeiboNote, WeiboNoteComment, and WeiboCreator.

Summary

• MediaCrawler provides a three-layer architecture for Weibo scraping: core orchestration, HTTP client, and data store.
• Configure your crawl via config/weibo_config.py using parameters like WEIBO_SEARCH_TYPE and WEIBO_KEYWORDS.
• Launch the crawler with python main.py --platform weibo, which executes login, search, pagination, and storage automatically.
• Access structured data through MongoDB models (WeiboNote, WeiboCreator) or flat JSON files in data/weibo/.
• Use the WeiboClient class directly for debugging single posts or building custom extraction logic.

Frequently Asked Questions

How do I search for specific user timelines instead of keywords?

Set WEIBO_SEARCH_TYPE = "user" in config/weibo_config.py and provide the target username in the WEIBO_KEYWORDS list. The crawler will navigate to the user profile endpoint rather than the search API, extracting posts from that specific account's timeline.

Where does MediaCrawler store the downloaded Weibo images?

Images are saved to the directory specified by SAVE_DATA_PATH (default: data/weibo/images/). The storage layer in store/weibo/_store_impl.py handles the download logic, preserving original filenames from the Weibo CDN URLs while organizing them by post ID.

Can I run the Weibo crawler without using MongoDB?

Yes. While the default implementation uses MongoDB for structured data, the crawler also writes JSON files to the local filesystem path defined by SAVE_DATA_PATH. You can modify store/weibo/_store_impl.py to disable MongoDB calls if you only need flat-file output.

What authentication method does MediaCrawler use for Weibo?

The framework uses QR-code authentication implemented in media_platform/weibo/login.py. It checks for existing cookie files first; if valid cookies exist, it reuses them to avoid repeated login flows. Otherwise, it generates a QR code for mobile app scanning to establish the session.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}