# How to Use MediaCrawler for Weibo Data: A Complete Guide to Scraping Weibo Posts

> Learn how to use MediaCrawler for Weibo data with this complete guide. Scrape Weibo posts efficiently using this powerful Python framework and its three-layer architecture.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-26

---

**MediaCrawler is a modular Python framework that scrapes Weibo data through a three-layer architecture: the crawler core in [`media_platform/weibo/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/core.py), an HTTP client in [`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py), and a persistence layer in [`store/weibo/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/weibo/_store_impl.py), configured via [`config/weibo_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/weibo_config.py) and executed with `python main.py --platform weibo`.**

The **NanmiCoder/MediaCrawler** repository provides a comprehensive open-source solution for extracting social media content from Chinese platforms. When you use MediaCrawler for Weibo data, you leverage a specialized pipeline designed to handle authentication, API communication, and structured storage of posts, comments, and media files.

## Understanding the Weibo Crawler Architecture

The Weibo implementation follows a clean separation of concerns across three distinct layers. This design allows you to customize behavior at the network, processing, or storage level without affecting other components.

### Crawler Core Layer

The orchestration logic resides in [`media_platform/weibo/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/core.py). The `WeiboCrawler` class manages the complete lifecycle: initializing the HTTP client, executing login flows, iterating through search results, and dispatching data to the store layer. The `start()` method implements the main control flow that coordinates all subsequent operations.

### HTTP Client Layer

Located in [`media_platform/weibo/client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/client.py), the `WeiboClient` class wraps the mobile Weibo API (`https://m.weibo.cn`). It maintains session state through cookies, headers, and token handling, exposing methods like `get_detail(mid_id)` to fetch specific post metadata and media URLs.

### Data Store Layer

The persistence implementation in [`store/weibo/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/weibo/_store_impl.py) handles writing records to MongoDB and local filesystems. It defines functions such as `update_weibo_note()` and `update_weibo_note_image()` to save structured data according to the schema defined in [`database/models.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/models.py).

## Configuration Settings for Weibo Scraping

All platform-specific parameters live in [`config/weibo_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/weibo_config.py). The framework loads these defaults and allows override via environment variables using `os.getenv`.

Key configuration options include:

```python

# config/weibo_config.py

WEIBO_SEARCH_TYPE = "keyword"          # Options: "keyword" or "user"

WEIBO_KEYWORDS = ["python", "AI"]      # List of search terms

CRAWLER_MAX_NOTES_COUNT = 200          # Maximum posts to fetch per run

SAVE_DATA_PATH = "data/weibo"           # Directory for images and JSON exports

```

You can modify these values directly or inject them at runtime through environment variables to avoid editing source files.

## Running the Weibo Crawler

The entry point for all platforms is [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py), which parses command-line arguments and instantiates the appropriate crawler class.

Execute the Weibo pipeline with:

```bash
python main.py --platform weibo

```

Behind the scenes, [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) imports and initializes the crawler:

```python

# main.py (excerpt)

from media_platform.weibo import WeiboCrawler

crawler = WeiboCrawler()
await crawler.start()

```

The `WeiboCrawler.start()` method executes the following sequence as implemented in [`media_platform/weibo/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/core.py):

1. **Login** – Checks for existing cookie files; if absent, triggers the QR-code flow defined in [`media_platform/weibo/login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/login.py).
2. **Search** – Iterates over `WEIBO_KEYWORDS`, constructs request URLs (`/search?type=...&key=...`), and paginates through results until reaching `CRAWLER_MAX_NOTES_COUNT`.
3. **Detail Extraction** – For each post ID, calls `client.get_detail(mid_id)` to retrieve JSON payloads containing content, images, and engagement metrics.
4. **Persistence** – Invokes the store layer to write `WeiboNote` documents to MongoDB and save image files locally.

## Customizing Your Weibo Data Extraction

You can tailor the crawler behavior through several configuration vectors:

- **Search Type** – Set `WEIBO_SEARCH_TYPE` to `"user"` to crawl a specific user’s timeline instead of keyword-based discovery.
- **Pagination Limits** – Adjust `CRAWLER_MAX_NOTES_COUNT` or modify the internal `weibo_limit_count` (hard-coded to 10 requests per page) to control data volume.
- **Proxy Support** – Configure HTTP proxies via the `proxy` package by setting `HTTPX_PROXY` in your environment or editing [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py) to route requests through intermediary servers.

## Accessing Extracted Weibo Data

The store layer writes to MongoDB collections defined in [`database/models.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/models.py):

- **`WeiboNote`** – Contains post content, `publish_time`, `like_count`, and metadata.
- **`WeiboNoteComment`** – Nested comments linked to parent posts via `note_id`.
- **`WeiboCreator`** – User profiles including `uid`, `screen_name`, and `followers_count`.

For flat-file output, the crawler writes JSON documents to the path specified by `SAVE_DATA_PATH` (default: `data/weibo/`). Images are downloaded to `data/weibo/images/` with filenames derived from the original URLs.

## Practical Code Examples

### Running the Full Crawler Pipeline

This script initializes the crawler and retrieves the first stored note from MongoDB:

```python
import asyncio
from media_platform.weibo import WeiboCrawler

async def main():
    crawler = WeiboCrawler()
    await crawler.start()  # Executes login, search, and storage

    
    # Retrieve first note from database

    from database.db_session import SessionLocal
    from database.models import WeiboNote

    async with SessionLocal() as session:
        result = await session.execute(
            WeiboNote.select().limit(1)
        )
        note = result.scalar_one()
        print("First fetched note:", note.content[:100], "...")

asyncio.run(main())

```

### Using the Low-Level Client for Debugging

For targeted testing or single-post extraction, use the `WeiboClient` directly:

```python
import asyncio
from media_platform.weibo.client import WeiboClient

async def fetch_one(mid):
    client = WeiboClient()
    await client.login()  # Handles QR code or cookie restoration

    detail = await client.get_detail(mid)
    
    print(detail["data"]["text"])
    for pic in detail["data"]["pics"]:
        print("Image URL:", pic["large"]["url"])

asyncio.run(fetch_one("1234567890"))

```

### Overriding Configuration at Runtime

You can inject configuration values without modifying files:

```python
import os
os.environ["WEIBO_KEYWORDS"] = "['data science', 'machine learning']"
os.environ["CRAWLER_MAX_NOTES_COUNT"] = "500"

import asyncio
from media_platform.weibo import WeiboCrawler

asyncio.run(WeiboCrawler().start())

```

## Key Source Files Reference

- [`main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main.py) – CLI entry point for platform selection.
- [`media_platform/weibo/core.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/core.py) – High-level crawling orchestration.
- [`media_platform/weibo/client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/client.py) – HTTP client for `m.weibo.cn` API.
- [`media_platform/weibo/login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/login.py) – QR-code authentication and cookie management.
- [`store/weibo/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/weibo/_store_impl.py) – MongoDB and local file persistence.
- [`config/weibo_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/weibo_config.py) – Platform-specific settings.
- [`model/m_weibo.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/model/m_weibo.py) – Pydantic models for JSON schema validation.
- [`database/models.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/database/models.py) – ORM definitions for `WeiboNote`, `WeiboNoteComment`, and `WeiboCreator`.

## Summary

- **MediaCrawler** provides a three-layer architecture for Weibo scraping: core orchestration, HTTP client, and data store.
- Configure your crawl via [`config/weibo_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/weibo_config.py) using parameters like `WEIBO_SEARCH_TYPE` and `WEIBO_KEYWORDS`.
- Launch the crawler with `python main.py --platform weibo`, which executes login, search, pagination, and storage automatically.
- Access structured data through MongoDB models (`WeiboNote`, `WeiboCreator`) or flat JSON files in `data/weibo/`.
- Use the `WeiboClient` class directly for debugging single posts or building custom extraction logic.

## Frequently Asked Questions

### How do I search for specific user timelines instead of keywords?

Set `WEIBO_SEARCH_TYPE = "user"` in [`config/weibo_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/weibo_config.py) and provide the target username in the `WEIBO_KEYWORDS` list. The crawler will navigate to the user profile endpoint rather than the search API, extracting posts from that specific account's timeline.

### Where does MediaCrawler store the downloaded Weibo images?

Images are saved to the directory specified by `SAVE_DATA_PATH` (default: `data/weibo/images/`). The storage layer in [`store/weibo/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/weibo/_store_impl.py) handles the download logic, preserving original filenames from the Weibo CDN URLs while organizing them by post ID.

### Can I run the Weibo crawler without using MongoDB?

Yes. While the default implementation uses MongoDB for structured data, the crawler also writes JSON files to the local filesystem path defined by `SAVE_DATA_PATH`. You can modify [`store/weibo/_store_impl.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/store/weibo/_store_impl.py) to disable MongoDB calls if you only need flat-file output.

### What authentication method does MediaCrawler use for Weibo?

The framework uses QR-code authentication implemented in [`media_platform/weibo/login.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/weibo/login.py). It checks for existing cookie files first; if valid cookies exist, it reuses them to avoid repeated login flows. Otherwise, it generates a QR code for mobile app scanning to establish the session.