How to Use MediaCrawler for Kuaishou Data: Configuration, Modes, and Implementation

MediaCrawler enables automated collection of Kuaishou video metadata, comments, and creator profiles through three configurable crawling modes backed by an async Playwright and GraphQL architecture.

MediaCrawler is an open-source multi-platform crawler framework maintained by NanmiCoder that supports automated data extraction from Chinese social media platforms. This guide explains how to use MediaCrawler for Kuaishou data collection, covering the specific configuration files, core components, and execution flow required to scrape video information and user-generated content.

MediaCrawler Architecture for Kuaishou

The Kuaishou implementation follows a modular design pattern with clear separation between configuration, client operations, and data persistence. Understanding these components is essential for customizing the crawler behavior.

Configuration Layer

The config/ks_config.py file defines runtime parameters including CRAWLER_TYPE, concurrency limits, and target ID lists. This configuration drives the behavior of the KuaishouCrawler class in media_platform/kuaishou/core.py. Key parameters include MAX_CONCURRENCY_NUM for controlling async workers and CRAWLER_MAX_SLEEP_SEC for rate limiting.

URL Parsing and ID Extraction

The media_platform/kuaishou/help.py module provides utility functions to extract video and creator identifiers from Kuaishou URLs. The parse_video_info_from_url function returns a VideoUrlInfo Pydantic model defined in model/m_kuaishou.py, which standardizes the extraction of video IDs from various URL formats.

Core Crawler and API Client

The KuaishouCrawler class in media_platform/kuaishou/core.py orchestrates browser initialization via launch_browser or launch_browser_with_cdp, session management, and task scheduling. It utilizes KuaiShouClient from media_platform/kuaishou/client.py to execute GraphQL queries against Kuaishou's private endpoints for search, detail, and creator data.

Authentication Handling

Session management is handled by KuaishouLogin in media_platform/kuaishou/login.py, supporting both QR-code and cookie-based authentication methods. The crawler validates session status using ks_client.pong before executing data collection tasks.

Storage Implementations

Data persistence is abstracted through store/kuaishou/_store_impl.py, which implements handlers for CSV, JSON/JSONL, SQLite, and MongoDB formats via AsyncFileWriter and database-specific adapters. The storage layer exports update_kuaishou_video, batch_update_ks_video_comments, and save_creator functions.

Supported Crawling Modes

MediaCrawler supports three distinct operational modes for Kuaishou data collection, controlled by the CRAWLER_TYPE configuration parameter in config/ks_config.py.

• Search Mode: Discovers videos by keyword using search_info_by_keyword in the client. Ideal for trend analysis and content discovery.
• Detail Mode: Retrieves specific video metadata and comments using get_video_info and get_video_all_comments. Requires video IDs or URLs in KS_SPECIFIED_ID_LIST.
• Creator Mode: Scrapes complete creator profiles and their video catalogs using get_all_videos_by_creator. Extracts all videos from specific user accounts.

Step-by-Step Configuration

To initiate Kuaishou data collection, configure the environment and target parameters before execution.

First, install dependencies:

pip install -r requirements.txt

Create a .env file based on .env.example. For QR-code login, leave COOKIES empty; the crawler will display a QR image in the console during execution via KuaishouLogin.

Configure config/ks_config.py for detail mode:

# config/ks_config.py

CRAWLER_TYPE = "detail"  # Options: "search", "detail", "creator"

KS_SPECIFIED_ID_LIST = [
    "https://www.kuaishou.com/short-video/3x3zxz4mjrsc8ke",
    "3xf8enb8dbj6uig",  # Raw video ID format also supported

]
MAX_CONCURRENCY_NUM = 5
CRAWLER_MAX_SLEEP_SEC = 2  # Rate limiting between requests

CRAWLER_MAX_NOTES_COUNT = 20  # Maximum items to fetch

Implementing the Crawl Workflow

Once configured, execute the crawler via main.py or programmatically using the core classes.

Command-line execution:

python main.py

The platform selector in main.py automatically instantiates KuaishouCrawler when config.PLATFORM is set appropriately.

Parsing Kuaishou URLs programmatically:

from media_platform.kuaishou.help import parse_video_info_from_url

url = "https://www.kuaishou.com/short-video/3x3zxz4mjrsc8ke"
info = parse_video_info_from_url(url)

# Returns VideoUrlInfo(video_id='3x3zxz4mjrsc8ke', url_type='normal')

print(info.video_id)  # Output: 3x3zxz4mjrsc8ke

Direct programmatic invocation:

import asyncio
from media_platform.kuaishou.core import KuaishouCrawler

async def run_crawler():
    crawler = KuaishouCrawler()
    await crawler.start()  # Executes based on config.CRAWLER_TYPE

asyncio.run(run_crawler())

The KuaishouCrawler.start() method in media_platform/kuaishou/core.py handles the complete lifecycle: browser launch (via launch_browser or launch_browser_with_cdp depending on ENABLE_CDP_MODE), session validation through ks_client.pong, login via KuaishouLogin if needed, and concurrent task execution limited by MAX_CONCURRENCY_NUM.

Data Storage Options

The storage implementation in store/kuaishou/_store_impl.py supports multiple backends. By default, data writes to CSV using AsyncFileWriter.

To switch storage backends, modify the import in store/kuaishou/__init__.py:

• CSV/JSON: Default AsyncFileWriter implementation
• MongoDB: Use KuaishouMongoStoreImplement
• SQLite: SQLAlchemy-based implementation

Key storage functions include:

• kuaishou_store.update_kuaishou_video() for video metadata
• kuaishou_store.batch_update_ks_video_comments() for comment threads
• kuaishou_store.save_creator() for creator profiles

Summary

• MediaCrawler provides three Kuaishou crawling modes: search, detail, and creator, configured via CRAWLER_TYPE in config/ks_config.py.
• The architecture separates concerns between media_platform/kuaishou/core.py (orchestration), media_platform/kuaishou/client.py (API calls), and store/kuaishou/_store_impl.py (persistence).
• Authentication supports both QR-code and cookie-based methods through media_platform/kuaishou/login.py.
• URL parsing utilities in media_platform/kuaishou/help.py handle extraction of video and creator IDs from standard Kuaishou URLs.
• Execution entry point is main.py, with configurable concurrency and rate limiting via MAX_CONCURRENCY_NUM and CRAWLER_MAX_SLEEP_SEC.

Frequently Asked Questions

How do I extract video IDs from Kuaishou URLs?

Use the parse_video_info_from_url function from media_platform/kuaishou/help.py. This function accepts a standard Kuaishou URL like https://www.kuaishou.com/short-video/3x3zxz4mjrsc8ke and returns a VideoUrlInfo Pydantic model containing the extracted video_id and url_type fields defined in model/m_kuaishou.py.

What is the difference between CDP mode and standard Playwright mode?

CDP (Chrome DevTools Protocol) mode, enabled by setting ENABLE_CDP_MODE = True in config/ks_config.py, launches the browser using launch_browser_with_cdp in media_platform/kuaishou/core.py for advanced debugging and proxy capabilities. Standard mode uses Playwright's launch_browser method without CDP hooks.

How do I change the data storage format from CSV to MongoDB?

Modify the storage implementation in store/kuaishou/__init__.py to import KuaishouMongoStoreImplement instead of the default CSV writer. The storage layer exports standardized functions like update_kuaishou_video and batch_update_ks_video_comments that work regardless of the backend selected.

Why does the crawler require login, and what methods are supported?

Kuaishou requires authentication for accessing video details and creator data. The KuaishouLogin class in media_platform/kuaishou/login.py supports QR-code authentication (displayed in console) and cookie-based authentication (set via the COOKIES environment variable). The crawler validates the session using ks_client.pong before executing tasks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}