How to Use Cookie-Based Login in MediaCrawler: Complete Guide

MediaCrawler supports cookie-based login via the --cookies CLI flag or COOKIES config setting, allowing you to bypass QR-code and phone verification by injecting pre-authenticated cookies directly into the Playwright browser context.

The NanmiCoder/MediaCrawler repository provides a robust open-source framework for crawling multiple Chinese social media platforms. When you already possess valid session cookies, you can eliminate interactive authentication steps entirely and proceed directly to data extraction.

How Cookie-Based Login Works in MediaCrawler

The cookie authentication system follows a standardized four-step pipeline across all supported platforms (Zhihu, XiaoHongShu, Douyin, etc.).

The Authentication Flow

1. Set the login type to cookie via LOGIN_TYPE in config/base_config.py or the --lt CLI argument.
2. Provide the raw cookie string using the --cookies parameter or the COOKIES configuration variable.
3. Convert and inject the cookie string into Playwright using utils.convert_str_cookie_to_dict from tools/crawler_util.py, which parses the semicolon-separated values into a dictionary for browser_context.add_cookies.
4. Proceed to crawling immediately through the platform-specific login_by_cookies() method, which skips QR-code scans and phone verification by marking the session as authenticated.

The conversion utility in tools/crawler_util.py (lines 59-74) handles both formatting requirements: returning a dictionary for Playwright context injection and maintaining the semicolon-separated string format for HTTP headers.

Configuration Methods for Cookie Login

MediaCrawler exposes cookie authentication through two primary interfaces.

Command-Line Interface

Pass the login type and cookie string directly when invoking the crawler:

python -m main \
  --platform zhihu \
  --lt cookie \
  --cookies "z_c0=YOUR_ZHIHU_COOKIE; d_c0=ANOTHER_COOKIE"

The cmd_arg/arg.py module parses these arguments and stores them in the global configuration object. The --lt flag accepts cookie as a valid option alongside qrcode and phone.

Base Configuration File

For persistent settings, edit config/base_config.py to set:

LOGIN_TYPE = "cookie"
COOKIES = "web_session=YOUR_SESSION_ID; other_cookie=value"

When using this method, you can omit the --cookies flag from the command line entirely. The AbstractLogin class reads these values during initialization in base/base_crawler.py.

Platform-Specific Cookie Login Examples

Each platform implements login_by_cookies() with domain-specific handling.

Zhihu

For Zhihu crawling, provide the z_c0 and d_c0 cookies:

python -m main \
  --platform zhihu \
  --lt cookie \
  --cookies "z_c0=abc123; d_c0=def456"

In media_platform/zhihu/login.py (lines 73-82), the ZhiHuLogin.login_by_cookies() method iterates over the parsed cookie dictionary and injects each name-value pair into the Playwright context with the domain set to .zhihu.com.

XiaoHongShu (XHS)

XiaoHongShu authentication typically requires only the web_session cookie:

python -m main \
  --platform xhs \
  --lt cookie \
  --cookies "web_session=YOUR_XHS_SESSION"

The implementation in media_platform/xhs/login.py (lines 71-78) dynamically selects the domain based on config.XHS_INTERNATIONAL, using either .xiaohongshu.com or .rednote.com (the international domain).

Douyin

Douyin requires session identifiers and a login status flag:

python -m main \
  --platform dy \
  --lt cookie \
  --cookies "sessionid=YOUR_SESSION; LOGIN_STATUS=1"

According to media_platform/douyin/login.py (lines 66-74), the DouYinLogin.login_by_cookies() method adds cookies to the .douyin.com domain. The crawler checks for LOGIN_STATUS=1 in check_login_state() to confirm successful authentication before commencing data extraction.

Programmatic API Usage

If embedding MediaCrawler in custom Python scripts, instantiate the login class directly:

from media_platform.zhihu.login import ZhiHuLogin
from playwright.async_api import async_playwright

async def run():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        context = await browser.new_context()
        page = await context.new_page()
        
        login = ZhiHuLogin(
            login_type="cookie",
            browser_context=context,
            context_page=page,
            cookie_str="z_c0=abc123; d_c0=def456"
        )
        await login.begin()  # Injects cookies and starts crawling

The convert_str_cookie_to_dict utility in tools/crawler_util.py handles the string-to-dictionary conversion automatically within the begin() method.

Summary

• Cookie-based login in MediaCrawler bypasses interactive authentication by injecting pre-validated session cookies into the Playwright browser context.
• Configuration requires setting LOGIN_TYPE="cookie" and providing a semicolon-separated cookie string via --cookies or config/base_config.py.
• Core utilities reside in tools/crawler_util.py, specifically convert_str_cookie_to_dict() for parsing and convert_browser_context_cookies() for refreshing sessions.
• Platform implementations handle domain-specific injection in their respective login_by_cookies() methods (e.g., media_platform/zhihu/login.py, media_platform/xhs/login.py).
• Format requirement follows standard HTTP cookie header syntax: key=value; key2=value2.

Frequently Asked Questions

What cookie format does MediaCrawler require?

MediaCrawler expects a standard HTTP cookie header string where key-value pairs are separated by semicolons (e.g., "sessionid=abc123; user=xyz"). The convert_str_cookie_to_dict function in tools/crawler_util.py parses this into a dictionary for Playwright's add_cookies() method.

Can I use cookie login for all supported platforms?

Yes. All major platforms (Zhihu, XiaoHongShu, Douyin, Kuaishou) implement the login_by_cookies() method in their respective media_platform/{platform}/login.py files. Each implementation handles domain-specific cookie injection automatically.

How do I extract cookies from my browser for use with MediaCrawler?

Use browser developer tools (Network tab or Application tab) to copy the cookie header from an authenticated request to the target platform. Alternatively, use browser extensions that export cookies in the standard HTTP header format, then paste the string into the --cookies argument or COOKIES config variable.

Does MediaCrawler refresh cookies automatically during crawling?

The framework includes convert_browser_context_cookies in tools/crawler_util.py (lines 48-56) to retrieve updated cookies from the Playwright context during execution. Some platform implementations use this to refresh the internal header dictionary if the session updates during long-running crawls.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}