Troubleshooting Common MediaCrawler Errors: A Complete Guide

Resolve MediaCrawler runtime failures by verifying Playwright page injection, validating login cookies through the pong() method, and ensuring proxy rotation via ProxyIpPool.is_current_proxy_expired().

MediaCrawler is a multi-platform social media data collection framework built on Playwright for browser automation. When troubleshooting common MediaCrawler errors, understanding the interaction between the platform clients, proxy management, and session handling is essential for maintaining stable data extraction pipelines.

Understanding MediaCrawler's Error-Prone Components

MediaCrawler's architecture separates concerns across three main subsystems that commonly generate runtime exceptions: the Playwright-based browser automation layer, the proxy rotation mechanism, and the JSON parsing pipeline. Most failures originate in media_platform/*/client.py files where these systems intersect.

Entry Point and Client Initialization

The execution flow begins in main/main.py, which parses CLI arguments and instantiates concrete platform clients like BaiduTieBaClient. These clients inherit from base/base_crawler.py and expect a validated Playwright Page object for browser operations.

Key methods in platform clients include:

• _fetch_json_by_browser() – Executes fetch requests inside Playwright and returns parsed JSON
• pong() – Validates login state by inspecting cookies through convert_browser_context_cookies
• request() – Performs synchronous HTTP calls wrapped with tenacity for retry logic

The Proxy Subsystem

The proxy/proxy_ip_pool.py module maintains a rotating pool of HTTP proxies. Before each request, clients invoke _refresh_proxy_if_expired() to check ProxyIpPool.is_current_proxy_expired() and potentially fetch a fresh proxy via get_or_refresh_proxy(). When this check fails or returns an expired proxy, you encounter "account blocked" or 403 Forbidden errors.

Fixing the "playwright_page is required" Error

This error occurs when instantiating a platform client without providing a valid Playwright Page object, which is mandatory for browser-based API requests.

Root Cause: The client constructor in media_platform/tieba/client.py (and similar platform clients) validates that playwright_page is not None before executing _ensure_tieba_origin() or similar origin checks.

Resolution: Launch Playwright and pass the page instance explicitly:

from playwright.async_api import async_playwright
from media_platform.tieba.client import BaiduTieBaClient

async def run():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        page = await browser.new_page()
        await page.goto("https://tieba.baidu.com")
        client = BaiduTieBaClient(playwright_page=page)
        notes = await client.get_notes_by_keyword("python")
        print(notes)

import asyncio
asyncio.run(run())

Reference: Lines 76-80 in media_platform/tieba/client.py enforce the Playwright page requirement.

Resolving Login and Session Invalidation Issues

When pong() returns False or logs "No valid login cookies found", the client cannot authenticate requests to the target platform.

Root Cause: Cookies have expired or were never initialized, often happening after a fresh browser launch without loading persisted session data.

Resolution Steps:

1. Execute QR-code login – Run the CLI with --lt qrcode to generate a login QR code and establish a valid session
2. Persist cookies – After authentication, call update_cookies() with the browser context:

# After successful QR login

await client.update_cookies(page.context)

# Verify session validity

is_logged_in = await client.pong(page.context)
if not is_logged_in:
    utils.logger.error("Login required – aborting crawl")

Reference: The update_cookies() implementation at lines 59-73 in client.py handles cookie conversion via tools/utils.py.

Handling Proxy Expiration and Account Blocking

Errors like Request failed, method: GET, status code: 403 or "account blocked" indicate IP-based rate limiting or blacklisting.

Root Cause: The current proxy expired or was flagged by the target platform, but is_current_proxy_expired() failed to trigger a refresh before the request.

Resolution Methods:

• Automatic refresh – Verify that _refresh_proxy_if_expired() is executing before requests (lines 18-35 in client.py). The client should automatically call this via the request wrapper.
• Manual override – Force a specific proxy for critical requests:

await client.request("GET", "https://tieba.baidu.com/api", proxy="http://127.0.0.1:8080")

• Adjust retry configuration – The @retry decorator from tenacity defaults to stop_after_attempt(3). Increase this in base/base_crawler.py for unstable networks.

Debugging JSON Decoding and API Parsing Failures

When encountering "Tieba PC API returned non-JSON" or similar parsing errors, the target likely returned an HTML error page instead of expected JSON.

Root Cause: Anti-scraping mechanisms intercepted the request, typically due to missing signatures or aggressive crawling speeds.

Resolution Strategies:

1. Verify request signing – Ensure use_sign=True is passed to PC API methods. The _sign_pc_params() method (lines 66-74 in client.py) generates required MD5 signatures.
2. Implement throttling – Respect config.CRAWLER_MAX_SLEEP_SEC or pass custom crawl_interval arguments to add delays between requests.
3. Inspect raw responses – Enable return_ori_content=True to capture the actual response body for debugging:

raw = await client.request("GET", client._host + "/mo/q/search/multsearch", return_ori_content=True)
print("RAW response:", raw[:500])

Practical Error-Resilient Implementation

This complete example demonstrates handling the three major failure modes (Playwright initialization, login validation, and proxy rotation) in a single script:

import asyncio
from playwright.async_api import async_playwright
from media_platform.tieba.client import BaiduTieBaClient
from tools import utils

async def resilient_crawl():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        page = await browser.new_page()
        await page.goto("https://tieba.baidu.com")
        
        client = BaiduTieBaClient(playwright_page=page)
        
        # Validate login before any data extraction

        if not await client.pong(page.context):
            utils.logger.error("Invalid login state – run QR login first")
            return
            
        try:
            notes = await client.get_notes_by_keyword("AI", page=1, page_size=5)
            for note in notes:
                utils.logger.info(f"Processing: {note.title}")
                comments = await client.get_note_all_comments(note, max_count=20)
                utils.logger.info(f"Collected {len(comments)} comments")
        except Exception as exc:
            utils.logger.exception("Crawl failed – check proxy and signing")
            # Force proxy refresh on failure

            await client._refresh_proxy_if_expired()

asyncio.run(resilient_crawl())

Summary

• Playwright Page Requirement – Always instantiate clients with a valid Page object from async_playwright() to avoid initialization errors.
• Session Management – Use pong() to verify login state and update_cookies() to persist sessions after QR-code authentication.
• Proxy Health – Monitor ProxyIpPool.is_current_proxy_expired() and manually refresh via _refresh_proxy_if_expired() when encountering 403 errors.
• Request Signing – Pass use_sign=True for PC APIs and utilize _sign_pc_params() for platforms requiring cryptographic signatures.
• Debugging – Enable return_ori_content=True in client.request() to inspect raw HTML responses when JSON parsing fails.

Frequently Asked Questions

How do I fix the "playwright_page is required" error in MediaCrawler?

This error occurs when you instantiate a platform client without passing a Playwright Page object. Launch a browser using async_playwright(), create a new page with browser.new_page(), and pass this page to the client constructor (e.g., BaiduTieBaClient(playwright_page=page)). The client validates this in lines 76-80 of media_platform/tieba/client.py.

Why does MediaCrawler show "account blocked" or 403 Forbidden errors?

These errors indicate your IP address has been flagged by the target platform. MediaCrawler handles this through proxy/proxy_ip_pool.py, which should automatically rotate proxies when is_current_proxy_expired() returns true. If blocking persists, manually call _refresh_proxy_if_expired() before requests or increase the retry attempts in the tenacity decorator configuration.

How do I verify that my MediaCrawler login session is still valid?

Call the pong() method on your client instance, passing the current BrowserContext (typically page.context). This method uses convert_browser_context_cookies from tools/utils.py to validate cookie freshness. If pong() returns False, execute a QR-code login via the CLI with --lt qrcode and then call update_cookies(page.context) to refresh the session.

What causes JSON decoding errors in MediaCrawler and how do I fix them?

JSON decoding failures occur when the target API returns HTML error pages instead of JSON, usually due to anti-bot measures. Ensure you are passing use_sign=True for PC API endpoints to trigger _sign_pc_params() for proper request signing. Additionally, reduce crawl speed by adjusting CRAWLER_MAX_SLEEP_SEC in config/base_config.py, and use return_ori_content=True in your request calls to inspect the raw response for debugging.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}