# Troubleshooting Common MediaCrawler Errors: A Complete Guide

> Troubleshoot MediaCrawler errors by fixing Playwright injection, validating login cookies with pong(), and ensuring proxy rotation via ProxyIpPool checks. Resolve runtime failures now.

- Repository: [程序员阿江-Relakkes/MediaCrawler](https://github.com/NanmiCoder/MediaCrawler)
- Tags: how-to-guide
- Published: 2026-06-28

---

**Resolve MediaCrawler runtime failures by verifying Playwright page injection, validating login cookies through the `pong()` method, and ensuring proxy rotation via `ProxyIpPool.is_current_proxy_expired()`.**

MediaCrawler is a multi-platform social media data collection framework built on Playwright for browser automation. When troubleshooting common MediaCrawler errors, understanding the interaction between the platform clients, proxy management, and session handling is essential for maintaining stable data extraction pipelines.

## Understanding MediaCrawler's Error-Prone Components

MediaCrawler's architecture separates concerns across three main subsystems that commonly generate runtime exceptions: the Playwright-based browser automation layer, the proxy rotation mechanism, and the JSON parsing pipeline. Most failures originate in `media_platform/*/client.py` files where these systems intersect.

### Entry Point and Client Initialization

The execution flow begins in [`main/main.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/main/main.py), which parses CLI arguments and instantiates concrete platform clients like `BaiduTieBaClient`. These clients inherit from [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py) and expect a validated Playwright `Page` object for browser operations.

Key methods in platform clients include:
- `_fetch_json_by_browser()` – Executes fetch requests inside Playwright and returns parsed JSON
- `pong()` – Validates login state by inspecting cookies through `convert_browser_context_cookies`
- `request()` – Performs synchronous HTTP calls wrapped with `tenacity` for retry logic

### The Proxy Subsystem

The [`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py) module maintains a rotating pool of HTTP proxies. Before each request, clients invoke `_refresh_proxy_if_expired()` to check `ProxyIpPool.is_current_proxy_expired()` and potentially fetch a fresh proxy via `get_or_refresh_proxy()`. When this check fails or returns an expired proxy, you encounter "account blocked" or 403 Forbidden errors.

## Fixing the "playwright_page is required" Error

This error occurs when instantiating a platform client without providing a valid Playwright `Page` object, which is mandatory for browser-based API requests.

**Root Cause:** The client constructor in [`media_platform/tieba/client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/tieba/client.py) (and similar platform clients) validates that `playwright_page` is not `None` before executing `_ensure_tieba_origin()` or similar origin checks.

**Resolution:** Launch Playwright and pass the page instance explicitly:

```python
from playwright.async_api import async_playwright
from media_platform.tieba.client import BaiduTieBaClient

async def run():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        page = await browser.new_page()
        await page.goto("https://tieba.baidu.com")
        client = BaiduTieBaClient(playwright_page=page)
        notes = await client.get_notes_by_keyword("python")
        print(notes)

import asyncio
asyncio.run(run())

```

*Reference:* Lines 76-80 in [`media_platform/tieba/client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/tieba/client.py) enforce the Playwright page requirement.

## Resolving Login and Session Invalidation Issues

When `pong()` returns `False` or logs "No valid login cookies found", the client cannot authenticate requests to the target platform.

**Root Cause:** Cookies have expired or were never initialized, often happening after a fresh browser launch without loading persisted session data.

**Resolution Steps:**

1. **Execute QR-code login** – Run the CLI with `--lt qrcode` to generate a login QR code and establish a valid session
2. **Persist cookies** – After authentication, call `update_cookies()` with the browser context:

```python

# After successful QR login

await client.update_cookies(page.context)

# Verify session validity

is_logged_in = await client.pong(page.context)
if not is_logged_in:
    utils.logger.error("Login required – aborting crawl")

```

*Reference:* The `update_cookies()` implementation at lines 59-73 in [`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py) handles cookie conversion via [`tools/utils.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/utils.py).

## Handling Proxy Expiration and Account Blocking

Errors like `Request failed, method: GET, status code: 403` or "account blocked" indicate IP-based rate limiting or blacklisting.

**Root Cause:** The current proxy expired or was flagged by the target platform, but `is_current_proxy_expired()` failed to trigger a refresh before the request.

**Resolution Methods:**

- **Automatic refresh** – Verify that `_refresh_proxy_if_expired()` is executing before requests (lines 18-35 in [`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py)). The client should automatically call this via the request wrapper.
- **Manual override** – Force a specific proxy for critical requests:

```python
await client.request("GET", "https://tieba.baidu.com/api", proxy="http://127.0.0.1:8080")

```

- **Adjust retry configuration** – The `@retry` decorator from `tenacity` defaults to `stop_after_attempt(3)`. Increase this in [`base/base_crawler.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/base/base_crawler.py) for unstable networks.

## Debugging JSON Decoding and API Parsing Failures

When encountering "Tieba PC API returned non-JSON" or similar parsing errors, the target likely returned an HTML error page instead of expected JSON.

**Root Cause:** Anti-scraping mechanisms intercepted the request, typically due to missing signatures or aggressive crawling speeds.

**Resolution Strategies:**

1. **Verify request signing** – Ensure `use_sign=True` is passed to PC API methods. The `_sign_pc_params()` method (lines 66-74 in [`client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/client.py)) generates required MD5 signatures.
2. **Implement throttling** – Respect `config.CRAWLER_MAX_SLEEP_SEC` or pass custom `crawl_interval` arguments to add delays between requests.
3. **Inspect raw responses** – Enable `return_ori_content=True` to capture the actual response body for debugging:

```python
raw = await client.request("GET", client._host + "/mo/q/search/multsearch", return_ori_content=True)
print("RAW response:", raw[:500])

```

## Practical Error-Resilient Implementation

This complete example demonstrates handling the three major failure modes (Playwright initialization, login validation, and proxy rotation) in a single script:

```python
import asyncio
from playwright.async_api import async_playwright
from media_platform.tieba.client import BaiduTieBaClient
from tools import utils

async def resilient_crawl():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        page = await browser.new_page()
        await page.goto("https://tieba.baidu.com")
        
        client = BaiduTieBaClient(playwright_page=page)
        
        # Validate login before any data extraction

        if not await client.pong(page.context):
            utils.logger.error("Invalid login state – run QR login first")
            return
            
        try:
            notes = await client.get_notes_by_keyword("AI", page=1, page_size=5)
            for note in notes:
                utils.logger.info(f"Processing: {note.title}")
                comments = await client.get_note_all_comments(note, max_count=20)
                utils.logger.info(f"Collected {len(comments)} comments")
        except Exception as exc:
            utils.logger.exception("Crawl failed – check proxy and signing")
            # Force proxy refresh on failure

            await client._refresh_proxy_if_expired()

asyncio.run(resilient_crawl())

```

## Summary

- **Playwright Page Requirement** – Always instantiate clients with a valid `Page` object from `async_playwright()` to avoid initialization errors.
- **Session Management** – Use `pong()` to verify login state and `update_cookies()` to persist sessions after QR-code authentication.
- **Proxy Health** – Monitor `ProxyIpPool.is_current_proxy_expired()` and manually refresh via `_refresh_proxy_if_expired()` when encountering 403 errors.
- **Request Signing** – Pass `use_sign=True` for PC APIs and utilize `_sign_pc_params()` for platforms requiring cryptographic signatures.
- **Debugging** – Enable `return_ori_content=True` in `client.request()` to inspect raw HTML responses when JSON parsing fails.

## Frequently Asked Questions

### How do I fix the "playwright_page is required" error in MediaCrawler?

This error occurs when you instantiate a platform client without passing a Playwright `Page` object. Launch a browser using `async_playwright()`, create a new page with `browser.new_page()`, and pass this page to the client constructor (e.g., `BaiduTieBaClient(playwright_page=page)`). The client validates this in lines 76-80 of [`media_platform/tieba/client.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/media_platform/tieba/client.py).

### Why does MediaCrawler show "account blocked" or 403 Forbidden errors?

These errors indicate your IP address has been flagged by the target platform. MediaCrawler handles this through [`proxy/proxy_ip_pool.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/proxy/proxy_ip_pool.py), which should automatically rotate proxies when `is_current_proxy_expired()` returns true. If blocking persists, manually call `_refresh_proxy_if_expired()` before requests or increase the retry attempts in the `tenacity` decorator configuration.

### How do I verify that my MediaCrawler login session is still valid?

Call the `pong()` method on your client instance, passing the current `BrowserContext` (typically `page.context`). This method uses `convert_browser_context_cookies` from [`tools/utils.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/tools/utils.py) to validate cookie freshness. If `pong()` returns `False`, execute a QR-code login via the CLI with `--lt qrcode` and then call `update_cookies(page.context)` to refresh the session.

### What causes JSON decoding errors in MediaCrawler and how do I fix them?

JSON decoding failures occur when the target API returns HTML error pages instead of JSON, usually due to anti-bot measures. Ensure you are passing `use_sign=True` for PC API endpoints to trigger `_sign_pc_params()` for proper request signing. Additionally, reduce crawl speed by adjusting `CRAWLER_MAX_SLEEP_SEC` in [`config/base_config.py`](https://github.com/NanmiCoder/MediaCrawler/blob/main/config/base_config.py), and use `return_ori_content=True` in your request calls to inspect the raw response for debugging.