How to Debug Login Failures (QR Code Scanning, Phone Verification) in MediaCrawler

Use a visible browser window, extend retry timeouts, and add URL logging to trace why MediaCrawler's QR code login fails when platforms redirect to phone verification.

MediaCrawler is an open-source multi-platform crawler that automates login via headless Playwright browsers. When QR code scanning triggers a secondary phone verification step, the default automation logic times out because it only polls for a specific session cookie. This guide walks through the exact source code paths and debugging techniques to diagnose and fix these failures.

Understanding the QR Code Login Flow

The login process spans three core components in the Zhihu implementation (the pattern applies to other platforms with minor variations):

Component	Purpose	Source Location
ZhiHuLogin.check_login_state	Polls browser cookies for the z_c0 session indicator	media_platform/zhihu/login.py lines 51-64
ZhiHuLogin.login_by_qrcode	Orchestrates QR fetching, display, and retry loop	media_platform/zhihu/login.py lines 81-110
find_qrcode_img_from_canvas	Extracts base64 QR image from page canvas	tools/crawler_util.py lines 66-86

The retry decorator on check_login_state uses 600 attempts with 1-second fixed intervals (10 minutes total). If the platform redirects to phone verification after QR scanning, the z_c0 cookie never appears and the loop exhausts all retries.

Common Failure Points and Diagnostic Steps

1. QR Image Extraction Fails (Empty Base64 String)

When find_qrcode_img_from_canvas returns an empty string, login aborts immediately with the log message:

[ZhiHu.login_by_qrcode] login failed , have not found qrcode please check ....

Debugging actions:

• Verify the canvas selector canvas.Qrcode-qrcode matches the live DOM (platforms update selectors periodically)
• Add await page.wait_for_load_state('networkidle') before QR extraction to ensure full page render
• Check that the login page URL hasn't changed (redirects to /signin vs /login)

2. Phone Verification Redirect Blocks Cookie Detection

Platforms increasingly require SMS verification after QR scanning for new devices. The current implementation at line 59 only checks for z_c0:

utils.logger.info(f"[ZhiHuLogin] waiting login success loop ..., loop time {loop_time}")
if "z_c0" in cookies_str:  # hardcoded cookie name

    return True

Debugging actions:

• Extend timeout window: Modify the decorator on line 51 from stop_after_attempt(600) to stop_after_attempt(1800) for 30-minute tolerance
• Log current URL: Insert debug output to detect verification page redirects
• Switch to exponential backoff: Replace wait_fixed(1) with wait_exponential(multiplier=1, min=1, max=30) to reduce request frequency during long waits

3. RetryError on Maximum Attempts Exhaustion

The exception handler at lines 106-108 catches RetryError and logs:

[ZhiHu.login_by_qrcode] Login zhihu failed by qrcode login method ...

At this point, the browser context remains open but unauthenticated. Capture state immediately: dump cookies, screenshot the page, and log the final URL.

Debugging Checklist for Phone Verification Scenarios

1. Confirm LOGIN_TYPE configuration — config/base_config.py line 28 must specify "qrcode" not "phone" (the latter routes to unimplemented login_by_mobile)

2. Launch with visible browser — edit tools/cdp_browser.py or tools/browser_launcher.py to set headless=False

3. Validate cookie name stability — after manual QR login, inspect await browser_context.cookies() to verify z_c0 hasn't been renamed (e.g., to z_c0_v2)

4. Monitor network idle state — phone verification pages often load additional XHR endpoints; watch for request patterns that indicate verification UI presence

Code Examples: Extended Timeout with Debug Logging

# Extended retry configuration with URL debugging

from tenacity import stop_after_attempt, wait_exponential, retry
from media_platform.zhihu.login import ZhiHuLogin
from tools import utils

# Store original method for reference

_original_check = ZhiHuLogin.check_login_state

# Create debug-wrapped version with 30-minute timeout

@retry(
    stop=stop_after_attempt(1800),
    wait=wait_exponential(multiplier=1, min=1, max=30),
    retry_error_callback=lambda retry_state: False
)
async def check_login_state_debug(self):
    """Enhanced check with URL logging for phone verification detection."""
    current_url = await self.context_page.url
    utils.logger.debug(f"[debug] Polling URL: {current_url}")
    
    # Detect phone verification redirect patterns

    if "verify" in current_url or "bind-phone" in current_url:
        utils.logger.warning(f"[debug] Phone verification detected at: {current_url}")
        # Optionally: pause here for manual intervention

        # await asyncio.sleep(60)  # wait for manual SMS entry

    
    # Original cookie check logic

    cookies_str, cookie_dict = utils.convert_cookies(await self.browser_context.cookies())
    return "z_c0" in cookies_str

# Monkey-patch for debugging session

ZhiHuLogin.check_login_state = check_login_state_debug

# Now instantiate and run

# login = ZhiHuLogin(login_type="qrcode", ...)

# Minimal reproduction with visible browser for interactive debugging

import asyncio
from playwright.async_api import async_playwright
from config.base_config import LOGIN_TYPE

async def debug_visible_login():
    async with async_playwright() as p:
        # Force visible browser for manual observation

        browser = await p.chromium.launch(
            headless=False,
            args=["--disable-blink-features=AutomationControlled"]
        )
        context = await browser.new_context(viewport={"width": 1280, "height": 720})
        page = await context.new_page()
        
        # Navigate and wait for QR render

        await page.goto("https://www.zhihu.com/signin")
        await page.wait_for_load_state("networkidle")
        
        print("Browser visible. Scan QR when displayed, or diagnose if missing.")
        print("Press Ctrl+C after observing behavior.")
        
        # Keep alive for inspection

        while True:
            await asyncio.sleep(1)
            url = await page.url()
            if "signin" not in url:
                print(f"Navigation detected: {url}")
                break

asyncio.run(debug_visible_login())

Summary

• Source of truth: media_platform/zhihu/login.py contains the retry logic (line 51 decorator) and cookie detection (line 59 z_c0 check)
• Primary failure mode: Phone verification redirects prevent cookie appearance, exhausting fixed 10-minute retry window
• Key fixes: Extend stop_after_attempt, add URL logging, use headless=False for visual confirmation
• Cookie validation: Verify z_c0 name hasn't changed; dump browser_context.cookies() after any successful manual login to confirm

Frequently Asked Questions

Why does MediaCrawler hang indefinitely on QR code login?

The default @retry decorator limits attempts to 600 with 1-second fixed waits. If phone verification appears, the z_c0 cookie never materializes and the loop times out. Increase stop_after_attempt or switch to wait_exponential for longer tolerance windows.

Where is the QR code selector defined if the platform changes its DOM?

Line 84 of media_platform/zhihu/login.py sets qrcode_img_selector = "canvas.Qrcode-qrcode". Update this string if inspection reveals a different class or element type (e.g., img instead of canvas).

Can I manually complete phone verification while MediaCrawler runs?

Yes. Launch with headless=False via tools/cdp_browser.py, extend the retry timeout substantially, and watch for verification page detection in your debug logs. The polling loop will continue checking for z_c0 while you complete SMS steps manually.

How do I verify the session cookie name hasn't changed?

After any successful login (manual or automated), execute:

cookies = await browser_context.cookies()
print([c["name"] for c in cookies if "z_c" in c["name"]])

Compare output against the hardcoded z_c0 string in check_login_state.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}