How to Handle Verification Challenges in MediaCrawler: Slider Captchas and SMS Verification

MediaCrawler handles verification challenges by automating slider captcha solving through image processing and human-like mouse movements, while SMS verification is managed via a FastAPI webhook that caches six-digit codes for automatic form submission.

The NanmiCoder/MediaCrawler repository provides a robust Python framework for scraping Chinese social media platforms like Douyin (TikTok China). To handle verification challenges in MediaCrawler effectively, you must understand its dual approach: programmatic image analysis for slider captchas and asynchronous webhook processing for SMS verification codes.

Understanding MediaCrawler's Verification Challenge Architecture

MediaCrawler faces two primary verification mechanisms common to Chinese social media platforms. The first involves slider captchas that require image recognition and simulated human interaction. The second involves SMS verification codes sent to mobile numbers during login flows. The architecture separates these concerns into dedicated utility modules while maintaining a unified login orchestration layer in the platform-specific login handlers.

Handling Slider Captchas (Image Verification)

Slider captchas appear when Douyin detects automated behavior. The crawler solves these challenges by analyzing the visual gap between images and simulating realistic mouse movements.

Detecting the Captcha Challenge

The verification detection begins in media_platform/douyin/login.py. The check_page_display_slider method waits for the CSS selector #captcha-verify-image to appear on the page. If the element is not found within 30 seconds, the function returns False, indicating no verification is required. When detected, the method initiates the solving workflow by calling move_slider with configurable difficulty parameters.

# From media_platform/douyin/login.py (lines 71-80)

async def check_page_display_slider(self, move_step: int = 5, slider_level: str = "easy"):
    # Wait for captcha element

    # If found, proceed to move_slider

    await self.move_slider(move_step, slider_level)

Image Processing and Gap Detection

Once detected, the crawler extracts the background and gap images from the DOM. In move_slider (lines 223-236), the code retrieves the src attributes of both the background image and the gap image using the XPath //*[@id="captcha_container"]/div/div[2]/img[2]. These URLs are passed to the Slide class in tools/slider_util.py.

The Slide.discern() method executes a three-step image processing pipeline:

1. Noise reduction: clear_white removes white noise from the images
2. Edge detection: cv2.Canny identifies edges in the processed images
3. Template matching: cv2.matchTemplate locates the gap position by comparing the gap image against the background

The method returns the x-coordinate of the matched gap, which determines the required slide distance in pixels.

# From tools/slider_util.py (lines 30-43)

def discern(self):
    # Download and resize images to temp_image/

    # Apply edge detection and template matching

    # Return the x-coordinate offset

Generating Human-Like Movement Tracks

After calculating the distance, MediaCrawler generates a movement track that mimics human acceleration and deceleration patterns. The utils.get_tracks(distance, slider_level) function in tools/slider_util.py (lines 45-84) supports two difficulty levels:

• Easy: Uses get_track_simple to create basic acceleration/deceleration curves
• Hard: Leverages the easing module to produce sophisticated exponential ease-out curves that replicate genuine user behavior more accurately

The hard level implementation in tools/easing.py significantly reduces detection rates by avoiding linear or predictable movement patterns.

Executing the Slider Drag

With the track calculated, DouYinLogin.move_slider grabs the gap element using Playwright, computes its center coordinates, and iterates through the generated track points. The method uses mouse.move to drag the slider across each increment, then releases the mouse button to complete the verification.

# From media_platform/douyin/login.py (lines 56-64)

async def move_slider(self, move_step: int, slider_level: str):
    gap_element = await self.context_page.locator('xpath=...').first
    # Calculate center and execute track

    for track in tracks:
        await self.context_page.mouse.move(track_x, track_y)

Retry Logic and Failure Handling

If the platform responds with "操作过慢" (operation too slow) or "提示重新操作" (please retry), the code automatically clicks the refresh link and retries up to 20 times. This resilience is implemented in media_platform/douyin/login.py (lines 94-108), ensuring the crawler can recover from transient validation failures without manual intervention.

Handling SMS Verification Challenges

SMS verification requires external coordination to receive codes sent to mobile numbers during login.

Receiving SMS via FastAPI Webhook

MediaCrawler runs a lightweight FastAPI server in recv_sms.py (lines 55-63) that accepts POST payloads containing SMS text from external SMS forwarding services. This decouples the crawler from direct mobile network access while enabling real-time code receipt.

# From recv_sms.py

@app.post("/")
async def receive_sms_notification(request: Request):
    # Receive JSON payload with platform, number, and content

    # Extract and cache the verification code

Extracting and Caching Verification Codes

The extract_verification_code function uses the regular expression \b[0-9]{6}\b to isolate the first six-digit token from the message text. The extracted code is stored in the configured cache (Redis by default) with a 3-minute TTL under the key format platform_number (e.g., dy_13800138000). This implementation appears in recv_sms.py (lines 46-53 and 73-78).

Polling and Auto-Submission

During mobile login (DouYinLogin.login_by_mobile), the crawler polls the cache every second for up to 2 minutes (max_get_sms_code_time). Once the code appears in the cache, the automation fills the verification input field and clicks the submit button. A TODO comment in the source code (line 68) notes that developers should validate the submission result, though the current implementation trusts the cached value.

# From media_platform/douyin/login.py (lines 53-68)

while time.time() - start_time < max_get_sms_code_time:
    sms_code = await self.cache.get(f"dy_{self.login_phone}")
    if sms_code:
        await self.context_page.locator("input...").fill(sms_code)
        await self.context_page.locator("button...").click()

Configuration and Customization

Several configuration options control verification challenge handling:

• Login Type: Set config.LOGIN_TYPE to qrcode, phone, or cookie in DouYinLogin.__init__ to determine whether SMS or slider-only paths are activated
• CDP Mode: ENABLE_CDP_MODE in config/base_config.py toggles between Chrome DevTools Protocol and full Playwright mode, affecting image capture reliability for slider detection
• Slider Difficulty: Pass slider_level="hard" to check_page_display_slider for platforms with stricter bot detection

Code Examples

Triggering Slider Handling Manually

For testing or custom workflows, you can force the slider verification path:

from tools import utils
from media_platform.douyin.login import DouYinLogin
from playwright.async_api import async_playwright

async def demo_slider():
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=False)
        context = await browser.new_context()
        page = await context.new_page()
        await page.goto("https://www.douyin.com/")
        
        login = DouYinLogin(
            login_type="phone",
            browser_context=context,
            context_page=page,
            login_phone="13800138000",
        )
        # Force the slider check with hard difficulty

        await login.check_page_display_slider(move_step=5, slider_level="hard")

Simulating SMS Webhook Delivery

Test the SMS reception flow by posting to the local endpoint:

import httpx

payload = {
    "platform": "douyin",
    "current_number": "13800138000",
    "from_number": "1069421xxx134",
    "sms_content": "【抖音】您的验证码是: 123456，3分钟内有效。",
    "timestamp": "1686720601614"
}

response = httpx.post("http://127.0.0.1:8000/", json=payload)
print(response.json())  # {"status": "ok"}

After this webhook, login_by_mobile will automatically retrieve code 123456 from the cache and submit it.

Customizing Motion Curves

Switch to the hard difficulty profile for sophisticated anti-detection:

from tools import utils

distance = 120  # Pixels from Slide.discern()

hard_tracks = utils.get_tracks(distance, level="hard")

# Returns a list of integers with exponential ease-out curve

Summary

• Slider captchas are handled through computer vision (OpenCV template matching) and human-like mouse trajectory generation in tools/slider_util.py
• SMS verification relies on an external FastAPI webhook (recv_sms.py) that extracts six-digit codes and caches them for polling by the login handler
• Retry mechanisms include up to 20 attempts for slider failures and 2-minute polling windows for SMS codes
• Configuration via LOGIN_TYPE and ENABLE_CDP_MODE allows adaptation to different platform requirements and detection sensitivities
• Modular architecture separates image processing, track generation, and verification receipt into distinct, testable components

Frequently Asked Questions

How does MediaCrawler detect when a slider captcha is present?

MediaCrawler uses check_page_display_slider in media_platform/douyin/login.py to poll for the CSS selector #captcha-verify-image every 30 seconds. If the element appears, the method immediately initiates the image capture and solving sequence. If the timeout elapses without detection, the crawler proceeds with normal login flows assuming no verification is required.

What makes the "hard" slider level more effective than the "easy" level?

The hard level uses the easing module in tools/easing.py to generate exponential ease-out curves that replicate genuine human mouse movements, including micro-adjustments and variable acceleration. The easy level uses simple linear interpolation with basic acceleration, which detection systems can identify as robotic movement patterns. For production deployments against strict anti-bot measures, the hard level significantly reduces ban rates.

Can MediaCrawler handle SMS verification without manual intervention?

Yes, when configured for phone login (LOGIN_TYPE="phone"), MediaCrawler automates SMS handling through the FastAPI webhook in recv_sms.py. The webhook receives the SMS content, extracts the six-digit code using regex \b[0-9]{6}\b, and stores it in Redis with a 3-minute TTL. The login_by_mobile method polls this cache every second for up to 2 minutes, automatically filling and submitting the code when available.

Where does MediaCrawler store the temporary images during slider processing?

The Slide class in tools/slider_util.py downloads the background and gap images to the temp_image/ directory during the discern() execution (lines 39-52). These files are processed using OpenCV for edge detection and template matching, then the gap offset is calculated and returned to the login handler for track generation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}