Handling Dynamic Content Loading with Scrapling's DynamicFetcher and Playwright

Use Scrapling's DynamicFetcher to control exactly when a browser-rendered page is considered "ready" by leveraging Playwright's DOM stability, network idle detection, and custom selector waits.

Handling dynamic content loading with Scrapling's DynamicFetcher and Playwright gives you deterministic control over JavaScript-heavy websites. The DynamicFetcher class in the D4Vinci/Scrapling repository wraps Playwright's browser automation capabilities, exposing fine-grained options to wait for specific DOM states, network conditions, or custom page actions before returning a Response object.

What is DynamicFetcher?

DynamicFetcher is a high-level interface that launches a Chromium browser (via Playwright) to fetch pages requiring full JavaScript execution. Unlike static fetchers, it allows you to specify exactly when the fetch operation completes—whether that's after the DOM finishes loading, when network activity ceases, or when a specific CSS selector becomes visible.

The fetcher is implemented in scrapling/fetchers/chrome.py and delegates actual browser management to session objects defined in scrapling/engines/_browsers/_controllers.py.

Architecture and Implementation

Public Entry Points

The DynamicFetcher class in scrapling/fetchers/chrome.py provides two primary methods:

• DynamicFetcher.fetch(url, **options) – Synchronous fetching
• DynamicFetcher.async_fetch(url, **options) – Asynchronous fetching

These methods prepare a unified selector_config (lines 39-46 of the file), then delegate to session contexts:

with DynamicSession(**kwargs) as session:
    return session.fetch(url)

And for async:

async with AsyncDynamicSession(**kwargs) as session:
    return await session.fetch(url)

Session Managers

The core browser logic lives in scrapling/engines/_browsers/_controllers.py, which defines:

• DynamicSession – Synchronous session managing Playwright's browser context, page pooling, and retry logic
• AsyncDynamicSession – Asynchronous equivalent using Playwright's async API

Session Lifecycle

1. Construction – Validates options via _validate and stores configuration
2. start() – Launches Chromium or connects to an existing CDP endpoint; decides between persistent context or fresh context per proxy
3. fetch() – Executes the retrieval workflow:
   • Generates convincing request headers and optional referer
   • Enters retry loop (self._config.retries)
   • Acquires page from pool via _page_generator
   • Calls page.goto(url, referer=...)
   • Executes optional page_action callback
   • Waits for wait_selector and/or load_dom / network_idle via _wait_for_page_stability
   • Sleeps for final wait period
   • Builds Response via ResponseFactory.from_playwright_response

The async version mirrors this flow using await and async with syntax.

Supporting Utilities

• scrapling/engines/toolbelt/convertor.py – Converts raw Playwright responses into Scrapling's Response objects containing html_content, status_code, headers, and the parsed Selector
• scrapling/engines/toolbelt/fingerprints.py – Generates realistic referer headers when google_search=True
• scrapling/engines/_browsers/_validators.py – Type-checks and validates the extensive set of fetch parameters

Key Configuration Options

When handling dynamic content, these parameters in DynamicFetcher provide precise control:

• headless – Run browser hidden (True, default) or visible for debugging
• load_dom – Wait for DOMContentLoaded event before returning
• network_idle – Ensure no network activity for ≥500ms (indicates JavaScript finished fetching data)
• wait_selector / wait_selector_state – Block until specific element reaches state (attached, visible, hidden, detached)
• page_action – User-supplied callback receiving the Playwright page object for clicks, scrolls, form fills, or infinite scroll triggers
• proxy – Route traffic through specified proxy with automatic retry on failure
• timeout – Maximum milliseconds to wait for navigation
• wait – Final sleep duration after page stabilizes before capturing HTML

Practical Code Examples

Synchronous Fetching

from scrapling import DynamicFetcher

# Basic fetch – wait until DOM is fully loaded and network is idle

response = DynamicFetcher.fetch(
    "https://news.ycombinator.com",
    headless=True,
    load_dom=True,
    network_idle=True,
    timeout=20000,          # ms

    wait=500,               # extra pause after page stabilises

)

print("Status:", response.status_code)
print("HTML length:", len(response.html_content))

Asynchronous Fetching

import asyncio
from scrapling import DynamicFetcher

async def main():
    response = await DynamicFetcher.async_fetch(
        "https://example.com",
        headless=False,               # visible browser for debugging

        wait_selector="div#main",    # wait for a specific element

        wait_selector_state="visible",
        page_action=lambda page: page.click("button#load-more"),
        timeout=30000,
    )
    print("Fetched with async:", response.status_code)

asyncio.run(main())

Advanced Usage with Proxies and Page Actions

from scrapling import DynamicFetcher

def scroll_to_bottom(page):
    # Scroll slowly to trigger infinite scroll loading

    page.evaluate(
        """() => {
            return new Promise(resolve => {
                let totalHeight = 0;
                const distance = 100;
                const timer = setInterval(() => {
                    const scrollHeight = document.body.scrollHeight;
                    window.scrollBy(0, distance);
                    totalHeight += distance;
                    if (totalHeight >= scrollHeight) {
                        clearInterval(timer);
                        resolve();
                    }
                }, 200);
            });
        }"""
    )

response = DynamicFetcher.fetch(
    "https://infinite-scroll-site.com",
    proxy="http://user:pass@proxy.example:3128",
    page_action=scroll_to_bottom,
    load_dom=False,                # we only need the final DOM after scrolling

    timeout=60000,
)
print(response.selector.css("article.title").all())

Command-Line Interface

Scrapling exposes the dynamic fetcher through the CLI (scrapling-cli):

scrapling fetch \
    --fetcher DynamicFetcher \
    --url https://example.com \
    --headless false \
    --wait-selector "#content" \
    --output out.html

The CLI wiring lives in scrapling/cli.py (see the @extract.command decorator around line 575). It forwards all options directly to the fetcher class.

Summary

• DynamicFetcher provides browser-level fetching with deterministic controls for when a page is considered "ready"
• Synchronous (fetch) and asynchronous (async_fetch) entry points in scrapling/fetchers/chrome.py delegate to DynamicSession and AsyncDynamicSession
• Core session logic in scrapling/engines/_browsers/_controllers.py handles browser lifecycle, page pooling, retries, and proxy rotation
• Key waiting strategies include load_dom, network_idle, wait_selector, and custom page_action callbacks
• All responses are converted to Scrapling's standard Response objects via ResponseFactory in scrapling/engines/toolbelt/convertor.py

Frequently Asked Questions

How does DynamicFetcher differ from Scrapling's static fetchers?

Static fetchers retrieve raw HTML using HTTP clients like httpx or requests, which cannot execute JavaScript. DynamicFetcher launches a real Chromium browser via Playwright, allowing it to render SPAs, execute AJAX calls, and wait for dynamically injected content before returning the final HTML.

What is the difference between load_dom and network_idle?

The load_dom option waits for the DOMContentLoaded event, indicating the initial HTML document has been fully loaded and parsed. The network_idle option waits until there are no network connections for at least 500ms, which typically indicates that JavaScript has finished fetching data from APIs and lazy-loading assets.

Can I interact with the page before Scrapling captures the HTML?

Yes. Pass a callable to the page_action parameter. This callback receives the raw Playwright page object, allowing you to click buttons, fill forms, scroll infinitely, or trigger specific JavaScript events before the fetcher waits for selectors or stability conditions and returns the Response.

Where is the retry and proxy rotation logic implemented?

The retry mechanism and proxy error detection are implemented in scrapling/engines/_browsers/_controllers.py within the DynamicSession and AsyncDynamicSession classes. When a proxy error is detected (is_proxy_error), the session logs a warning and automatically retries with a new proxy from the rotation pool up to the configured retry limit.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}