# Handling Dynamic Content Loading with Scrapling's DynamicFetcher and Playwright

> Master dynamic content loading with Scrapling DynamicFetcher and Playwright. Control page readiness using DOM stability, network idle, and custom selectors.

- Repository: [Karim shoair/Scrapling](https://github.com/D4Vinci/Scrapling)
- Tags: tutorial
- Published: 2026-03-08

---

**Use Scrapling's `DynamicFetcher` to control exactly when a browser-rendered page is considered "ready" by leveraging Playwright's DOM stability, network idle detection, and custom selector waits.**

Handling dynamic content loading with Scrapling's DynamicFetcher and Playwright gives you deterministic control over JavaScript-heavy websites. The `DynamicFetcher` class in the D4Vinci/Scrapling repository wraps Playwright's browser automation capabilities, exposing fine-grained options to wait for specific DOM states, network conditions, or custom page actions before returning a `Response` object.

## What is DynamicFetcher?

`DynamicFetcher` is a high-level interface that launches a Chromium browser (via Playwright) to fetch pages requiring full JavaScript execution. Unlike static fetchers, it allows you to specify exactly when the fetch operation completes—whether that's after the DOM finishes loading, when network activity ceases, or when a specific CSS selector becomes visible.

The fetcher is implemented in [`scrapling/fetchers/chrome.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/fetchers/chrome.py) and delegates actual browser management to session objects defined in [`scrapling/engines/_browsers/_controllers.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/_browsers/_controllers.py).

## Architecture and Implementation

### Public Entry Points

The `DynamicFetcher` class in [`scrapling/fetchers/chrome.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/fetchers/chrome.py) provides two primary methods:

- **`DynamicFetcher.fetch(url, **options)`** – Synchronous fetching
- **`DynamicFetcher.async_fetch(url, **options)`** – Asynchronous fetching

These methods prepare a unified `selector_config` (lines 39-46 of the file), then delegate to session contexts:

```python
with DynamicSession(**kwargs) as session:
    return session.fetch(url)

```

And for async:

```python
async with AsyncDynamicSession(**kwargs) as session:
    return await session.fetch(url)

```

### Session Managers

The core browser logic lives in [`scrapling/engines/_browsers/_controllers.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/_browsers/_controllers.py), which defines:

- **`DynamicSession`** – Synchronous session managing Playwright's browser context, page pooling, and retry logic
- **`AsyncDynamicSession`** – Asynchronous equivalent using Playwright's async API

#### Session Lifecycle

1. **Construction** – Validates options via `_validate` and stores configuration
2. **`start()`** – Launches Chromium or connects to an existing CDP endpoint; decides between persistent context or fresh context per proxy
3. **`fetch()`** – Executes the retrieval workflow:
   - Generates convincing request headers and optional `referer`
   - Enters retry loop (`self._config.retries`)
   - Acquires page from pool via `_page_generator`
   - Calls `page.goto(url, referer=...)`
   - Executes optional `page_action` callback
   - Waits for `wait_selector` and/or `load_dom` / `network_idle` via `_wait_for_page_stability`
   - Sleeps for final `wait` period
   - Builds `Response` via `ResponseFactory.from_playwright_response`

The async version mirrors this flow using `await` and `async with` syntax.

### Supporting Utilities

- **[`scrapling/engines/toolbelt/convertor.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/toolbelt/convertor.py)** – Converts raw Playwright responses into Scrapling's `Response` objects containing `html_content`, `status_code`, `headers`, and the parsed `Selector`
- **[`scrapling/engines/toolbelt/fingerprints.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/toolbelt/fingerprints.py)** – Generates realistic referer headers when `google_search=True`
- **[`scrapling/engines/_browsers/_validators.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/_browsers/_validators.py)** – Type-checks and validates the extensive set of fetch parameters

## Key Configuration Options

When handling dynamic content, these parameters in `DynamicFetcher` provide precise control:

- **`headless`** – Run browser hidden (`True`, default) or visible for debugging
- **`load_dom`** – Wait for `DOMContentLoaded` event before returning
- **`network_idle`** – Ensure no network activity for ≥500ms (indicates JavaScript finished fetching data)
- **`wait_selector`** / **`wait_selector_state`** – Block until specific element reaches state (`attached`, `visible`, `hidden`, `detached`)
- **`page_action`** – User-supplied callback receiving the Playwright `page` object for clicks, scrolls, form fills, or infinite scroll triggers
- **`proxy`** – Route traffic through specified proxy with automatic retry on failure
- **`timeout`** – Maximum milliseconds to wait for navigation
- **`wait`** – Final sleep duration after page stabilizes before capturing HTML

## Practical Code Examples

### Synchronous Fetching

```python
from scrapling import DynamicFetcher

# Basic fetch – wait until DOM is fully loaded and network is idle

response = DynamicFetcher.fetch(
    "https://news.ycombinator.com",
    headless=True,
    load_dom=True,
    network_idle=True,
    timeout=20000,          # ms

    wait=500,               # extra pause after page stabilises

)

print("Status:", response.status_code)
print("HTML length:", len(response.html_content))

```

### Asynchronous Fetching

```python
import asyncio
from scrapling import DynamicFetcher

async def main():
    response = await DynamicFetcher.async_fetch(
        "https://example.com",
        headless=False,               # visible browser for debugging

        wait_selector="div#main",    # wait for a specific element

        wait_selector_state="visible",
        page_action=lambda page: page.click("button#load-more"),
        timeout=30000,
    )
    print("Fetched with async:", response.status_code)

asyncio.run(main())

```

### Advanced Usage with Proxies and Page Actions

```python
from scrapling import DynamicFetcher

def scroll_to_bottom(page):
    # Scroll slowly to trigger infinite scroll loading

    page.evaluate(
        """() => {
            return new Promise(resolve => {
                let totalHeight = 0;
                const distance = 100;
                const timer = setInterval(() => {
                    const scrollHeight = document.body.scrollHeight;
                    window.scrollBy(0, distance);
                    totalHeight += distance;
                    if (totalHeight >= scrollHeight) {
                        clearInterval(timer);
                        resolve();
                    }
                }, 200);
            });
        }"""
    )

response = DynamicFetcher.fetch(
    "https://infinite-scroll-site.com",
    proxy="http://user:pass@proxy.example:3128",
    page_action=scroll_to_bottom,
    load_dom=False,                # we only need the final DOM after scrolling

    timeout=60000,
)
print(response.selector.css("article.title").all())

```

## Command-Line Interface

Scrapling exposes the dynamic fetcher through the CLI (`scrapling-cli`):

```bash
scrapling fetch \
    --fetcher DynamicFetcher \
    --url https://example.com \
    --headless false \
    --wait-selector "#content" \
    --output out.html

```

The CLI wiring lives in [`scrapling/cli.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/cli.py) (see the `@extract.command` decorator around line 575). It forwards all options directly to the fetcher class.

## Summary

- **DynamicFetcher** provides browser-level fetching with deterministic controls for when a page is considered "ready"
- Synchronous (`fetch`) and asynchronous (`async_fetch`) entry points in [`scrapling/fetchers/chrome.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/fetchers/chrome.py) delegate to `DynamicSession` and `AsyncDynamicSession`
- Core session logic in [`scrapling/engines/_browsers/_controllers.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/_browsers/_controllers.py) handles browser lifecycle, page pooling, retries, and proxy rotation
- Key waiting strategies include `load_dom`, `network_idle`, `wait_selector`, and custom `page_action` callbacks
- All responses are converted to Scrapling's standard `Response` objects via `ResponseFactory` in [`scrapling/engines/toolbelt/convertor.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/toolbelt/convertor.py)

## Frequently Asked Questions

### How does DynamicFetcher differ from Scrapling's static fetchers?

Static fetchers retrieve raw HTML using HTTP clients like `httpx` or `requests`, which cannot execute JavaScript. **DynamicFetcher** launches a real Chromium browser via Playwright, allowing it to render SPAs, execute AJAX calls, and wait for dynamically injected content before returning the final HTML.

### What is the difference between `load_dom` and `network_idle`?

The **`load_dom`** option waits for the `DOMContentLoaded` event, indicating the initial HTML document has been fully loaded and parsed. The **`network_idle`** option waits until there are no network connections for at least 500ms, which typically indicates that JavaScript has finished fetching data from APIs and lazy-loading assets.

### Can I interact with the page before Scrapling captures the HTML?

Yes. Pass a callable to the **`page_action`** parameter. This callback receives the raw Playwright `page` object, allowing you to click buttons, fill forms, scroll infinitely, or trigger specific JavaScript events before the fetcher waits for selectors or stability conditions and returns the `Response`.

### Where is the retry and proxy rotation logic implemented?

The retry mechanism and proxy error detection are implemented in **[`scrapling/engines/_browsers/_controllers.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/_browsers/_controllers.py)** within the `DynamicSession` and `AsyncDynamicSession` classes. When a proxy error is detected (`is_proxy_error`), the session logs a warning and automatically retries with a new proxy from the rotation pool up to the configured retry limit.