# How to Wait for JavaScript to Execute and Handle Dynamic Content with crawl4ai

> Learn to wait for JavaScript execution and handle dynamic content in crawl4ai. Use the wait_for parameter to ensure content is rendered before extraction. Unlock full web scraping potential.

- Repository: [UncleCode/crawl4ai](https://github.com/unclecode/crawl4ai)
- Tags: how-to-guide
- Published: 2026-03-05

---

**Use the `wait_for` parameter in `CrawlerRunConfig` to pause execution until a CSS selector appears or a JavaScript expression returns `true`, ensuring dynamic content is fully rendered before extraction.**

The **crawl4ai** library renders web pages using a real Playwright-controlled browser, making it ideal for scraping modern JavaScript-heavy applications. When dealing with dynamic content that loads asynchronously, you need a reliable mechanism to pause crawling until specific conditions are met. The library provides this through the `wait_for` configuration option, implemented primarily in [`crawl4ai/async_crawler_strategy.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_crawler_strategy.py).

## Understanding the `wait_for` Mechanism

The core waiting logic resides in the `smart_wait` method within [`crawl4ai/async_crawler_strategy.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_crawler_strategy.py) (lines 231-290). This method intelligently interprets your wait condition and delegates to the appropriate Playwright API or custom JavaScript evaluator.

### How `smart_wait` Detects Wait Types

The method analyzes the `wait_for` string to determine the waiting strategy:

1. **`js:` prefix** — Treats the remainder as a JavaScript snippet to evaluate repeatedly until it returns `true`
2. **`css:` prefix** — Treats the remainder as a CSS selector to wait for via `page.wait_for_selector()`
3. **No prefix** — First attempts to parse as a JavaScript function (detecting `() =>` or `function` syntax), then falls back to CSS selector if parsing fails

### CSP-Compliant JavaScript Execution

For JavaScript waits, `smart_wait` calls `csp_compliant_wait` (lines 294-314 in [`crawl4ai/async_crawler_strategy.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_crawler_strategy.py)). This helper repeatedly evaluates your JavaScript expression every 100 milliseconds until it returns a truthy value or the timeout expires (defaulting to the `page_timeout` value, approximately 60 seconds).

If the condition never becomes true, the method raises a clear `ValueError` indicating the wait condition timed out, making debugging straightforward.

## Waiting for CSS Selectors

Use the `css:` prefix when you need to wait for a specific element to appear in the DOM. This leverages Playwright's built-in `wait_for_selector` method, which is efficient and handles mutation observation automatically.

```python
import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig

async def main():
    browser_cfg = BrowserConfig(headless=True)
    
    async with AsyncWebCrawler(config=browser_cfg) as crawler:
        # Wait until an element with class "article" appears

        result = await crawler.arun(
            url="https://example.com/news",
            wait_for="css:.article",
            screenshot=True
        )
        print(result.markdown)

asyncio.run(main())

```

In this example, `smart_wait` detects the `css:` prefix and calls `page.wait_for_selector('.article')`, pausing execution until the element is present or the timeout occurs.

## Waiting for JavaScript Conditions

For complex dynamic content that doesn't correspond to simple DOM elements—such as waiting for specific data to load, animations to complete, or custom JavaScript variables to be set—use the `js:` prefix.

```python
import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        # Wait until at least 5 product elements are loaded

        result = await crawler.arun(
            url="https://example.com/shop",
            wait_for="js:return document.querySelectorAll('.product').length >= 5"
        )
        print(result.markdown)

asyncio.run(main())

```

Here, `csp_compliant_wait` repeatedly evaluates the JavaScript expression every 100ms until it returns `true` (indicating five or more products are present) or the default timeout expires.

## Implicit Detection Without Prefixes

You can omit prefixes entirely, and `smart_wait` will attempt to infer the correct type. This is useful for quick scripts but requires careful syntax to avoid ambiguity.

```python
import asyncio
from crawl4ai import AsyncWebCrawler

async def main():
    async with AsyncWebCrawler() as crawler:
        # No prefix: treated as JS due to arrow function syntax

        result = await crawler.arun(
            url="https://example.com/gallery",
            wait_for="() => window.galleryReady === true"
        )
        print(result.markdown)

asyncio.run(main())

```

The method detects the `() =>` pattern and routes this to `csp_compliant_wait`. If the string fails JavaScript parsing, it falls back to treating the string as a CSS selector.

## Using the Synchronous API

The `wait_for` parameter works identically in the synchronous `WebCrawler` class, which wraps the same underlying logic for simpler scripting scenarios.

```python
from crawl4ai import WebCrawler, BrowserConfig

crawler = WebCrawler(config=BrowserConfig())
result = crawler.run(
    url="https://example.com",
    wait_for="css:#main-content"
)
print(result.markdown)

```

Both async and sync implementations delegate to the same `smart_wait` logic in [`crawl4ai/async_crawler_strategy.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_crawler_strategy.py), ensuring consistent behavior across APIs.

## Summary

- **crawl4ai** handles dynamic content through the `wait_for` parameter in `CrawlerRunConfig`, implemented in [`crawl4ai/async_crawler_strategy.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_crawler_strategy.py).
- Use **`css:`** prefix to wait for DOM elements via Playwright's `wait_for_selector`.
- Use **`js:`** prefix to wait for custom JavaScript conditions evaluated by `csp_compliant_wait` in 100ms polling intervals.
- Omit prefixes to let `smart_wait` auto-detect JavaScript functions (detecting `() =>` or `function` syntax) or fall back to CSS selectors.
- Both async (`AsyncWebCrawler`) and sync (`WebCrawler`) APIs support identical `wait_for` functionality with default timeouts around 60 seconds.

## Frequently Asked Questions

### What is the default timeout for `wait_for` in crawl4ai?

The default timeout follows the `page_timeout` configuration value, which is approximately 60 seconds. If the condition (CSS selector or JavaScript expression) is not satisfied within this window, `smart_wait` raises a `ValueError` indicating the wait condition timed out. You can adjust this timeout by modifying the `page_timeout` parameter in your `BrowserConfig`.

### Can I wait for multiple conditions simultaneously?

Currently, `wait_for` accepts a single string condition. To wait for multiple conditions, you should combine them into a single JavaScript expression using logical operators. For example: `wait_for="js:return document.querySelector('.products') && document.querySelector('.reviews')"`. This ensures both elements exist before proceeding with content extraction.

### How does crawl4ai handle Content Security Policy (CSP) restrictions when executing JavaScript waits?

The library uses `csp_compliant_wait` (lines 294-314 in [`crawl4ai/async_crawler_strategy.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_crawler_strategy.py)), which evaluates JavaScript within the page context using Playwright's `page.evaluate()`. This approach respects the page's CSP because it runs inside the browser's normal execution environment rather than injecting external scripts. The method polls every 100 milliseconds until the condition returns true or the timeout expires.

### Is there a performance difference between CSS and JavaScript waits?

Yes. CSS selector waits leverage Playwright's native `wait_for_selector`, which uses browser-level mutation observers and is generally more efficient with lower overhead. JavaScript waits require repeated execution of `page.evaluate` every 100ms via `csp_compliant_wait`, consuming slightly more CPU and network resources. For simple element presence, prefer `css:` selectors; use `js:` only when you need complex logic like checking element text content, computed styles, or global state variables.