# How to Maintain Browser Sessions and Cookies Across Requests in crawl4ai

> Learn to maintain browser sessions and cookies across requests in crawl4ai. Configure user_data_dir and reuse session_id for persistent browser contexts in your crawls.

- Repository: [UncleCode/crawl4ai](https://github.com/unclecode/crawl4ai)
- Tags: how-to-guide
- Published: 2026-03-05

---

**To maintain browser sessions and cookies across requests in crawl4ai, configure a persistent `user_data_dir` in `BrowserConfig` to save the Chromium profile to disk, and reuse the same `session_id` in `CrawlerRunConfig` to keep browser contexts alive across multiple crawl operations.**

Maintaining state between HTTP requests is essential when scraping authenticated web applications or multi-page workflows. In `unclecode/crawl4ai`, session persistence is handled through a combination of persistent browser profiles and in-memory session caching, allowing cookies and local storage to survive across independent crawl jobs. This article explains the four core mechanisms that enable you to maintain browser sessions and cookies across requests according to the actual source implementation.

## The Architecture of Session Persistence

Crawl4AI implements session continuity through four coordinated systems defined in [`crawl4ai/browser_manager.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/browser_manager.py) and [`crawl4ai/managed_browser.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/managed_browser.py):

- **Persistent User Data Directories**: The `ManagedBrowser` class launches Chromium with a `--user-data-dir` argument that preserves cookies, cache, and localStorage on disk.
- **Session ID Mapping**: The `BrowserManager` maintains an internal `self.sessions` dictionary that maps `session_id` strings to active browser contexts and pages.
- **Automatic Cookie Synchronization**: The `setup_context` method injects cookies from `BrowserConfig.cookies` into new contexts.
- **Runtime State Cloning**: The `clone_runtime_state` helper copies cookies and localStorage between contexts for deep-crawl resumption.

## Method 1: Persistent Browser Profiles with user_data_dir

The most reliable way to maintain cookies across separate Python runs is to specify a persistent profile directory. In [`crawl4ai/managed_browser.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/managed_browser.py), the `ManagedBrowser.__init__` method (around lines 49-57) accepts a `user_data_dir` parameter that gets passed directly to the Chromium launch arguments.

When `ManagedBrowser.start` is called, it only creates a temporary directory if `user_data_dir` is `None`. Otherwise, it reuses the existing profile folder, preserving all browser state including cookies and localStorage.

```python
from crawl4ai import Crawl4AI, BrowserConfig

browser_cfg = BrowserConfig(
    user_data_dir="/tmp/crawl4ai_session",  # Persistent profile location

    headless=False
)

crawler = Crawl4AI(browser_config=browser_cfg)

```

## Method 2: Session Caching via session_id

For maintaining state across multiple `run()` calls within the same Python process, use the `session_id` parameter in `CrawlerRunConfig`. The `BrowserManager.__init__` method (around lines 10807-10810) initializes `self.sessions` as a dictionary and sets `self.session_ttl` to 30 minutes.

When you call `crawler.run()` with a `session_id`, the `BrowserManager.get_page` method checks `self.sessions` for an existing context. If found, it returns the cached page instead of creating a new one, preserving all cookies and JavaScript state.

```python
from crawl4ai import Crawl4AI, CrawlerRunConfig, BrowserConfig

browser_cfg = BrowserConfig(headless=True)
crawler = Crawl4AI(browser_config=browser_cfg)

# First request: creates the session

result1 = await crawler.run(CrawlerRunConfig(
    url="https://example.com/login",
    session_id="auth_session"
))

# Second request: reuses the same browser context and cookies

result2 = await crawler.run(CrawlerRunConfig(
    url="https://example.com/dashboard",
    session_id="auth_session"
))

```

Idle sessions are automatically purged by `_cleanup_expired_sessions` when their age exceeds `session_ttl` (30 minutes).

## Method 3: Injecting Custom Cookies

The `BrowserManager.setup_context` method (around lines 10871-10876) automatically propagates any cookies defined in `BrowserConfig.cookies` to new browser contexts. This is useful for adding authentication tokens or preferences before navigation.

```python
custom_cookies = [
    {"name": "auth_token", "value": "xyz123", "url": "https://example.com"},
    {"name": "theme", "value": "dark", "url": "https://example.com"}
]

browser_cfg = BrowserConfig(
    cookies=custom_cookies,
    headless=True
)

```

## Method 4: Cloning Runtime State

When you need to create a fresh browser context but retain exact state from a previous session, use the `clone_runtime_state` function in [`crawl4ai/browser_manager.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/browser_manager.py) (around lines 10122-10158). This utility copies cookies, localStorage, headers, and geolocation from a source context to a destination context.

This is particularly useful for resuming deep crawls or duplicating authenticated sessions across parallel workers.

```python
from crawl4ai.browser_manager import clone_runtime_state

# Assuming src_ctx and dst_ctx are BrowserContext objects

await clone_runtime_state(
    src_ctx, 
    dst_ctx, 
    crawlerRunConfig=run_cfg, 
    browserConfig=browser_cfg
)

```

## Summary

- **Use `user_data_dir`** in `BrowserConfig` to persist cookies to disk across Python processes via `ManagedBrowser` in [`crawl4ai/managed_browser.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/managed_browser.py).
- **Use `session_id`** in `CrawlerRunConfig` to cache live browser contexts in memory via `BrowserManager.sessions`, with automatic TTL cleanup after 30 minutes.
- **Set `BrowserConfig.cookies`** to inject specific cookies on context creation in `BrowserManager.setup_context` (lines 10871-10876).
- **Call `clone_runtime_state`** to copy cookies and localStorage between contexts when resuming complex crawl workflows (lines 10122-10158).

## Frequently Asked Questions

### How long do sessions stay cached in BrowserManager?

Sessions remain cached for **30 minutes** by default, controlled by `self.session_ttl` in `BrowserManager.__init__`. The `_cleanup_expired_sessions` method runs automatically before each new page request and removes entries older than this threshold to prevent memory leaks.

### Can I use multiple different sessions simultaneously?

Yes. The `self.sessions` dictionary in `BrowserManager` supports multiple concurrent session IDs. Each unique `session_id` maintains its own isolated browser context, cookies, and localStorage, allowing you to crawl as different users concurrently within the same process.

### Where are cookies physically stored when using user_data_dir?

When `BrowserConfig.user_data_dir` is set, Chromium writes its SQLite cookie database and localStorage files to that directory path. According to the `ManagedBrowser.start` implementation in [`crawl4ai/managed_browser.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/managed_browser.py), this directory persists between program runs unless explicitly deleted.

### How do I clear cookies between requests?

To clear cookies, either omit the `session_id` to create a fresh anonymous context, or use a new `session_id` to start a separate session. If using `user_data_dir`, delete the profile directory or point to a different path to ensure a clean state.