# crawl4ai | UncleCode | Knowledge Base | Instagit

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN

GitHub Stars: 61.4k

Repository: https://github.com/unclecode/crawl4ai

---

## Articles

### [How to Deploy Crawl4AI in Docker with the Official Image](/unclecode/crawl4ai/how-to-deploy-crawl4ai-in-docker-with-the-official-image)

Quickly deploy Crawl4AI using the official Docker image. Follow simple steps to run it locally or use docker-compose for production-ready deployments. Get started now.

- Tags: how-to-guide
- Published: 2026-03-05

### [How to Use URL Seeding and Filtering for Focused Crawling in crawl4ai](/unclecode/crawl4ai/how-to-use-url-seeding-and-filtering-for-focused-crawling-in-crawl4ai)

Master focused crawling in crawl4ai using URL seeding and filtering. Discover, filter, and extract targeted data efficiently with AsyncUrlSeeder and AsyncWebCrawler for precise web scraping.

- Tags: how-to-guide
- Published: 2026-03-05

### [How to Implement Crash Recovery with resume_state for Long Crawls in crawl4ai](/unclecode/crawl4ai/how-to-implement-crash-recovery-with-resume-state-for-long-crawls-in-crawl4ai)

Implement crash recovery for long crawls in crawl4ai using resume_state. Prevent data loss and restore crawl progress seamlessly with our guide.

- Tags: how-to-guide
- Published: 2026-03-05

### [How to Bypass Anti-Bot Detection with Stealth Settings in Crawl4AI](/unclecode/crawl4ai/how-to-bypass-anti-bot-detection-with-stealth-settings-in-crawl4ai)

Learn to bypass anti-bot detection using Crawl4AI's stealth settings. Enable stealth mode to mask browser fingerprints and evade common bot detection checks with ease.

- Tags: how-to-guide
- Published: 2026-03-05

### [How to Switch Between Chromium, Firefox, and WebKit Browsers in Crawl4AI](/unclecode/crawl4ai/how-to-switch-between-chromium-firefox-and-webkit-browsers-in-crawl4ai)

Quickly switch between Chromium Firefox and WebKit browsers in Crawl4AI by setting the browser_type parameter in BrowserConfig No code changes needed for your crawling logic

- Tags: how-to-guide
- Published: 2026-03-05

### [How to Set Up Webhooks for Crawling Event Notifications in Crawl4AI](/unclecode/crawl4ai/how-to-set-up-webhooks-for-crawling-event-notifications-in-crawl4ai)

Learn how to set up webhooks for crawling event notifications in Crawl4AI. Configure webhook_config to receive automatic event POSTs to your endpoint with retries.

- Tags: how-to-guide
- Published: 2026-03-05

### [How to Handle Cookies and Simulate Authenticated Sessions in Crawl4AI](/unclecode/crawl4ai/how-to-handle-cookies-and-simulate-authenticated-sessions-in-crawl4ai)

Learn to handle cookies and simulate authenticated sessions in Crawl4AI using static injection dynamic management and session reuse. Boost your web scraping efficiency.

- Tags: how-to-guide
- Published: 2026-03-05

### [How to Set Up Persistent Browser Profiles with Authentication in Crawl4AI](/unclecode/crawl4ai/how-to-set-up-persistent-browser-profiles-with-authentication-in-crawl4ai)

Learn to set up persistent browser profiles with authentication in Crawl4AI. Use create profile and user data dir to maintain authenticated sessions across multiple crawl jobs.

- Tags: how-to-guide
- Published: 2026-03-05

### [How to Use LinkPreview for URL Metadata Extraction in crawl4ai](/unclecode/crawl4ai/how-to-use-link-preview-for-url-metadata-extraction-in-crawl4ai)

Learn how to use LinkPreview for URL metadata extraction in crawl4ai. Fetch link head sections in parallel with configurable patterns and BM25 scoring.

- Tags: how-to-guide
- Published: 2026-03-05

### [How to Create a Custom Markdown Generation Strategy in Crawl4AI](/unclecode/crawl4ai/how-to-create-a-custom-markdown-generation-strategy-in-crawl4ai)

Learn how to create a custom Markdown generation strategy in Crawl4AI. Subclass MarkdownGenerationStrategy, implement generate_markdown, and inject your custom generator for tailored output.

- Tags: how-to-guide
- Published: 2026-03-05

### [How to Extract Tables from HTML with TableExtractionStrategy in crawl4ai](/unclecode/crawl4ai/how-to-extract-tables-from-html-with-table-extraction-strategy-in-crawl4ai)

Learn to extract tables from HTML using crawl4ai's TableExtractionStrategy. Customize parsing logic or use heuristic-based detection for efficient data extraction.

- Tags: how-to-guide
- Published: 2026-03-05

### [How to Configure Rate Limiting and Memory-Adaptive Dispatchers in Crawl4AI](/unclecode/crawl4ai/how-to-configure-rate-limiting-and-memory-adaptive-dispatchers-in-crawl4ai)

Learn to configure rate limiting and memory-adaptive dispatchers in Crawl4AI. Efficiently manage per-domain delays and prevent OOM crashes with this guide.

- Tags: how-to-guide
- Published: 2026-03-05

### [How to Use the crwl CLI for Quick Crawling Tasks in Crawl4AI](/unclecode/crawl4ai/how-to-use-the-crwl-cli-for-quick-crawling-tasks-in-crawl4ai)

Learn to use the crwl CLI in Crawl4AI to quickly fetch pages, extract data, and query LLMs from the terminal without writing Python code.

- Tags: how-to-guide
- Published: 2026-03-05

### [How Adaptive Crawling Learns Site Patterns Automatically in crawl4ai](/unclecode/crawl4ai/how-does-adaptive-crawling-learn-site-patterns-automatically-in-crawl4ai)

Discover how crawl4ai's adaptive crawling automatically learns site patterns. It scores links for relevance, novelty, and semantic gap coverage to optimize your crawls efficiently.

- Tags: deep-dive
- Published: 2026-03-05

### [How to Wait for JavaScript to Execute and Handle Dynamic Content with crawl4ai](/unclecode/crawl4ai/how-to-wait-for-javascript-to-execute-and-handle-dynamic-content-with-crawl4ai)

Learn to wait for JavaScript execution and handle dynamic content in crawl4ai. Use the wait_for parameter to ensure content is rendered before extraction. Unlock full web scraping potential.

- Tags: how-to-guide
- Published: 2026-03-05

### [How to Use BM25 Content Filtering to Remove Noise in crawl4ai](/unclecode/crawl4ai/how-to-use-bm25-content-filtering-to-remove-noise-in-crawl4ai)

Learn how to use BM25 content filtering in crawl4ai to automatically remove ads and navigation. Preserve relevant content with this powerful tool.

- Tags: tutorial
- Published: 2026-03-05

### [How to Extract Data Using CSS Selectors with JsonCssExtractionStrategy in crawl4ai](/unclecode/crawl4ai/how-to-extract-data-using-css-selectors-with-jsoncss-extraction-strategy-in-crawl4ai)

Learn to extract structured data from HTML using CSS selectors and JsonCssExtractionStrategy in crawl4ai. Map CSS selectors to JSON schema fields with BeautifulSoup for efficient parsing.

- Tags: how-to-guide
- Published: 2026-03-05

### [Understanding Crawl4AI Caching Modes: BYPASS, READ_ONLY, WRITE_ONLY, and More](/unclecode/crawl4ai/what-are-the-different-caching-modes-in-crawl4ai-bypass-read-only-write-only)

Explore Crawl4AI caching modes like BYPASS, READ_ONLY, and WRITE_ONLY. Understand how these options control disk access for efficient web crawling and data management with the AsyncWebCrawler.

- Tags: deep-dive
- Published: 2026-03-05

### [How to Maintain Browser Sessions and Cookies Across Requests in crawl4ai](/unclecode/crawl4ai/how-to-maintain-browser-sessions-and-cookies-across-requests-in-crawl4ai)

Learn to maintain browser sessions and cookies across requests in crawl4ai. Configure user_data_dir and reuse session_id for persistent browser contexts in your crawls.

- Tags: how-to-guide
- Published: 2026-03-05

### [How to Implement Custom Hooks at Different Crawling Stages in crawl4ai](/unclecode/crawl4ai/how-to-implement-custom-hooks-at-different-crawling-stages-in-crawl4ai)

Learn to implement custom hooks in crawl4ai for pre crawl, pre fetch, post fetch, and post crawl stages. Inject your logic easily using dictionaries or decorators.

- Tags: how-to-guide
- Published: 2026-03-05

### [Deep Crawling Strategies in crawl4ai: BFS vs DFS vs Best-First Explained](/unclecode/crawl4ai/explain-deep-crawling-strategies-bfs-vs-dfs-vs-best-first-in-crawl4ai)

Master deep crawling strategies BFS DFS and Best-First in crawl4ai. Understand how each method discovers URLs to optimize your web crawling and data extraction.

- Tags: deep-dive
- Published: 2026-03-05

### [How to Use LLM Extraction Strategies for Structured JSON Data with Crawl4AI](/unclecode/crawl4ai/how-to-use-llm-extraction-strategies-for-structured-json-data-with-crawl4ai)

Unlock structured JSON data from web pages using LLM extraction strategies with Crawl4AI. Convert unstructured HTML into typed JSON objects effortlessly.

- Tags: how-to-guide
- Published: 2026-03-05