# firecrawl | Firecrawl | Knowledge Base | Instagit

🔥 The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data

GitHub Stars: 87k

Repository: https://github.com/firecrawl/firecrawl

---

## Articles

### [Self-Hosting Firecrawl: Deployment Considerations and Environment Configuration](/firecrawl/firecrawl/self-hosting-deployment-considerations-and-environment-configuration)

Deploy Firecrawl on-premises with Docker Compose. Configure your environment and understand the five containerized services required for this self-hosting solution.

- Tags: deployment-guide
- Published: 2026-03-01

### [Firecrawl Search API: How to Search Web, Images, News, GitHub, Research and PDF Categories](/firecrawl/firecrawl/search-api-with-web-images-news-sources-and-github-research-pdf-categories)

Discover the Firecrawl Search API to query web pages, images, news, GitHub, research, and PDFs. Access unified content search and scraping through a single API endpoint.

- Tags: how-to-guide
- Published: 2026-03-01

### [Complete Guide to Firecrawl Cache Control with maxAge and minAge Parameters](/firecrawl/firecrawl/cache-control-with-maxage-and-minage-parameters)

Master Firecrawl cache control using maxAge and minAge. Optimize scrape results freshness and enforce cache retrieval for efficient web scraping.

- Tags: guide
- Published: 2026-03-01

### [URL Discovery and Sitemap Parsing with Firecrawl’s /map Endpoint: A Technical Deep Dive](/firecrawl/firecrawl/url-discovery-and-sitemap-parsing-with-the-map-endpoint)

Discover and parse URLs from any domain with Firecrawl's /map endpoint. Automate sitemap ingestion, redirect resolution, and intelligent filtering for a clean URL list.

- Tags: deep-dive
- Published: 2026-03-01

### [Handling Rate Limit Errors and Robots.txt Compliance in Firecrawl: A Complete Guide](/firecrawl/firecrawl/handling-rate-limit-errors-and-robotstxt-compliance)

Master firecrawl rate limit errors & robots.txt compliance. Learn how firecrawl uses Redis for 429 responses and blocks disallowed URLs via robots.txt for seamless web scraping.

- Tags: how-to-guide
- Published: 2026-03-01

### [How to Monitor Crawl Status, Track Job Progress, and Handle Pagination with Firecrawl](/firecrawl/firecrawl/monitoring-crawl-status-tracking-job-progress-and-pagination)

Monitor crawl status and track job progress with Firecrawl. Learn how to handle pagination efficiently using jobId and next cursors for seamless data retrieval.

- Tags: how-to-guide
- Published: 2026-03-01

### [Understanding Credit Usage, Billing, and Cost Tracking in Firecrawl](/firecrawl/firecrawl/understanding-credit-usage-billing-and-cost-tracking)

Master Firecrawl credit usage and billing. Learn to track costs in real time with ledger services and SDK methods for efficient scraping operations.

- Tags: billing-and-usage
- Published: 2026-03-01

### [Using the Agent API for AI-Powered Autonomous Data Extraction in Firecrawl](/firecrawl/firecrawl/using-the-agent-api-for-ai-powered-autonomous-data-extraction)

Learn how to use the Firecrawl Agent API for autonomous AI data extraction. Crawl URLs, execute logic, and get structured data with this simple API workflow.

- Tags: how-to-guide
- Published: 2026-03-01

### [How to Filter Content with includeTags and excludeTags in Firecrawl](/firecrawl/firecrawl/filtering-content-with-includetags-and-excludetags-options)

Master content filtering in Firecrawl using includeTags and excludeTags. Precisely control extracted content by whitelisting or blacklisting HTML elements, classes, and IDs for targeted scraping.

- Tags: how-to-guide
- Published: 2026-03-01

### [Taking Screenshots with Custom Viewport and Quality Settings in Firecrawl](/firecrawl/firecrawl/taking-screenshots-with-custom-viewport-and-quality-settings)

Learn to take custom screenshots with Firecrawl using custom viewport and quality settings. Control dimensions and JPEG compression for precise web page captures.

- Tags: how-to-guide
- Published: 2026-03-01

### [How to Extract Specific Attributes Using CSS Selectors with Firecrawl](/firecrawl/firecrawl/extracting-specific-attributes-using-css-selectors)

Easily extract specific attributes like href or src using CSS selectors with Firecrawl. Leverage a fast Rust engine for precise data extraction from web pages.

- Tags: how-to-guide
- Published: 2026-03-01

### [Batch Scraping Multiple URLs with maxConcurrency Control in Firecrawl](/firecrawl/firecrawl/batch-scraping-multiple-urls-with-maxconcurrency-control)

Discover how to batch scrape multiple URLs with maxConcurrency control in Firecrawl. Protect your plan and infrastructure by limiting concurrent requests to thousands of URLs efficiently.

- Tags: how-to-guide
- Published: 2026-03-01

### [Working with Authentication Walls Using Click, Write, and Press Actions in Firecrawl](/firecrawl/firecrawl/working-with-authentication-walls-using-click-write-and-press-actions)

Learn to bypass authentication walls in Firecrawl using click, write, and press actions. Our SDKs automatically manage token renewal for seamless scraping.

- Tags: how-to-guide
- Published: 2026-03-01

### [Implementing Change Tracking to Monitor Website Updates Over Time with Firecrawl](/firecrawl/firecrawl/implementing-change-tracking-to-monitor-website-updates-over-time)

Learn how to implement website change tracking with Firecrawl. Monitor content updates, generate git-style diffs, and get structured JSON diffs comparing past and present scrapes.

- Tags: how-to-guide
- Published: 2026-03-01

### [How Firecrawl Handles Geolocation and Country-Specific Content Targeting](/firecrawl/firecrawl/geolocation-and-country-specific-content-targeting)

Discover how Firecrawl achieves geolocation and country-specific content targeting. Learn about country-specific proxies and IP-based blocking for precise content delivery.

- Tags: how-to-guide
- Published: 2026-03-01

### [Complete Guide to PDF and Document Parsing from Scraped Content in Firecrawl](/firecrawl/firecrawl/pdf-and-document-parsing-from-scraped-content)

Master PDF and document parsing from scraped content with Firecrawl. Learn how this powerful tool extracts text and metadata effortlessly for your projects.

- Tags: tutorial
- Published: 2026-03-01

### [How to Crawl Entire Websites with Firecrawl crawlerOptions: Depth Limits, Include/Exclude Paths](/firecrawl/firecrawl/crawling-entire-websites-with-crawleroptions-depth-limits-include-exclude-paths)

Master website crawling with Firecrawl crawlerOptions. Set depth limits and include/exclude paths to precisely control your crawls. Get started now.

- Tags: how-to-guide
- Published: 2026-03-01

### [How to Set Up Webhooks for Async Scrape and Crawl Job Completion in Firecrawl](/firecrawl/firecrawl/setting-up-webhooks-for-async-scrape-and-crawl-job-completion)

Learn to set up webhooks for async scrape and crawl job completion in Firecrawl. Get real-time job status updates without polling by configuring HTTP POST requests.

- Tags: how-to-guide
- Published: 2026-03-01

### [Firecrawl Rate Limiting Configuration and Concurrency Controls: A Technical Deep Dive](/firecrawl/firecrawl/rate-limiting-configuration-and-concurrency-controls-across-different-endpoints)

Master Firecrawl rate limiting and concurrency controls. Explore per-endpoint limits and team-wide job management using Redis for robust infrastructure protection.

- Tags: deep-dive
- Published: 2026-03-01

### [Handling JavaScript-Rendered Pages with waitFor and actions in Firecrawl](/firecrawl/firecrawl/handling-javascript-rendered-pages-with-waitfor-and-actions)

Effortlessly scrape JavaScript-rendered pages with Firecrawl. Learn to use waitFor for delays and actions for complex browser interactions like clicks and scrolls to master dynamic SPAs.

- Tags: how-to-guide
- Published: 2026-03-01

### [Configuring Proxy Settings and Stealth Mode in Firecrawl to Bypass Anti-Bot Protection](/firecrawl/firecrawl/configuring-proxy-settings-and-stealth-mode-for-bypassing-anti-bot-protection)

Learn to configure proxy settings and stealth mode in Firecrawl to bypass anti-bot protection. Utilize advanced proxy networking for seamless web scraping.

- Tags: how-to-guide
- Published: 2026-03-01

### [How to Extract Structured JSON Data from Websites Using JSON Schema in Firecrawl](/firecrawl/firecrawl/how-to-extract-structured-json-data-from-websites-using-json-schema)

Extract structured JSON data from websites using JSON schema with Firecrawls powerful API. Ensure strict type conformance for your extracted data.

- Tags: how-to-guide
- Published: 2026-03-01