How to Set Up and Use the MCP Server for AI-Assisted Web Scraping with Scrapling

Install Scrapling with AI extras, start the MCP server via scrapling mcp, and configure your AI client to call structured scraping tools like get, fetch, and stealthy_fetch for token-efficient web data extraction.

The Scrapling MCP server exposes the library's scraping capabilities as structured tools that any Model Context Protocol (MCP) compatible AI assistant can invoke. According to the D4Vinci/Scrapling source code, this integration allows AI systems to perform AI-assisted web scraping with Scrapling by calling specific functions that return concise, structured JSON instead of raw HTML, dramatically reducing token consumption and improving reliability.

What Is the Scrapling MCP Server?

The MCP server implementation in scrapling/core/ai.py wraps Scrapling's scraping engines behind a FastMCP interface. When you run scrapling mcp, the ScraplingMCPServer.serve method instantiates the server and registers six core tools: get, bulk_get, fetch, bulk_fetch, stealthy_fetch, and bulk_stealthy_fetch【ai.py L59-L106】.

Each tool returns a ResponseModel Pydantic object containing status, content (a list of extracted strings), and url【ai.py L32-L38】. This structured output ensures AI assistants receive clean, parseable data without HTML noise.

Installation Requirements

Before starting the MCP server for AI-assisted web scraping with Scrapling, install the package with AI extras and download the required browser binaries:

# Install Scrapling with MCP dependencies (FastMCP and server entry-point)

pip install "scrapling[ai]"

# Install Playwright browsers required for dynamic scraping

scrapling install

These steps are documented in the official MCP server guide at docs/ai/mcp-server.md【mcp-server.md L39-L48】.

Starting the MCP Server

The server supports two transport modes: stdio (default for local AI clients) and streamable-HTTP (for remote or containerized deployments).

Stdio Mode (Default)

Run the server in stdio mode for direct integration with Claude Desktop and other local MCP clients:

scrapling mcp

This command invokes ScraplingMCPServer().serve(False, "0.0.0.0", 8000) where the first argument False disables HTTP mode【cli.py L56-L61】.

Streamable-HTTP Mode

For remote access or Docker deployments, start the server with HTTP transport:

scrapling mcp --http --host 127.0.0.1 --port 8000

The --http flag switches the transport to "streamable-http" as implemented in scrapling/cli.py【cli.py L43-L55】. You can customize the bind address using --host and --port arguments.

Configuring AI Clients

Once the server is running, register it with your AI assistant using the Model Context Protocol configuration format.

Claude Desktop Configuration

Add the following entry to your claude_desktop_config.json file to enable AI-assisted web scraping with Scrapling:

{
  "mcpServers": {
    "ScraplingServer": {
      "command": "/usr/local/bin/scrapling",
      "args": ["mcp"]
    }
  }
}

For HTTP mode, replace the command with a wrapper script or Docker container that invokes scrapling mcp --http --host 127.0.0.1 --port 8000【mcp-server.md L13-L24】.

After saving the configuration and restarting Claude Desktop, a wrench icon (🔧) will appear in the interface, confirming the server connection.

Other MCP-Compatible Tools

The same configuration format works for Cursor, WindSurf, and other MCP-compatible editors. Simply adjust the "command" and "args" fields to point to your Scrapling installation and desired transport mode.

Available Scraping Tools

The MCP server exposes six structured tools for different scraping scenarios:

• get – Fast HTTP GET requests for static content
• bulk_get – Concurrent GET requests for multiple URLs
• fetch – Browser-based fetching for JavaScript-rendered pages
• bulk_fetch – Concurrent browser fetching for multiple URLs
• stealthy_fetch – Stealth browser mode for anti-bot protected sites (Cloudflare, etc.)
• bulk_stealthy_fetch – Concurrent stealth fetching for multiple URLs

Each tool accepts parameters like url, css_selector, extraction_type (markdown/html/text), and browser-specific options like network_idle or wait_selector.

Practical Usage Examples

Basic Web Scraping Prompts

Once configured, prompt your AI assistant naturally to invoke the MCP server for AI-assisted web scraping with Scrapling:

Example 1: "Scrape the main content from https://example.com and convert it to markdown."

This invokes the get tool with extraction_type="markdown" and main_content_only=True.

Example 2: "Extract all product titles from https://shop.example.com using selector .product-title."

This calls get with css_selector=".product-title" to limit extraction to specific elements, saving tokens and improving accuracy.

Example 3: "Fetch these three pages concurrently and return their text."

This triggers bulk_fetch with a list of URLs and extraction_type="text" for parallel processing.

Example 4: "Bypass Cloudflare on https://protected.com and get the price."

This invokes stealthy_fetch to use anti-detection browser techniques for protected sites.

Programmatic Access via Python

You can also interact with the MCP server programmatically using stdio communication:

import json
import subprocess

# Start the server in stdio mode

proc = subprocess.Popen(
    ["scrapling", "mcp"],
    stdout=subprocess.PIPE,
    stdin=subprocess.PIPE,
    text=True
)

def mcp_call(method, params):
    """Send JSON-RPC request to the MCP server."""
    request = json.dumps({
        "jsonrpc": "2.0",
        "id": 1,
        "method": method,
        "params": params
    })
    proc.stdin.write(request + "\n")
    proc.stdin.flush()
    response = json.loads(proc.stdout.readline())
    return response["result"]

# Fetch a page and extract as markdown

result = mcp_call(
    "get",
    {
        "url": "https://example.com",
        "extraction_type": "markdown",
        "main_content_only": True
    }
)

print(f"Status: {result['status']}")
print(f"Content: {result['content'][0]}")

The get tool is defined in ScraplingMCPServer.get at scrapling/core/ai.py【ai.py L59-L84】 and automatically registers with FastMCP during the serve initialization.

Best Practices for AI-Assisted Scraping

Based on the Scrapling source code and documentation, follow these guidelines for optimal results:

• Choose the right tool for your target site. Use get for static pages, fetch for JavaScript-rendered content, and stealthy_fetch for Cloudflare or anti-bot protected sites【mcp-server.md L61-L65】.

• Use bulk variants when processing multiple URLs. The bulk_get, bulk_fetch, and bulk_stealthy_fetch tools execute requests concurrently, reducing overall latency【mcp-server.md L66-L68】.

• Apply CSS selectors to limit extraction to relevant elements. This reduces token consumption and improves the accuracy of AI reasoning by removing navigation noise【mcp-server.md L24-L27】.

• Configure browser waits for dynamic sites. Use network_idle or wait_selector parameters when calling fetch or stealthy_fetch to ensure Single Page Application (SPA) content has loaded【mcp-server.md L72-L75】.

Key Implementation Files

Understanding the source structure helps with debugging and extending the MCP server:

File	Role	Location
scrapling/core/ai.py	Implements ScraplingMCPServer, registers tools, defines ResponseModel	View on GitHub
scrapling/cli.py	Exposes scrapling mcp command, parses --http, --host, --port flags	View on GitHub
docs/ai/mcp-server.md	User documentation for installation and configuration	View on GitHub
scrapling/engines/toolbelt/custom.py	Defines low-level Response objects used by the extraction engine	View on GitHub

Summary

Setting up the MCP server for AI-assisted web scraping with Scrapling involves four key steps:

• Install Scrapling with AI extras using pip install "scrapling[ai]" and run scrapling install to download browser binaries.
• Start the server via scrapling mcp (stdio mode) or scrapling mcp --http --host 127.0.0.1 --port 8000 (HTTP mode).
• Configure your AI client (Claude Desktop, Cursor, etc.) by adding the server to your MCP configuration file with the appropriate command and arguments.
• Prompt the AI to use tools like get, fetch, or stealthy_fetch with parameters such as css_selector and extraction_type to receive structured JSON responses instead of raw HTML.

Frequently Asked Questions

What is the difference between get and fetch in the Scrapling MCP server?

The get tool performs fast HTTP GET requests ideal for static HTML pages, while fetch uses a headless browser to render JavaScript-heavy content. According to the source code in scrapling/core/ai.py, fetch supports additional parameters like network_idle and wait_selector to handle dynamic Single Page Applications (SPAs).

Can I use the Scrapling MCP server with Claude Code or only Claude Desktop?

The Scrapling MCP server works with any MCP-compatible client, including Claude Desktop, Claude Code, Cursor, WindSurf, and other AI assistants that support the Model Context Protocol. You configure each client by pointing to the scrapling mcp command in their respective MCP configuration files, using either stdio mode for local integration or HTTP mode for remote connections.

How do I handle anti-bot protection like Cloudflare when using the MCP server?

Use the stealthy_fetch or bulk_stealthy_fetch tools instead of standard fetch. These tools implement anti-detection browser techniques specifically designed to bypass Cloudflare and similar protections. As documented in the MCP server guide, stealth mode should be reserved for protected sites since it incurs higher resource overhead than standard fetching methods.

Is it possible to run the MCP server in a Docker container or remote server?

Yes, you can deploy the Scrapling MCP server remotely by using the --http flag to enable streamable-HTTP transport instead of the default stdio mode. Start the server with scrapling mcp --http --host 0.0.0.0 --port 8000 and configure your AI client to connect via HTTP to the container or remote host address. This setup is particularly useful for centralized scraping services or when running Scrapling in cloud environments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}