# How to Set Up and Use the MCP Server for AI-Assisted Web Scraping with Scrapling

> Learn to set up and use the MCP server for AI-assisted web scraping with Scrapling. Extract web data efficiently using structured scraping tools.

- Repository: [Karim shoair/Scrapling](https://github.com/D4Vinci/Scrapling)
- Tags: how-to-guide
- Published: 2026-03-08

---

**Install Scrapling with AI extras, start the MCP server via `scrapling mcp`, and configure your AI client to call structured scraping tools like `get`, `fetch`, and `stealthy_fetch` for token-efficient web data extraction.**

The **Scrapling MCP server** exposes the library's scraping capabilities as structured tools that any Model Context Protocol (MCP) compatible AI assistant can invoke. According to the D4Vinci/Scrapling source code, this integration allows AI systems to perform AI-assisted web scraping with Scrapling by calling specific functions that return concise, structured JSON instead of raw HTML, dramatically reducing token consumption and improving reliability.

## What Is the Scrapling MCP Server?

The MCP server implementation in [`scrapling/core/ai.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/ai.py) wraps Scrapling's scraping engines behind a FastMCP interface. When you run `scrapling mcp`, the `ScraplingMCPServer.serve` method instantiates the server and registers six core tools: `get`, `bulk_get`, `fetch`, `bulk_fetch`, `stealthy_fetch`, and `bulk_stealthy_fetch`【ai.py L59-L106】.

Each tool returns a `ResponseModel` Pydantic object containing `status`, `content` (a list of extracted strings), and `url`【ai.py L32-L38】. This structured output ensures AI assistants receive clean, parseable data without HTML noise.

## Installation Requirements

Before starting the MCP server for AI-assisted web scraping with Scrapling, install the package with AI extras and download the required browser binaries:

```bash

# Install Scrapling with MCP dependencies (FastMCP and server entry-point)

pip install "scrapling[ai]"

# Install Playwright browsers required for dynamic scraping

scrapling install

```

These steps are documented in the official MCP server guide at [`docs/ai/mcp-server.md`](https://github.com/D4Vinci/Scrapling/blob/main/docs/ai/mcp-server.md)【mcp-server.md L39-L48】.

## Starting the MCP Server

The server supports two transport modes: stdio (default for local AI clients) and streamable-HTTP (for remote or containerized deployments).

### Stdio Mode (Default)

Run the server in stdio mode for direct integration with Claude Desktop and other local MCP clients:

```bash
scrapling mcp

```

This command invokes `ScraplingMCPServer().serve(False, "0.0.0.0", 8000)` where the first argument `False` disables HTTP mode【cli.py L56-L61】.

### Streamable-HTTP Mode

For remote access or Docker deployments, start the server with HTTP transport:

```bash
scrapling mcp --http --host 127.0.0.1 --port 8000

```

The `--http` flag switches the transport to "streamable-http" as implemented in [`scrapling/cli.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/cli.py)【cli.py L43-L55】. You can customize the bind address using `--host` and `--port` arguments.

## Configuring AI Clients

Once the server is running, register it with your AI assistant using the Model Context Protocol configuration format.

### Claude Desktop Configuration

Add the following entry to your [`claude_desktop_config.json`](https://github.com/D4Vinci/Scrapling/blob/main/claude_desktop_config.json) file to enable AI-assisted web scraping with Scrapling:

```json
{
  "mcpServers": {
    "ScraplingServer": {
      "command": "/usr/local/bin/scrapling",
      "args": ["mcp"]
    }
  }
}

```

For HTTP mode, replace the command with a wrapper script or Docker container that invokes `scrapling mcp --http --host 127.0.0.1 --port 8000`【mcp-server.md L13-L24】.

After saving the configuration and restarting Claude Desktop, a wrench icon (🔧) will appear in the interface, confirming the server connection.

### Other MCP-Compatible Tools

The same configuration format works for Cursor, WindSurf, and other MCP-compatible editors. Simply adjust the `"command"` and `"args"` fields to point to your Scrapling installation and desired transport mode.

## Available Scraping Tools

The MCP server exposes six structured tools for different scraping scenarios:

- **`get`** – Fast HTTP GET requests for static content
- **`bulk_get`** – Concurrent GET requests for multiple URLs
- **`fetch`** – Browser-based fetching for JavaScript-rendered pages
- **`bulk_fetch`** – Concurrent browser fetching for multiple URLs
- **`stealthy_fetch`** – Stealth browser mode for anti-bot protected sites (Cloudflare, etc.)
- **`bulk_stealthy_fetch`** – Concurrent stealth fetching for multiple URLs

Each tool accepts parameters like `url`, `css_selector`, `extraction_type` (markdown/html/text), and browser-specific options like `network_idle` or `wait_selector`.

## Practical Usage Examples

### Basic Web Scraping Prompts

Once configured, prompt your AI assistant naturally to invoke the MCP server for AI-assisted web scraping with Scrapling:

**Example 1:** "Scrape the main content from https://example.com and convert it to markdown."

This invokes the `get` tool with `extraction_type="markdown"` and `main_content_only=True`.

**Example 2:** "Extract all product titles from https://shop.example.com using selector `.product-title`."

This calls `get` with `css_selector=".product-title"` to limit extraction to specific elements, saving tokens and improving accuracy.

**Example 3:** "Fetch these three pages concurrently and return their text."

This triggers `bulk_fetch` with a list of URLs and `extraction_type="text"` for parallel processing.

**Example 4:** "Bypass Cloudflare on https://protected.com and get the price."

This invokes `stealthy_fetch` to use anti-detection browser techniques for protected sites.

### Programmatic Access via Python

You can also interact with the MCP server programmatically using stdio communication:

```python
import json
import subprocess

# Start the server in stdio mode

proc = subprocess.Popen(
    ["scrapling", "mcp"],
    stdout=subprocess.PIPE,
    stdin=subprocess.PIPE,
    text=True
)

def mcp_call(method, params):
    """Send JSON-RPC request to the MCP server."""
    request = json.dumps({
        "jsonrpc": "2.0",
        "id": 1,
        "method": method,
        "params": params
    })
    proc.stdin.write(request + "\n")
    proc.stdin.flush()
    response = json.loads(proc.stdout.readline())
    return response["result"]

# Fetch a page and extract as markdown

result = mcp_call(
    "get",
    {
        "url": "https://example.com",
        "extraction_type": "markdown",
        "main_content_only": True
    }
)

print(f"Status: {result['status']}")
print(f"Content: {result['content'][0]}")

```

The `get` tool is defined in `ScraplingMCPServer.get` at [`scrapling/core/ai.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/ai.py)【ai.py L59-L84】 and automatically registers with FastMCP during the `serve` initialization.

## Best Practices for AI-Assisted Scraping

Based on the Scrapling source code and documentation, follow these guidelines for optimal results:

- **Choose the right tool** for your target site. Use `get` for static pages, `fetch` for JavaScript-rendered content, and `stealthy_fetch` for Cloudflare or anti-bot protected sites【mcp-server.md L61-L65】.

- **Use bulk variants** when processing multiple URLs. The `bulk_get`, `bulk_fetch`, and `bulk_stealthy_fetch` tools execute requests concurrently, reducing overall latency【mcp-server.md L66-L68】.

- **Apply CSS selectors** to limit extraction to relevant elements. This reduces token consumption and improves the accuracy of AI reasoning by removing navigation noise【mcp-server.md L24-L27】.

- **Configure browser waits** for dynamic sites. Use `network_idle` or `wait_selector` parameters when calling `fetch` or `stealthy_fetch` to ensure Single Page Application (SPA) content has loaded【mcp-server.md L72-L75】.

## Key Implementation Files

Understanding the source structure helps with debugging and extending the MCP server:

| File | Role | Location |
|------|------|----------|
| [`scrapling/core/ai.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/ai.py) | Implements `ScraplingMCPServer`, registers tools, defines `ResponseModel` | [View on GitHub](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/ai.py) |
| [`scrapling/cli.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/cli.py) | Exposes `scrapling mcp` command, parses `--http`, `--host`, `--port` flags | [View on GitHub](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/cli.py) |
| [`docs/ai/mcp-server.md`](https://github.com/D4Vinci/Scrapling/blob/main/docs/ai/mcp-server.md) | User documentation for installation and configuration | [View on GitHub](https://github.com/D4Vinci/Scrapling/blob/main/docs/ai/mcp-server.md) |
| [`scrapling/engines/toolbelt/custom.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/toolbelt/custom.py) | Defines low-level `Response` objects used by the extraction engine | [View on GitHub](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/engines/toolbelt/custom.py) |

## Summary

Setting up the MCP server for AI-assisted web scraping with Scrapling involves four key steps:

- **Install** Scrapling with AI extras using `pip install "scrapling[ai]"` and run `scrapling install` to download browser binaries.
- **Start** the server via `scrapling mcp` (stdio mode) or `scrapling mcp --http --host 127.0.0.1 --port 8000` (HTTP mode).
- **Configure** your AI client (Claude Desktop, Cursor, etc.) by adding the server to your MCP configuration file with the appropriate command and arguments.
- **Prompt** the AI to use tools like `get`, `fetch`, or `stealthy_fetch` with parameters such as `css_selector` and `extraction_type` to receive structured JSON responses instead of raw HTML.

## Frequently Asked Questions

### What is the difference between `get` and `fetch` in the Scrapling MCP server?

The `get` tool performs fast HTTP GET requests ideal for static HTML pages, while `fetch` uses a headless browser to render JavaScript-heavy content. According to the source code in [`scrapling/core/ai.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/ai.py), `fetch` supports additional parameters like `network_idle` and `wait_selector` to handle dynamic Single Page Applications (SPAs).

### Can I use the Scrapling MCP server with Claude Code or only Claude Desktop?

The Scrapling MCP server works with any MCP-compatible client, including Claude Desktop, Claude Code, Cursor, WindSurf, and other AI assistants that support the Model Context Protocol. You configure each client by pointing to the `scrapling mcp` command in their respective MCP configuration files, using either stdio mode for local integration or HTTP mode for remote connections.

### How do I handle anti-bot protection like Cloudflare when using the MCP server?

Use the `stealthy_fetch` or `bulk_stealthy_fetch` tools instead of standard `fetch`. These tools implement anti-detection browser techniques specifically designed to bypass Cloudflare and similar protections. As documented in the MCP server guide, stealth mode should be reserved for protected sites since it incurs higher resource overhead than standard fetching methods.

### Is it possible to run the MCP server in a Docker container or remote server?

Yes, you can deploy the Scrapling MCP server remotely by using the `--http` flag to enable streamable-HTTP transport instead of the default stdio mode. Start the server with `scrapling mcp --http --host 0.0.0.0 --port 8000` and configure your AI client to connect via HTTP to the container or remote host address. This setup is particularly useful for centralized scraping services or when running Scrapling in cloud environments.