# How to Use Scrapling's Interactive Shell for Testing and Debugging

> Test and debug scrapers faster with Scrapling's interactive shell. Prototype easily using its IPython REPL with auto-injected fetchers and selectors. No full scripts needed.

- Repository: [Karim shoair/Scrapling](https://github.com/D4Vinci/Scrapling)
- Tags: how-to-guide
- Published: 2026-03-08

---

**Scrapling's interactive shell provides an IPython-based REPL that automatically injects fetchers, selectors, and curl-to-code converters into your namespace, letting you prototype and debug scrapers without writing full scripts.**

Scrapling's interactive shell is a powerful debugging environment shipped with the D4Vinci/Scrapling repository. It eliminates the traditional "write-run-repeat" cycle by giving you instant access to HTTP fetchers, CSS selectors, and session management tools directly in an interactive prompt. Whether you're reverse-engineering a complex API or testing extraction logic on a single page, this shell accelerates your workflow by maintaining state between commands and automatically tracking your request history.

## Starting Scrapling's Interactive Shell

The shell entry point is registered in [`scrapling/cli.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/cli.py) as the `shell` subcommand. When you execute `scrapling shell`, the CLI instantiates the `CustomShell` class from [`scrapling/core/shell.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/shell.py) and launches an embedded IPython terminal.

Launch the REPL with default settings:

```bash
scrapling shell

```

Run a one-off command and exit immediately:

```bash
scrapling shell -c "get('https://quotes.toscrape.com'); print(len(page.css('.quote')))"

```

Adjust logging verbosity for debugging HTTP details:

```bash
scrapling shell --loglevel debug

```

When the shell initializes, you’ll see a startup banner and the standard `>>>` prompt. No import statements are required—the namespace is pre-populated with Scrapling's core primitives.

## Built-in Helpers and Variables

The `CustomShell.get_namespace` method in [`scrapling/core/shell.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/shell.py) constructs the interactive environment by wrapping fetcher functions and injecting helper utilities. These wrappers automatically save results to the `page` variable and maintain a rolling history in `pages`.

### Automatic Variables

- **`page`** – Contains the most recent `Response` or `Selector` object returned by any fetcher call.
- **`pages`** – A `Selectors` collection holding the last five fetched pages, allowing you to compare previous responses.
- **`response`** – Alias for the raw response object of the current page.

### Fetcher Shortcuts

- **`get`**, **`post`**, **`put`**, **`delete`** – Wrapped methods of the standard `Fetcher` class.
- **`fetch`** – Shortcut for `DynamicFetcher`, enabling headless browser automation.
- **`stealthy_fetch`** – Shortcut for `StealthyFetcher`, using stealth plugins to bypass bot detection.

### Debugging Utilities

- **`view(page)`** – Opens the current page HTML in your default browser via a temporary file (automatically cleaned up on close).
- **`uncurl(curl_string)`** – Parses a raw curl command into a `Request` named-tuple using the `CurlParser` class.
- **`curl2fetcher(curl_string)`** – Parses the curl command and immediately executes it, updating `page` with the result.

## Practical Debugging Workflows

The following examples demonstrate how to leverage Scrapling's interactive shell for common testing and debugging scenarios. All code blocks assume you have already started the shell with `scrapling shell`.

### Basic GET Request and CSS Extraction

Test selector logic against live HTML without writing a script:

```python
>>> get('https://news.ycombinator.com')
>>> titles = page.css('.titleline>a::text')
>>> for t in titles[:5]:
...     print(t)

```

The `get` wrapper automatically stores the response in `page`, allowing immediate CSS selection using Scrapling's selector syntax.

### Session-Based Crawling for Efficiency

Debug multi-page crawls using `FetcherSession` to maintain cookies and connection pooling:

```python
>>> from scrapling.fetchers import FetcherSession
>>> with FetcherSession() as sess:
...     catalog = sess.get('https://quotes.toscrape.com')
...     links = catalog.css('.quote a::attr(href)')
...     for link in links[:3]:
...         page = sess.get(f"https://quotes.toscrape.com{link}")
...         print(page.css('.author::text').get())

```

The session reuses the underlying HTTP connection pool, making your interactive debugging faster while preserving state across requests.

### Dynamic Content and Headless Browsing

Test JavaScript-rendered pages using the dynamic fetcher:

```python
>>> fetch('https://example.com/slow-page', headless=False, wait=2000)
>>> page.css('#dynamic-content::text').get()

```

`fetch` wraps `DynamicFetcher`, allowing you to toggle `headless` mode, set `wait` times for rendering, or use `stealthy_fetch` for sites with bot protection.

### Converting Browser curl Commands

Reverse-engineer API calls from browser DevTools:

```python
>>> curl_cmd = (
...  "curl 'https://httpbin.org/post' -X POST "
...  "-H 'Content-Type: application/json' "
...  "-d '{\"name\":\"Alice\",\"age\":30}'"
... )
>>> request = uncurl(curl_cmd)
>>> request.method, request.url
('post', 'https://httpbin.org/post')
>>> curl2fetcher(curl_cmd)
>>> page.json()
{'json': {'name': 'Alice', 'age': 30}, ...}

```

The `CurlParser` class in [`scrapling/core/shell.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/shell.py) tokenizes the command using `shlex_split`, extracts headers, cookies, and payloads, and returns a `Request` named-tuple compatible with Scrapling's fetcher signatures.

### Inspecting Request History

Debug failures by comparing previous responses:

```python
>>> len(pages)          # Check how many pages are stored (max 5)

>>> pages[0].url        # URL of the first page in history

>>> for i, p in enumerate(pages):
...     print(f"{i}: {p.url} ({p.status})")

```

The `pages` variable maintains a rolling buffer of the last five requests, allowing you to step back through your debugging session without re-fetching.

## Core Implementation Details

Understanding the architecture of Scrapling's interactive shell helps you extend its functionality or debug issues with the REPL itself.

The `CustomShell` class in **[`scrapling/core/shell.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/shell.py)** orchestrates the environment. Its `get_namespace` method (lines 90-105) constructs wrapper functions via `create_wrapper` that preserve the original fetcher signatures using `_unpack_signature`. These wrappers automatically invoke `update_page` to synchronize the `page`, `response`, and `pages` variables with the IPython user namespace.

The **`CurlParser`** class handles the `uncurl` and `curl2fetcher` utilities. It uses `shlex_split` to safely tokenize curl commands, then extracts HTTP method, URL, headers, cookies, data payloads, and proxy settings into a `Request` named-tuple that matches the parameter signatures of Scrapling's fetcher methods.

The shell launcher resides in **[`scrapling/cli.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/cli.py)** (lines 63-86), where the `shell` subcommand is registered using Click. It accepts optional `-c` arguments for code execution and `--loglevel` for verbosity control, forwarding these to the `CustomShell` instance.

## Summary

- **Scrapling's interactive shell** provides an IPython-based REPL pre-loaded with fetchers, selectors, and debugging utilities, eliminating the need for repetitive script writing during prototyping.
- Launch the shell with `scrapling shell`, execute one-off commands with `-c`, or adjust verbosity via `--loglevel`.
- The environment automatically tracks your scraping history in the `pages` variable (last 5 requests) and exposes the current response as `page`.
- Use `uncurl` and `curl2fetcher` to convert browser DevTools curl commands directly into Scrapling requests for rapid API reverse-engineering.
- Access `view(page)` to open responses in your default browser, and leverage `FetcherSession` for efficient multi-page debugging with connection pooling.

## Frequently Asked Questions

### How do I exit Scrapling's interactive shell?

Press **Ctrl+D** (Unix/Linux/Mac) or **Ctrl+Z** then Enter (Windows) to exit the IPython session and return to your system terminal. You can also type `exit()` or `quit()` at the `>>>` prompt.

### Can I save my interactive session to a Python script?

Yes. IPython provides the `%save` magic command. After prototyping your scraper, run `%save my_scraper.py 1-20` to save lines 1 through 20 of your session history to a file. You can then edit [`my_scraper.py`](https://github.com/D4Vinci/Scrapling/blob/main/my_scraper.py) to clean up the code for production use.

### Why does the `pages` variable only store the last five requests?

The `CustomShell` class in [`scrapling/core/shell.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/shell.py) implements a rolling buffer via the `update_page` method to prevent memory bloat during long debugging sessions. If you need to persist more responses, manually assign them to your own variables (e.g., `saved_page = page`) or use `FetcherSession` to manage state programmatically.

### How do I debug HTTP headers when using `curl2fetcher`?

First, use `uncurl(curl_string)` instead of `curl2fetcher` to inspect the parsed `Request` named-tuple without executing the request. This shows you the extracted method, URL, headers, cookies, and payload. Once verified, execute the request with `curl2fetcher` or manually pass the components to `get`/`post` with the `headers=` parameter for fine-grained control.