How to Use Scrapling's Interactive Shell for Testing and Debugging

Scrapling's interactive shell provides an IPython-based REPL that automatically injects fetchers, selectors, and curl-to-code converters into your namespace, letting you prototype and debug scrapers without writing full scripts.

Scrapling's interactive shell is a powerful debugging environment shipped with the D4Vinci/Scrapling repository. It eliminates the traditional "write-run-repeat" cycle by giving you instant access to HTTP fetchers, CSS selectors, and session management tools directly in an interactive prompt. Whether you're reverse-engineering a complex API or testing extraction logic on a single page, this shell accelerates your workflow by maintaining state between commands and automatically tracking your request history.

Starting Scrapling's Interactive Shell

The shell entry point is registered in scrapling/cli.py as the shell subcommand. When you execute scrapling shell, the CLI instantiates the CustomShell class from scrapling/core/shell.py and launches an embedded IPython terminal.

Launch the REPL with default settings:

scrapling shell

Run a one-off command and exit immediately:

scrapling shell -c "get('https://quotes.toscrape.com'); print(len(page.css('.quote')))"

Adjust logging verbosity for debugging HTTP details:

scrapling shell --loglevel debug

When the shell initializes, you’ll see a startup banner and the standard >>> prompt. No import statements are required—the namespace is pre-populated with Scrapling's core primitives.

Built-in Helpers and Variables

The CustomShell.get_namespace method in scrapling/core/shell.py constructs the interactive environment by wrapping fetcher functions and injecting helper utilities. These wrappers automatically save results to the page variable and maintain a rolling history in pages.

Automatic Variables

• page – Contains the most recent Response or Selector object returned by any fetcher call.
• pages – A Selectors collection holding the last five fetched pages, allowing you to compare previous responses.
• response – Alias for the raw response object of the current page.

Fetcher Shortcuts

• get, post, put, delete – Wrapped methods of the standard Fetcher class.
• fetch – Shortcut for DynamicFetcher, enabling headless browser automation.
• stealthy_fetch – Shortcut for StealthyFetcher, using stealth plugins to bypass bot detection.

Debugging Utilities

• view(page) – Opens the current page HTML in your default browser via a temporary file (automatically cleaned up on close).
• uncurl(curl_string) – Parses a raw curl command into a Request named-tuple using the CurlParser class.
• curl2fetcher(curl_string) – Parses the curl command and immediately executes it, updating page with the result.

Practical Debugging Workflows

The following examples demonstrate how to leverage Scrapling's interactive shell for common testing and debugging scenarios. All code blocks assume you have already started the shell with scrapling shell.

Basic GET Request and CSS Extraction

Test selector logic against live HTML without writing a script:

>>> get('https://news.ycombinator.com')
>>> titles = page.css('.titleline>a::text')
>>> for t in titles[:5]:
...     print(t)

The get wrapper automatically stores the response in page, allowing immediate CSS selection using Scrapling's selector syntax.

Session-Based Crawling for Efficiency

Debug multi-page crawls using FetcherSession to maintain cookies and connection pooling:

>>> from scrapling.fetchers import FetcherSession
>>> with FetcherSession() as sess:
...     catalog = sess.get('https://quotes.toscrape.com')
...     links = catalog.css('.quote a::attr(href)')
...     for link in links[:3]:
...         page = sess.get(f"https://quotes.toscrape.com{link}")
...         print(page.css('.author::text').get())

The session reuses the underlying HTTP connection pool, making your interactive debugging faster while preserving state across requests.

Dynamic Content and Headless Browsing

Test JavaScript-rendered pages using the dynamic fetcher:

>>> fetch('https://example.com/slow-page', headless=False, wait=2000)
>>> page.css('#dynamic-content::text').get()

fetch wraps DynamicFetcher, allowing you to toggle headless mode, set wait times for rendering, or use stealthy_fetch for sites with bot protection.

Converting Browser curl Commands

Reverse-engineer API calls from browser DevTools:

>>> curl_cmd = (
...  "curl 'https://httpbin.org/post' -X POST "
...  "-H 'Content-Type: application/json' "
...  "-d '{\"name\":\"Alice\",\"age\":30}'"
... )
>>> request = uncurl(curl_cmd)
>>> request.method, request.url
('post', 'https://httpbin.org/post')
>>> curl2fetcher(curl_cmd)
>>> page.json()
{'json': {'name': 'Alice', 'age': 30}, ...}

The CurlParser class in scrapling/core/shell.py tokenizes the command using shlex_split, extracts headers, cookies, and payloads, and returns a Request named-tuple compatible with Scrapling's fetcher signatures.

Inspecting Request History

Debug failures by comparing previous responses:

>>> len(pages)          # Check how many pages are stored (max 5)

>>> pages[0].url        # URL of the first page in history

>>> for i, p in enumerate(pages):
...     print(f"{i}: {p.url} ({p.status})")

The pages variable maintains a rolling buffer of the last five requests, allowing you to step back through your debugging session without re-fetching.

Core Implementation Details

Understanding the architecture of Scrapling's interactive shell helps you extend its functionality or debug issues with the REPL itself.

The CustomShell class in scrapling/core/shell.py orchestrates the environment. Its get_namespace method (lines 90-105) constructs wrapper functions via create_wrapper that preserve the original fetcher signatures using _unpack_signature. These wrappers automatically invoke update_page to synchronize the page, response, and pages variables with the IPython user namespace.

The CurlParser class handles the uncurl and curl2fetcher utilities. It uses shlex_split to safely tokenize curl commands, then extracts HTTP method, URL, headers, cookies, data payloads, and proxy settings into a Request named-tuple that matches the parameter signatures of Scrapling's fetcher methods.

The shell launcher resides in scrapling/cli.py (lines 63-86), where the shell subcommand is registered using Click. It accepts optional -c arguments for code execution and --loglevel for verbosity control, forwarding these to the CustomShell instance.

Summary

• Scrapling's interactive shell provides an IPython-based REPL pre-loaded with fetchers, selectors, and debugging utilities, eliminating the need for repetitive script writing during prototyping.
• Launch the shell with scrapling shell, execute one-off commands with -c, or adjust verbosity via --loglevel.
• The environment automatically tracks your scraping history in the pages variable (last 5 requests) and exposes the current response as page.
• Use uncurl and curl2fetcher to convert browser DevTools curl commands directly into Scrapling requests for rapid API reverse-engineering.
• Access view(page) to open responses in your default browser, and leverage FetcherSession for efficient multi-page debugging with connection pooling.

Frequently Asked Questions

How do I exit Scrapling's interactive shell?

Press Ctrl+D (Unix/Linux/Mac) or Ctrl+Z then Enter (Windows) to exit the IPython session and return to your system terminal. You can also type exit() or quit() at the >>> prompt.

Can I save my interactive session to a Python script?

Yes. IPython provides the %save magic command. After prototyping your scraper, run %save my_scraper.py 1-20 to save lines 1 through 20 of your session history to a file. You can then edit my_scraper.py to clean up the code for production use.

Why does the pages variable only store the last five requests?

The CustomShell class in scrapling/core/shell.py implements a rolling buffer via the update_page method to prevent memory bloat during long debugging sessions. If you need to persist more responses, manually assign them to your own variables (e.g., saved_page = page) or use FetcherSession to manage state programmatically.

How do I debug HTTP headers when using curl2fetcher?

First, use uncurl(curl_string) instead of curl2fetcher to inspect the parsed Request named-tuple without executing the request. This shows you the extracted method, URL, headers, cookies, and payload. Once verified, execute the request with curl2fetcher or manually pass the components to get/post with the headers= parameter for fine-grained control.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}