# How Adaptive Scraping Handles Website Structure Changes in Scrapling

> Discover how Scrapling's adaptive scraping automatically handles website structure changes by storing element signatures and using similarity scoring to match them on new pages.

- Repository: [Karim shoair/Scrapling](https://github.com/D4Vinci/Scrapling)
- Tags: how-to-guide
- Published: 2026-03-08

---

**Scrapling's adaptive scraping automatically relocates elements when website layouts change by storing element signatures and matching them against new pages using similarity scoring.**

Adaptive scraping in the [Scrapling](https://github.com/D4Vinci/Scrapling) library provides a robust solution for maintaining data extraction pipelines when target websites modify their HTML structure. By recording unique element properties and employing rule-based matching algorithms, Scrapling ensures your selectors remain functional across site updates without requiring manual intervention or external AI services.

## How Adaptive Scraping Works in Scrapling

When you enable **adaptive mode** on a `Selector` or globally on a `Fetcher`, Scrapling creates a lightweight signature for each element the first time it is selected. This signature captures the element's tag name, attributes, text content, and CSS path, then stores it in a local SQLite database by default.

If a subsequent scraping run finds that the original selector no longer matches any elements—indicating the page structure has changed—Scrapling executes a three-step recovery process:

1. **Retrieve** the stored element signature from the SQLite backend.
2. **Score** every element on the current page against the saved signature, calculating similarity based on attributes, text content, and structural cues.
3. **Return** the element with the highest similarity score that exceeds the configurable `percentage` threshold. If no element meets the minimum similarity requirement, Scrapling returns an empty result.

This matching process is entirely **rule-based** and operates locally without dependencies on external AI services or cloud APIs.

## Enabling Adaptive Scraping in Your Code

Scrapling offers flexible configuration options for implementing adaptive scraping at different scopes within your extraction pipeline.

### Per-Selector Configuration

You can enable adaptive scraping for individual selector calls by passing `adaptive=True`. The first execution with `auto_save=True` records the element signature, while subsequent calls automatically attempt recovery if the selector fails.

```python
from scrapling import Selector

# Initialize with adaptive mode enabled

page = Selector(html_source, adaptive=True, url="example.com")

# First run – automatically saves element signature

button = page.css("#submit", adaptive=True, auto_save=True)[0]

# Later, after DOM changes, the same selector call still works

button = page.css("#submit", adaptive=True)[0]  # Finds best match

```

### Global Fetcher Configuration

For consistent adaptive behavior across all requests, configure the `Fetcher` class globally before instantiating selectors:

```python
from scrapling import Fetcher, fetch

# Enable adaptive scraping for all fetchers

Fetcher.adaptive = True

# All subsequent fetches return Selectors with adaptive enabled

page = fetch("https://example.com")

# Locate elements even if original XPath vanishes

link = page.xpath("//a[text()='Contact']", adaptive=True)[0]

```

### Domain-Scoped Adaptive Scraping

To prevent cross-contamination of element signatures between different websites, use the `adaptive_domain` argument. This isolates storage namespaces per domain, ensuring that similar selectors on different sites do not interfere with each other:

```python

# Isolate adaptive data for specific domain

Fetcher.configure(adaptive=True, adaptive_domain="example.com")

# Fetch from archive or mirror while using original domain's signatures

page = fetch("https://archive.org/web/20200101/http://example.com")

```

## The Technical Implementation

The adaptive scraping functionality is implemented primarily in [[`scrapling/parser.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/parser.py)](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/parser.py), which defines the `Selector` class and its adaptive capabilities.

The core logic relies on the `__adaptive_enabled` flag to determine whether adaptive features should be active. Guard clauses throughout the parser enforce this check, ensuring that when `adaptive=False` is set on a `Selector` instance, all adaptive-related arguments are ignored and normal static selection is performed. These guard clauses appear around **line 555** in the source file.

The actual adaptive matching logic—including signature retrieval, similarity scoring, and threshold validation—is implemented between **lines 870-891** in [`scrapling/parser.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/parser.py). This section handles the comparison of stored element signatures against the current DOM and implements the percentage-based threshold system.

The storage backend interface is defined to support SQLite by default, with the ability to swap in alternative storage implementations. Element signatures are serialized and stored with unique identifiers that can be scoped to specific domains via the `adaptive_domain` parameter.

Comprehensive test coverage for the adaptive functionality is available in [[`tests/parser/test_adaptive.py`](https://github.com/D4Vinci/Scrapling/blob/main/tests/parser/test_adaptive.py)](https://github.com/D4Vinci/Scrapling/blob/main/tests/parser/test_adaptive.py), which validates that selectors correctly relocate elements after simulated DOM changes and that domain isolation works as expected.

## Summary

- **Adaptive scraping** in Scrapling maintains selector functionality across website structure changes by storing and matching element signatures.
- The system captures element properties (tag, attributes, text, CSS path) in SQLite and uses rule-based similarity scoring to find matches when original selectors fail.
- Enable adaptive mode globally via `Fetcher.adaptive = True` or per-selector with `adaptive=True`, using `auto_save=True` to record initial signatures.
- Domain scoping via `adaptive_domain` prevents cross-site contamination of element signatures.
- The implementation resides in [`scrapling/parser.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/parser.py) with guard clauses at line 555 and matching logic at lines 870-891, requiring no external AI services.

## Frequently Asked Questions

### Does adaptive scraping require external AI services?

No. Scrapling's adaptive scraping operates entirely through local, rule-based algorithms. The system compares stored element signatures against the current DOM using similarity scoring based on attributes, text content, and structural cues. No calls to external AI APIs, machine learning models, or cloud services are required, making it suitable for air-gapped environments and privacy-conscious scraping operations.

### How does Scrapling store element signatures for adaptive matching?

Element signatures are stored in a lightweight SQLite database by default. When you enable adaptive mode and use `auto_save=True` (or manually call `storage.save()`), Scrapling serializes the element's tag name, attributes, text content, and CSS path into a signature record. This data persists between scraping sessions, allowing the adaptive matcher to retrieve historical element properties when the current selector fails to find matches.

### Can I use adaptive scraping with XPath selectors?

Yes. Adaptive scraping works with any selection method supported by Scrapling, including `css()`, `xpath()`, and other selector types. The adaptive system operates at the element level after selection occurs, meaning it can capture and later match elements regardless of whether you originally located them via CSS selectors, XPath expressions, or other parsing methods. Simply pass `adaptive=True` to your preferred selection method.

### What happens if no element meets the similarity threshold?

If no element on the current page achieves a similarity score exceeding the configurable `percentage` threshold, Scrapling returns an empty result for that selector call. The system does not guess or return low-confidence matches when the threshold is not met. You can adjust the threshold percentage when calling the selector (e.g., `percentage=70`) to control the strictness of the matching based on your tolerance for structural changes.