How Scrapling's Adaptive Storage System Relocates Elements After Website Updates

Scrapling's adaptive storage system persists structural fingerprints of HTML elements to an SQLite database, then uses a weighted similarity algorithm to automatically relocate those elements when DOM structures change, returning the best-matching nodes even if the original CSS selectors no longer match.

Web scraping scripts frequently break when websites update their layouts, but the D4Vinci/Scrapling library solves this through an innovative adaptive storage system that tracks elements across page revisions. By combining persistent storage with intelligent similarity scoring, Scrapling can relocate elements even when their original selectors fail to match the updated DOM structure.

Understanding the Adaptive Storage Architecture

The adaptive storage system centers on the Selector class in scrapling/parser.py, which integrates with a pluggable storage backend—defaulting to SQLiteStorageSystem from scrapling/core/storage.py. When adaptive mode is enabled, the system captures a structural fingerprint of each target element, including tag name, text content, attributes, and DOM path information. This fingerprint persists to disk, creating a historical record that survives between scraping sessions.

The Relocation Pipeline: Step-by-Step

Enabling Adaptive Mode and Initializing Storage

Adaptive functionality activates during Selector instantiation. In Selector.__init__ (parser.py#L62-L71), the constructor checks the adaptive argument and initializes self.__adaptive_enabled. It also instantiates the storage system, creating the SQLite database connection if one doesn't exist.

Saving Element Fingerprints to SQLite

Before relocation can occur, the system must store baseline element data. The Selector.save method (parser.py#L63-L79) accepts an element and identifier, then delegates to _StorageTools.element_to_dict to serialize the HtmlElement into a comparable dictionary. This dictionary writes to the database via SQLiteStorageSystem.save (storage.py#L9-L26), preserving tag names, attributes, text content, and structural metadata.

Querying and Detecting Structural Changes

When executing a query with selector.css(..., adaptive=True), the system first attempts standard CSS selection. If the XPath evaluation returns no nodes and adaptive mode is active, the trigger occurs in Selector.xpath (parser.py#L50-L62). This detection mechanism identifies when the original DOM structure has changed sufficiently to break existing selectors.

Retrieving Stored Fingerprints

Upon detecting a structural mismatch, the system fetches the historical fingerprint. The Selector.retrieve method calls SQLiteStorageSystem.retrieve (storage.py#L30-L44), which queries the SQLite database by identifier and returns the previously saved dictionary containing the element's structural characteristics.

Scoring and Relocating Elements

The core relocation logic resides in Selector.relocate (parser.py#L496-L520). This method iterates over every node in the new DOM via _find_all_elements(self._root), computing a similarity score for each candidate against the stored fingerprint. The algorithm returns nodes meeting the specified percentage threshold, wrapped in a Selectors collection for further manipulation.

Deep Dive into the Similarity Algorithm

The intelligence behind element relocation stems from __calculate_similarity_score (parser.py#L889-L954). This private method implements a weighted multi-factor comparison that evaluates:

• Tag name matching
• Text content similarity using difflib.SequenceMatcher
• Attribute sets comparison via custom dictionary differencing
• Class, ID, href, and src attributes (weighted heavily)
• DOM path and parent node characteristics
• Sibling relationships and structural position

The algorithm normalizes these factors into a unified percentage score, allowing the system to identify the same logical element even when superficial attributes change but core content remains consistent.

Practical Implementation Example

The following example demonstrates the complete workflow: saving an element from an original page structure, then relocating it after the DOM changes:

from scrapling import Selector

# 1️⃣ Load the original page and save the target element

orig_html = "<html><body><p id='price' data-id='p1'>USD 10</p></body></html>"
sel = Selector(orig_html, adaptive=True)               # adaptive enabled

price_elem = sel.css("#price", identifier="price_id")  # locate element

# Store its fingerprint for later use

sel.save(price_elem[0], "price_id")                    # <-- saved in SQLite DB

# 2️⃣ Later, after the site layout changed

new_html = """
<html><body>
  <div class='product'>
    <p class='new-price' data-id='p1'>USD 10</p>   <!-- moved, new class -->
  </div>
</body></html>
"""

# Use the same selector, but ask it to adapt if the original selector fails

sel2 = Selector(new_html, adaptive=True)
price_new = sel2.css("#price", adaptive=True, identifier="price_id")
print(price_new[0].attrib["data-id"])   # → "p1"

print(price_new[0].has_class("new-price"))  # → True (relocated)

In this scenario, the original selector #price fails on the updated HTML because the id attribute no longer exists. However, because adaptive=True and the identifier "price_id" were provided, Scrapling retrieves the stored fingerprint, scores all paragraph elements in the new DOM, and correctly identifies the relocated price element by matching its data-id attribute and text content.

Key Source Files and Components

The adaptive storage system spans several modules within the D4Vinci/Scrapling repository:

• scrapling/parser.py – Contains the Selector class with adaptive logic, save, relocate, and __calculate_similarity_score methods.
• scrapling/core/storage.py – Implements SQLiteStorageSystem and the StorageSystemMixin interface for persisting element fingerprints.
• scrapling/core/_utils.py – Provides _StorageTools.element_to_dict for serializing HTML elements into comparable dictionaries.
• scrapling/core/mixins.py – Defines SelectorsGeneration and the Selectors collection class used to wrap relocated elements.

Summary

• Scrapling's adaptive storage system combines the Selector class with an SQLite backend to persist element structural fingerprints across scraping sessions.
• When a CSS selector fails due to DOM changes, the system automatically triggers relocation mode, retrieving stored fingerprints and scoring all nodes in the new document.
• The similarity algorithm in __calculate_similarity_score weighs tag names, text content, attributes, and DOM position to identify the best match.
• Developers enable this functionality by passing adaptive=True during Selector instantiation or individual queries, using identifier parameters to link queries with stored records.

Frequently Asked Questions

What storage backends does Scrapling's adaptive system support?

By default, Scrapling uses SQLiteStorageSystem from scrapling/core/storage.py, which persists fingerprints to a local SQLite database file. The architecture uses a mixin pattern (StorageSystemMixin), allowing developers to implement custom backends (such as Redis or PostgreSQL) by subclassing the mixin and overriding the save and retrieve methods.

How does the similarity scoring algorithm handle text content changes?

The __calculate_similarity_score method in parser.py uses difflib.SequenceMatcher to compute string similarity ratios for text content. This allows the algorithm to recognize elements even when text undergoes minor modifications (such as price updates or formatting changes), assigning partial credit based on character-level similarity rather than requiring exact matches.

Can I use adaptive mode without explicitly saving elements first?

No, adaptive relocation requires a stored fingerprint to compare against. The system triggers relocation only when a query includes an identifier parameter that corresponds to a previously saved record. If you attempt to use adaptive=True without a matching stored identifier, the query will simply return empty results when the selector fails, as there is no historical data to facilitate relocation.

What performance impact does the relocation algorithm have?

The relocation process in Selector.relocate iterates over every node in the DOM (_find_all_elements(self._root)) and computes similarity scores for each candidate. For large documents, this O(n) traversal with complex string comparisons can introduce measurable latency. However, the system allows tuning via the percentage parameter, which sets a minimum similarity threshold and can short-circuit processing once a sufficiently high-scoring match is found.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}