# How Scrapling's Adaptive Storage System Relocates Elements After Website Updates > Learn how Scrapling's adaptive storage system relocates HTML elements after website updates using structural fingerprints and a weighted similarity algorithm. Get accurate matches even with DOM changes. - Repository: [Karim shoair/Scrapling](https://github.com/D4Vinci/Scrapling) - Tags: deep-dive - Published: 2026-03-08 --- **Scrapling's adaptive storage system persists structural fingerprints of HTML elements to an SQLite database, then uses a weighted similarity algorithm to automatically relocate those elements when DOM structures change, returning the best-matching nodes even if the original CSS selectors no longer match.** Web scraping scripts frequently break when websites update their layouts, but the D4Vinci/Scrapling library solves this through an innovative adaptive storage system that tracks elements across page revisions. By combining persistent storage with intelligent similarity scoring, Scrapling can relocate elements even when their original selectors fail to match the updated DOM structure. ## Understanding the Adaptive Storage Architecture The adaptive storage system centers on the `Selector` class in [`scrapling/parser.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/parser.py), which integrates with a pluggable storage backend—defaulting to `SQLiteStorageSystem` from [`scrapling/core/storage.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/storage.py). When **adaptive mode** is enabled, the system captures a structural fingerprint of each target element, including tag name, text content, attributes, and DOM path information. This fingerprint persists to disk, creating a historical record that survives between scraping sessions. ## The Relocation Pipeline: Step-by-Step ### Enabling Adaptive Mode and Initializing Storage Adaptive functionality activates during `Selector` instantiation. In `Selector.__init__` (`parser.py#L62-L71`), the constructor checks the `adaptive` argument and initializes `self.__adaptive_enabled`. It also instantiates the storage system, creating the SQLite database connection if one doesn't exist. ### Saving Element Fingerprints to SQLite Before relocation can occur, the system must store baseline element data. The `Selector.save` method (`parser.py#L63-L79`) accepts an element and identifier, then delegates to `_StorageTools.element_to_dict` to serialize the `HtmlElement` into a comparable dictionary. This dictionary writes to the database via `SQLiteStorageSystem.save` (`storage.py#L9-L26`), preserving tag names, attributes, text content, and structural metadata. ### Querying and Detecting Structural Changes When executing a query with `selector.css(..., adaptive=True)`, the system first attempts standard CSS selection. If the XPath evaluation returns no nodes and adaptive mode is active, the trigger occurs in `Selector.xpath` (`parser.py#L50-L62`). This detection mechanism identifies when the original DOM structure has changed sufficiently to break existing selectors. ### Retrieving Stored Fingerprints Upon detecting a structural mismatch, the system fetches the historical fingerprint. The `Selector.retrieve` method calls `SQLiteStorageSystem.retrieve` (`storage.py#L30-L44`), which queries the SQLite database by identifier and returns the previously saved dictionary containing the element's structural characteristics. ### Scoring and Relocating Elements The core relocation logic resides in `Selector.relocate` (`parser.py#L496-L520`). This method iterates over every node in the new DOM via `_find_all_elements(self._root)`, computing a similarity score for each candidate against the stored fingerprint. The algorithm returns nodes meeting the specified `percentage` threshold, wrapped in a `Selectors` collection for further manipulation. ## Deep Dive into the Similarity Algorithm The intelligence behind element relocation stems from `__calculate_similarity_score` (`parser.py#L889-L954`). This private method implements a **weighted multi-factor comparison** that evaluates: - **Tag name** matching - **Text content** similarity using `difflib.SequenceMatcher` - **Attribute sets** comparison via custom dictionary differencing - **Class, ID, href, and src** attributes (weighted heavily) - **DOM path** and parent node characteristics - **Sibling relationships** and structural position The algorithm normalizes these factors into a unified percentage score, allowing the system to identify the same logical element even when superficial attributes change but core content remains consistent. ## Practical Implementation Example The following example demonstrates the complete workflow: saving an element from an original page structure, then relocating it after the DOM changes: ```python from scrapling import Selector # 1️⃣ Load the original page and save the target element orig_html = "

USD 10

" sel = Selector(orig_html, adaptive=True) # adaptive enabled price_elem = sel.css("#price", identifier="price_id") # locate element # Store its fingerprint for later use sel.save(price_elem[0], "price_id") # <-- saved in SQLite DB # 2️⃣ Later, after the site layout changed new_html = """

USD 10

""" # Use the same selector, but ask it to adapt if the original selector fails sel2 = Selector(new_html, adaptive=True) price_new = sel2.css("#price", adaptive=True, identifier="price_id") print(price_new[0].attrib["data-id"]) # → "p1" print(price_new[0].has_class("new-price")) # → True (relocated) ``` In this scenario, the original selector `#price` fails on the updated HTML because the `id` attribute no longer exists. However, because `adaptive=True` and the identifier `"price_id"` were provided, Scrapling retrieves the stored fingerprint, scores all paragraph elements in the new DOM, and correctly identifies the relocated price element by matching its `data-id` attribute and text content. ## Key Source Files and Components The adaptive storage system spans several modules within the D4Vinci/Scrapling repository: - **[`scrapling/parser.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/parser.py)** – Contains the `Selector` class with adaptive logic, `save`, `relocate`, and `__calculate_similarity_score` methods. - **[`scrapling/core/storage.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/storage.py)** – Implements `SQLiteStorageSystem` and the `StorageSystemMixin` interface for persisting element fingerprints. - **[`scrapling/core/_utils.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/_utils.py)** – Provides `_StorageTools.element_to_dict` for serializing HTML elements into comparable dictionaries. - **[`scrapling/core/mixins.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/mixins.py)** – Defines `SelectorsGeneration` and the `Selectors` collection class used to wrap relocated elements. ## Summary - **Scrapling's adaptive storage system** combines the `Selector` class with an SQLite backend to persist element structural fingerprints across scraping sessions. - When a CSS selector fails due to DOM changes, the system automatically triggers **relocation mode**, retrieving stored fingerprints and scoring all nodes in the new document. - The **similarity algorithm** in `__calculate_similarity_score` weighs tag names, text content, attributes, and DOM position to identify the best match. - Developers enable this functionality by passing `adaptive=True` during `Selector` instantiation or individual queries, using `identifier` parameters to link queries with stored records. ## Frequently Asked Questions ### What storage backends does Scrapling's adaptive system support? By default, Scrapling uses `SQLiteStorageSystem` from [`scrapling/core/storage.py`](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/storage.py), which persists fingerprints to a local SQLite database file. The architecture uses a mixin pattern (`StorageSystemMixin`), allowing developers to implement custom backends (such as Redis or PostgreSQL) by subclassing the mixin and overriding the `save` and `retrieve` methods. ### How does the similarity scoring algorithm handle text content changes? The `__calculate_similarity_score` method in [`parser.py`](https://github.com/D4Vinci/Scrapling/blob/main/parser.py) uses `difflib.SequenceMatcher` to compute string similarity ratios for text content. This allows the algorithm to recognize elements even when text undergoes minor modifications (such as price updates or formatting changes), assigning partial credit based on character-level similarity rather than requiring exact matches. ### Can I use adaptive mode without explicitly saving elements first? No, adaptive relocation requires a stored fingerprint to compare against. The system triggers relocation only when a query includes an `identifier` parameter that corresponds to a previously saved record. If you attempt to use `adaptive=True` without a matching stored identifier, the query will simply return empty results when the selector fails, as there is no historical data to facilitate relocation. ### What performance impact does the relocation algorithm have? The relocation process in `Selector.relocate` iterates over every node in the DOM (`_find_all_elements(self._root)`) and computes similarity scores for each candidate. For large documents, this O(n) traversal with complex string comparisons can introduce measurable latency. However, the system allows tuning via the `percentage` parameter, which sets a minimum similarity threshold and can short-circuit processing once a sufficiently high-scoring match is found.