Using find_similar() for Element Similarity Matching and Relocation in Scrapling

find_similar() is a high-level Scrapling selector method that locates elements sharing the same structural depth and similar attributes to a reference element, enabling robust pattern matching without brittle CSS or XPath selectors.

The find_similar() method in the D4Vinci/Scrapling repository provides a resilient alternative to static selectors by combining structural context (ancestor hierarchy and depth) with attribute-based fuzzy matching. This approach is ideal for scraping repeated items like product cards or review blocks that share layout patterns but may have varying identifiers.

How find_similar() Works Under the Hood

Selector Core Architecture

In Scrapling, every parsed element is wrapped by a Selector object defined in scrapling/parser.py. This abstraction encapsulates an lxml HtmlElement (self._root) and provides navigation methods including css(), xpath(), and find_similar(). When you call find_similar(), the method analyzes the reference element's position in the DOM tree to generate an intelligent search scope.

Method Signature and Parameters

The find_similar() implementation in scrapling/parser.py (lines 995–1005) exposes three tunable parameters:

def find_similar(
    self,
    similarity_threshold: float = 0.2,
    ignore_attributes: List | Tuple = ("href", "src"),
    match_text: bool = False,
) -> "Selectors":

• similarity_threshold – Minimum similarity score (0.0 to 1.0) required for inclusion in results. The default 0.2 is permissive enough to catch variants while filtering noise.
• ignore_attributes – Tuple of attribute names excluded from comparison. Defaults to ("href", "src") because URLs typically vary across similar elements.
• match_text – When True, includes normalized text content in the similarity calculation, useful for distinguishing items with different labels but identical markup.

The Search Algorithm: Depth Filtering and Attribute Scoring

The method executes a two-phase search strategy defined in scrapling/parser.py (lines 31–93):

1. Structural Pre-filtering: The method calculates the reference element's depth (current_depth = len(list(root.iterancestors()))) and constructs an XPath matching elements with the same tag name and same ancestor chain length. This limits the search space to structurally comparable candidates rather than scanning the entire document.

path_parts = [self.tag]
if (parent := root.getparent()) is not None:
    path_parts.insert(0, parent.tag)
    if (grandparent := parent.getparent()) is not None:
        path_parts.insert(0, grandparent.tag)

xpath_path = "//{}".format("/".join(path_parts))
potential_matches = root.xpath(f"{xpath_path}[count(ancestor::*) = {current_depth}]")

2. Attribute Similarity Scoring: For each candidate, the private __are_alike() method computes a match score using difflib.SequenceMatcher on attribute values (excluding those in ignore_attributes). If match_text is enabled, text content also contributes to the score. Elements with no attributes receive a perfect match assumption (score += 1). Candidates meeting the similarity_threshold are converted back into Selector objects and returned as a Selectors collection.

Practical Examples: Scrapling find_similar() in Action

Extracting Repeated Product Cards

The most common use case involves locating all instances of a repeated component after identifying the first one. In tests/parser/test_general.py (lines 46–48), the pattern is demonstrated for product listings:

from scrapling import ScraplingSelector

page = ScraplingSelector(html_content)
first_product = page.css(".product").first
similar_products = first_product.find_similar()

for prod in similar_products:
    print(prod.attrib["data-id"], prod.text.strip())

This returns every product card sharing the first card's DOM depth and attribute structure, regardless of varying data-id values.

Filtering Reviews by Rating

You can chain find_similar() with attribute filtering to extract specific subsets. The test suite shows how to isolate high-rated reviews:

first_review = page.find("div", class_="review")
similar_reviews = first_review.find_similar(match_text=False)

high_rated = [
    rev for rev in similar_reviews
    if int(rev.attrib.get("data-rating", 0)) >= 4
]
print(f"Found {len(high_rated)} high-rated reviews")

Tuning Similarity Thresholds and Attributes

For stricter matching or custom attribute exclusion, adjust the parameters:

# Ignore tracking attributes and require 50% similarity

strict_matches = first_product.find_similar(
    similarity_threshold=0.5,
    ignore_attributes=("href", "data-tracking"),
    match_text=True
)

Chaining with Other Selectors

As shown in benchmarks.py (lines 10–13), find_similar() integrates into complex selection pipelines:

from scrapling import ScraplingSelector

def benchmark_similar(html):
    return (
        ScraplingSelector(html, adaptive=False)
        .find_by_text("Tipping the Velvet", first_match=True, clean_match=False)
        .find_similar(ignore_attributes=["title"])
    )

Performance and Implementation Details

The depth-based XPath pre-filtering in scrapling/parser.py (lines 31–41) dramatically reduces the candidate pool before similarity calculations begin, avoiding expensive full-tree scans. The attribute comparison relies on pure-Python difflib.SequenceMatcher, which provides sufficient speed for typical scraping workloads while handling fuzzy matches on class names or IDs that may contain hash suffixes.

The default similarity_threshold of 0.2 strikes a balance between precision and recall, accepting elements with minor attribute variations (such as incremental ID numbers) while rejecting structurally different components.

Summary

• find_similar() combines DOM depth filtering with attribute fuzzy matching to locate structurally similar elements without brittle selectors.
• The method returns a Selectors collection (defined in scrapling/core/mixins.py) containing all matches above the configurable similarity_threshold.
• Default behavior ignores href and src attributes to avoid URL variations breaking pattern matches.
• Performance optimization uses XPath ancestor counting to limit search scope before applying difflib.SequenceMatcher comparisons.
• Test coverage in tests/parser/test_general.py validates correctness for product grids and review lists.

Frequently Asked Questions

What is the default similarity threshold in Scrapling's find_similar()?

The default similarity_threshold is 0.2 (20% similarity), as defined in the method signature in scrapling/parser.py. This permissive default ensures the method captures variants of repeated components while filtering out unrelated elements with completely different attribute sets.

How does find_similar() handle elements without attributes?

According to the implementation in scrapling/parser.py (lines 54–93), when a candidate element has no attributes, the similarity scorer assumes a perfect match for that component (score += 1). This prevents attribute-less structural elements from being penalized during the comparison.

Can I use find_similar() to relocate elements after page layout changes?

Yes. Because find_similar() relies on structural depth and attribute patterns rather than absolute CSS paths or IDs, it remains effective when page layouts shift. By matching on tag hierarchy and similar class structures, the method can relocate the "spiritual successor" of an element even when specific coordinates or parent containers change.

What attributes are ignored by default in the similarity calculation?

By default, the ignore_attributes parameter is set to ("href", "src") according to the source code in scrapling/parser.py (lines 995–1005). These URL-based attributes are excluded because they typically vary across similar elements (e.g., different product links or image sources), while structural and styling attributes remain comparable.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}