How the Developer-Portfolios Script Removes Exact Duplicate Adjacent Links

The script removes exact duplicate adjacent links by scanning the README line-by-line in the remove_exact_duplicate_links function, comparing normalized URLs and trimmed link text to identify consecutive duplicates, keeping only the first occurrence.

The emmabostian/developer-portfolios repository maintains a curated list of developer portfolios in a single Markdown file. To ensure data quality and prevent redundancy, the Python-based automation pipeline specifically targets exact duplicate links that are adjacent using a dedicated two-stage detection and removal algorithm implemented in src/alphabetical.py.

The Core Detection Algorithm in src/alphabetical.py

The primary logic resides in remove_exact_duplicate_links, spanning lines 504–555 of src/alphabetical.py. This function implements a sliding-window approach that processes the README content after alphabetical sorting and global deduplication have occurred.

Line-by-Line Parsing and Pattern Extraction

The function iterates through the input list using a while loop (while i < len(lines)) to enable non-sequential index jumps when duplicates are found. For each line, it extracts the link components using two regular expressions:

• bracket_re – captures the link text inside […]
• paren_re – captures the URL inside (…)

As shown in lines 525–537, the algorithm performs specific preprocessing before comparison:

1. Text trimming: The link text has trailing spaces stripped (but case and internal spacing remain untouched)
2. URL normalization: The extracted URL is processed through the shared _normalize_url helper to standardize scheme, host, and path casing

Lines 525–537 handle this extraction and trimming logic, ensuring that only lines containing both a bracketed text segment and a parenthesized URL are considered for duplicate detection.

Normalization and Comparison Logic

The duplicate detection occurs in lines 540–549. After extracting the normalized URL and trimmed text from the current line (index i), the algorithm initializes j = i + 1 and enters an inner loop that peeks at subsequent lines:

• If line j matches the current line's trimmed text and normalized URL exactly, it is skipped and the removed counter increments
• The inner loop continues until a non-matching line or end-of-file is encountered
• The outer loop then resumes at i = j, effectively jumping past the entire run of duplicates

This approach preserves the first occurrence of any duplicate run while eliminating all adjacent copies. The function returns a tuple containing the cleaned list of lines and the total count of removed duplicates.

Integration into the Processing Pipeline

The duplicate removal occurs as a final cleanup step in the main workflow, specifically at lines 694–697 of src/alphabetical.py. After the README has been title-cased, cleaned, URL-deduplicated, globally deduplicated, and alphabetically re-ordered via sort_lists_alphabetically, the script executes:

sorted_lines, header_indices = sort_lists_alphabetically(reorganized_lines)
final_lines, post_removed = remove_exact_duplicate_links(sorted_lines)
if post_removed:
    print(f"Removed {post_removed} adjacent exact duplicate link(s) after sorting.")

This placement ensures that only exact duplicates that survived earlier normalization stages and ended up adjacent after alphabetical sorting are removed. The accompanying log message reports the number of eliminated links to provide transparency during execution.

Practical Example: Before and After

Consider the following Markdown input containing adjacent duplicates:

- [Awesome Portfolio](https://example.com)
- [Awesome Portfolio](https://example.com)   
- [Another Portfolio](https://example.org)

When processed through the pipeline:

python -m src.run_dup_report

# Output:

# Removed 1 adjacent exact duplicate link(s) after sorting.

The resulting output contains only unique adjacent entries:

- [Awesome Portfolio](https://example.com)
- [Another Portfolio](https://example.org)

The second line is eliminated because its link text (after trailing space removal) and normalized URL exactly match the previous line. Non-Markdown lines, lines missing either bracketed text or parenthesized URLs, or lines differing in either component remain untouched.

Summary

• remove_exact_duplicate_links in src/alphabetical.py (lines 504–555) implements the core logic for identifying and removing adjacent exact duplicate Markdown links
• The algorithm uses regex extraction (bracket_re, paren_re) combined with text trimming (trailing spaces only) and URL normalization via _normalize_url to establish equality
• Comparison is case-sensitive for link text but case-insensitive for URL components affected by normalization
• The function runs after alphabetical sorting in the main pipeline (lines 694–697), targeting only consecutive duplicates that survive earlier deduplication stages
• Non-adjacent duplicates or non-Markdown lines are not affected by this specific removal stage

Frequently Asked Questions

What defines an "exact duplicate" adjacent link?

An exact duplicate requires both the link text inside […] and the URL inside (…) to match exactly after preprocessing. The link text must be identical after stripping trailing whitespace (case-sensitive), while the URL must match after normalization through _normalize_url (case-insensitive for scheme and host). Both conditions must be satisfied for consecutive lines to trigger removal.

Where does the duplicate removal occur in the processing order?

According to the source code in src/alphabetical.py, the remove_exact_duplicate_links function executes after title-casing, cleaning, URL deduplication, global deduplication, and alphabetical sorting. Specifically, lines 694–697 show the function receiving output from sort_lists_alphabetically and returning final_lines used for subsequent feed generation.

How does URL normalization affect duplicate detection?

The script applies the _normalize_url helper to standardize URLs before comparison. This normalization typically converts scheme and host components to lowercase and may standardize path formatting, making the comparison case-insensitive for those URL segments. However, the link text comparison remains case-sensitive, preserving intentional capitalization differences in portfolio names.

Can this script remove duplicate links that are not adjacent?

No. The remove_exact_duplicate_links function specifically targets adjacent duplicates using its sliding-window peek logic (j = i + 1). Non-adjacent duplicates that appear elsewhere in the document are handled by earlier pipeline stages such as the global deduplication step that runs before alphabetical sorting. The adjacent duplicate removal serves as a final cleanup for duplicates that converge next to each other during the sorting process.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}