# How the Developer-Portfolios Script Removes Exact Duplicate Adjacent Links

> Learn how the developer-portfolios script removes exact duplicate adjacent links by scanning line by line and comparing normalized URLs and link text. Discover efficient link management.

- Repository: [Emma Bostian/developer-portfolios](https://github.com/emmabostian/developer-portfolios)
- Tags: internals
- Published: 2026-06-01

---

**The script removes exact duplicate adjacent links by scanning the README line-by-line in the `remove_exact_duplicate_links` function, comparing normalized URLs and trimmed link text to identify consecutive duplicates, keeping only the first occurrence.**

The `emmabostian/developer-portfolios` repository maintains a curated list of developer portfolios in a single Markdown file. To ensure data quality and prevent redundancy, the Python-based automation pipeline specifically targets **exact duplicate links that are adjacent** using a dedicated two-stage detection and removal algorithm implemented in [`src/alphabetical.py`](https://github.com/emmabostian/developer-portfolios/blob/main/src/alphabetical.py).

## The Core Detection Algorithm in [`src/alphabetical.py`](https://github.com/emmabostian/developer-portfolios/blob/main/src/alphabetical.py)

The primary logic resides in `remove_exact_duplicate_links`, spanning lines 504–555 of [`src/alphabetical.py`](https://github.com/emmabostian/developer-portfolios/blob/main/src/alphabetical.py). This function implements a sliding-window approach that processes the README content after alphabetical sorting and global deduplication have occurred.

### Line-by-Line Parsing and Pattern Extraction

The function iterates through the input list using a `while` loop (`while i < len(lines)`) to enable non-sequential index jumps when duplicates are found. For each line, it extracts the link components using two regular expressions:

- `bracket_re` – captures the link text inside `[…]`
- `paren_re` – captures the URL inside `(…)`

As shown in lines 525–537, the algorithm performs specific preprocessing before comparison:

1. **Text trimming**: The link text has trailing spaces stripped (but case and internal spacing remain untouched)
2. **URL normalization**: The extracted URL is processed through the shared `_normalize_url` helper to standardize scheme, host, and path casing

Lines 525–537 handle this extraction and trimming logic, ensuring that only lines containing both a bracketed text segment and a parenthesized URL are considered for duplicate detection.

### Normalization and Comparison Logic

The duplicate detection occurs in lines 540–549. After extracting the normalized URL and trimmed text from the current line (index `i`), the algorithm initializes `j = i + 1` and enters an inner loop that peeks at subsequent lines:

- If line `j` matches the current line's **trimmed text** and **normalized URL** exactly, it is skipped and the `removed` counter increments
- The inner loop continues until a non-matching line or end-of-file is encountered
- The outer loop then resumes at `i = j`, effectively jumping past the entire run of duplicates

This approach preserves the first occurrence of any duplicate run while eliminating all adjacent copies. The function returns a tuple containing the cleaned list of lines and the total count of removed duplicates.

## Integration into the Processing Pipeline

The duplicate removal occurs as a final cleanup step in the main workflow, specifically at lines 694–697 of [`src/alphabetical.py`](https://github.com/emmabostian/developer-portfolios/blob/main/src/alphabetical.py). After the README has been title-cased, cleaned, URL-deduplicated, globally deduplicated, and alphabetically re-ordered via `sort_lists_alphabetically`, the script executes:

```python
sorted_lines, header_indices = sort_lists_alphabetically(reorganized_lines)
final_lines, post_removed = remove_exact_duplicate_links(sorted_lines)
if post_removed:
    print(f"Removed {post_removed} adjacent exact duplicate link(s) after sorting.")

```

This placement ensures that only exact duplicates that survived earlier normalization stages and ended up adjacent after alphabetical sorting are removed. The accompanying log message reports the number of eliminated links to provide transparency during execution.

## Practical Example: Before and After

Consider the following Markdown input containing adjacent duplicates:

```markdown
- [Awesome Portfolio](https://example.com)
- [Awesome Portfolio](https://example.com)   
- [Another Portfolio](https://example.org)

```

When processed through the pipeline:

```bash
python -m src.run_dup_report

# Output:

# Removed 1 adjacent exact duplicate link(s) after sorting.

```

The resulting output contains only unique adjacent entries:

```markdown
- [Awesome Portfolio](https://example.com)
- [Another Portfolio](https://example.org)

```

The second line is eliminated because its link text (after trailing space removal) and normalized URL exactly match the previous line. Non-Markdown lines, lines missing either bracketed text or parenthesized URLs, or lines differing in either component remain untouched.

## Summary

- **`remove_exact_duplicate_links`** in [`src/alphabetical.py`](https://github.com/emmabostian/developer-portfolios/blob/main/src/alphabetical.py) (lines 504–555) implements the core logic for identifying and removing adjacent exact duplicate Markdown links
- The algorithm uses regex extraction (`bracket_re`, `paren_re`) combined with **text trimming** (trailing spaces only) and **URL normalization** via `_normalize_url` to establish equality
- Comparison is **case-sensitive** for link text but **case-insensitive** for URL components affected by normalization
- The function runs after alphabetical sorting in the main pipeline (lines 694–697), targeting only consecutive duplicates that survive earlier deduplication stages
- Non-adjacent duplicates or non-Markdown lines are not affected by this specific removal stage

## Frequently Asked Questions

### What defines an "exact duplicate" adjacent link?

An exact duplicate requires both the link text inside `[…]` and the URL inside `(…)` to match exactly after preprocessing. The link text must be identical after stripping trailing whitespace (case-sensitive), while the URL must match after normalization through `_normalize_url` (case-insensitive for scheme and host). Both conditions must be satisfied for consecutive lines to trigger removal.

### Where does the duplicate removal occur in the processing order?

According to the source code in [`src/alphabetical.py`](https://github.com/emmabostian/developer-portfolios/blob/main/src/alphabetical.py), the `remove_exact_duplicate_links` function executes after title-casing, cleaning, URL deduplication, global deduplication, and alphabetical sorting. Specifically, lines 694–697 show the function receiving output from `sort_lists_alphabetically` and returning `final_lines` used for subsequent feed generation.

### How does URL normalization affect duplicate detection?

The script applies the `_normalize_url` helper to standardize URLs before comparison. This normalization typically converts scheme and host components to lowercase and may standardize path formatting, making the comparison **case-insensitive** for those URL segments. However, the link text comparison remains **case-sensitive**, preserving intentional capitalization differences in portfolio names.

### Can this script remove duplicate links that are not adjacent?

No. The `remove_exact_duplicate_links` function specifically targets adjacent duplicates using its sliding-window peek logic (`j = i + 1`). Non-adjacent duplicates that appear elsewhere in the document are handled by earlier pipeline stages such as the global deduplication step that runs before alphabetical sorting. The adjacent duplicate removal serves as a final cleanup for duplicates that converge next to each other during the sorting process.