How to Detect and Remove Duplicate Portfolio URLs in emmabostian/developer-portfolios

The emmabostian/developer-portfolios repository detects and removes duplicate portfolio URLs by first normalizing URLs to a canonical form, then scanning the entire README.md to eliminate later occurrences of the same URL, and finally collapsing adjacent lines that contain identical link text and normalized URLs.

The emmabostian/developer-portfolios repository maintains a curated list of developer portfolio links in README.md. To ensure the list remains clean and free of redundant entries, the codebase implements automated duplicate detection that processes URLs before sorting and validation. The cleaning logic resides primarily in src/alphabetical.py and is exposed via command-line utilities for both automated CI pipelines and manual reporting.

URL Normalization

All duplicate detection relies on a shared _normalize_url function defined in src/alphabetical.py (lines 64-89). This function converts raw URLs into a canonical form by lowercasing the scheme and netloc, removing default ports (e.g., :80 for HTTP), and stripping a single trailing slash while preserving query parameters and fragments.

This normalization ensures that variations such as https://example.com/, https://EXAMPLE.COM, and http://example.com:80 are treated as identical during deduplication.

Document-Wide Duplicate URL Removal

The remove_duplicate_urls function (lines 481-505 in src/alphabetical.py) implements the primary deduplication strategy. It scans each line of the README to detect and remove duplicate portfolio URLs across the entire document, preserving only the first occurrence of each unique URL.

The function uses a regular expression pattern—paren_re = re.compile(r"\(([^)]+)\)")—to extract the first parenthesized URL on each line. After normalizing the extracted URL via _normalize_url, it checks against a seen_urls set. If the normalized URL already exists in the set, the entire line is omitted from the output; otherwise, the URL is added to the set and the line is retained.

This process returns a filtered list of lines along with a count of removed duplicates. The function is invoked in the main processing pipeline at lines 555-560.

Adjacent Exact-Duplicate Link Removal

After sorting the list alphabetically, the script performs a secondary cleaning pass using remove_exact_duplicate_links (lines 504-556 in src/alphabetical.py). This function targets adjacent lines that represent exact duplicates of the same markdown link.

The logic walks through the line list and compares consecutive entries. For each line, it trims trailing whitespace from the bracket text and normalizes the URL. If both the link text and the normalized URL match the next line exactly, the subsequent line is dropped. A while loop handles cases where multiple identical lines appear in sequence.

This step runs immediately before writing the final README, called at lines 994-998 in the main execution flow.

Reporting Duplicates Without Modifying Files

For CI verification or manual inspection, the repository provides src/run_dup_report.py, a read-only utility that mirrors the detection logic without mutating files. Running the module executes two checks: it identifies document-wide duplicate URLs (lines 22-33) and flags adjacent exact-duplicate links (lines 35-60).

python -m src.run_dup_report

The script outputs a concise summary such as:

URL-based duplicates found … : 3
Adjacent exact‑duplicate links : 2

This allows maintainers to audit the list before triggering the automated cleaning workflow.

Automated Execution Pipeline

When the repository's main() function executes—typically invoked via python src/alphabetical.py or through the top-level run_tests.py in CI—the duplicate removal steps run automatically as part of the standard cleaning pipeline. The orchestration follows this sequence:

1. Basic cleaning prepares the raw README content.
2. remove_duplicate_urls() eliminates document-wide URL duplicates.
3. Alphabetical sorting organizes the remaining entries.
4. remove_exact_duplicate_links() collapses any accidental adjacent duplicates created during sorting.

Developers can also run the cleaning steps manually:

import src.alphabetical as a

with open('README.md') as f:
    lines = f.readlines()

lines, _ = a.remove_duplicate_urls(lines)
lines, _ = a.remove_exact_duplicate_links(lines)

with open('README.md', 'w') as f:
    f.writelines(lines)

Summary

• URL normalization in _normalize_url (lines 64-89) canonicalizes URLs by lowercasing components, removing default ports, and stripping trailing slashes.
• Document-wide detection via remove_duplicate_urls (lines 481-505) uses a regex parser and set tracking to eliminate later occurrences of identical URLs.
• Adjacent duplicate removal through remove_exact_duplicate_links (lines 504-556) removes consecutive lines with matching bracket text and normalized URLs.
• Non-destructive reporting is available via src/run_dup_report.py for CI checks and manual audits.
• The cleaning pipeline automatically executes these steps during the main processing flow in src/alphabetical.py.

Frequently Asked Questions

How does the repository normalize URLs before checking for duplicates?

The _normalize_url function in src/alphabetical.py (lines 64-89) converts URLs to a canonical form by lowercasing the scheme and netloc, stripping default ports like :80 or :443, and removing a single trailing slash. This ensures superficial variations in URL formatting do not hide actual duplicates.

What is the difference between document-wide duplicate removal and adjacent duplicate removal?

Document-wide removal, handled by remove_duplicate_urls, scans the entire README and removes any line containing a URL that appeared earlier in the file. Adjacent duplicate removal, performed by remove_exact_duplicate_links, targets only consecutive lines where both the link text and normalized URL match exactly, typically catching duplicates introduced during the sorting phase.

How can I check for duplicates without modifying the README.md file?

Run python -m src.run_dup_report to execute a read-only analysis. This script uses the same detection logic as the main cleaner but outputs a summary report showing how many URL-based duplicates and adjacent exact-duplicate links exist without writing changes to the file.

Where is the duplicate detection logic implemented in the codebase?

The core logic resides in src/alphabetical.py, specifically the _normalize_url helper (lines 64-89), the remove_duplicate_urls function (lines 481-505), and the remove_exact_duplicate_links function (lines 504-556). The reporting utility is implemented separately in src/run_dup_report.py (lines 22-60).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}