How Job Posting Deduplication Works Across Multiple Portals

Job posting deduplication in the AI-Job-Search repository works by generating a deterministic canonical key for each scraped listing—using either the portal’s native job ID or a truncated SHA-256 hash of the title, company, and URL—and checking that key against a persistent SQLite database before allowing the posting into the ranking pipeline.

The MadsLorentzen/ai-job-search system aggregates listings from disparate sources like LinkedIn, Indeed, and GitHub Jobs. To ensure the same position advertised on multiple portals appears only once, the repository implements a deduplication layer defined in the job-scraper skill documentation and invoked via the top-level rank command.

Canonical Key Generation Strategy

The deduplication engine creates a stable identifier for every posting by prioritizing native portal identifiers and falling back to content hashing when those are unavailable.

Native Portal IDs

When a scraper extracts a listing that includes a platform-specific job identifier, the system constructs the canonical key by concatenating the portal name and the native ID:

"{portal}:{native_id}"

This approach captures exact duplicates where the same employer has cross-posted a job using the platform’s internal tracking system.

Derived Hash Fallback

If the portal does not expose a native ID, the system generates a 12-character hexadecimal hash from the most stable textual fields. As implemented in the conceptual utils/dedup.py module, the fallback key uses a SHA-256 digest of the pipe-delimited title, company name, and URL:

def _hash_fields(title: str, company: str, url: str) -> str:
    """Fallback hash when a portal-native id is missing."""
    raw = f"{title}|{company}|{url}".encode("utf-8")
    return hashlib.sha256(raw).hexdigest()[:12]

The final key prefixes this hash with the portal name to prevent collisions across different sources:

"{portal}:{hash}"

The Deduplication Pipeline

According to .claude/skills/job-scraper/SKILL.md, the scraper executes a four-step validation sequence for every retrieved listing.

Step 1: Raw Field Extraction

The scraper first normalizes the raw HTML or API response into a structured dictionary containing:

• portal — the source platform (e.g., linkedin, indeed)
• id — the native posting ID (may be None)
• title — the job title
• company — the employer name
• url — the canonical application URL

Step 2: Key Composition

The make_key function in utils/dedup.py (illustrated in the repository documentation) branches based on the presence of a native ID:

def make_key(portal: str, native_id: str | None,
             title: str, company: str, url: str) -> str:
    """Return the deterministic deduplication key."""
    if native_id:
        return f"{portal}:{native_id}"
    return f"{portal}:{_hash_fields(title, company, url)}"

Step 3: SQLite Store Lookup

The system queries a local SQLite database (dedup.db) to check for existing entries. The is_duplicate function executes a parameterized lookup against the postings table:

def is_duplicate(key: str) -> bool:
    """True if the posting has already been seen."""
    cur = conn.execute("SELECT 1 FROM postings WHERE key = ?", (key,))
    return cur.fetchone() is not None

If the key exists, the posting is discarded immediately and never reaches the ranking or notification stages.

Step 4: Duplicate Handling and Registration

New keys are persisted via register_posting, which inserts a row containing the key, current timestamp, and portal source:

def register_posting(key: str, portal: str) -> None:
    """Persist a newly-seen posting."""
    conn.execute(
        "INSERT INTO postings (key, first_seen, last_seen, portal) "
        "VALUES (?, datetime('now'), datetime('now'), ?)",
        (key, portal)
    )
    conn.commit()

Source Code Implementation

The deduplication logic is orchestrated through Claude-specific command files rather than a standalone executable script. The workflow definition resides in .claude/skills/job-scraper/SKILL.md, which specifies the canonical-ID generation rules and the SQLite-based storage requirements. The .claude/commands/rank.md file invokes this skill, triggering the scraper, running the deduplication step, and forwarding unique postings to the ranking engine.

A typical scraper integration looks like this:

# ----------------------------------------------------------------------

# job_scraper/main.py – Example usage during a scrape run

# ----------------------------------------------------------------------

from utils.dedup import make_key, is_duplicate, register_posting

def process_listing(raw):
    portal = raw["portal"]
    native_id = raw.get("id")               # May be None

    title = raw["title"]
    company = raw["company"]
    url = raw["url"]

    key = make_key(portal, native_id, title, company, url)

    if is_duplicate(key):
        # Skip – already in the feed

        return None

    register_posting(key, portal)
    # Continue with ranking / storage …

    return raw

Database Schema and Maintenance

The SQLite schema defined in the initialization block of utils/dedup.py creates a single table optimized for fast key lookups:

CREATE TABLE IF NOT EXISTS postings (
    key TEXT PRIMARY KEY,
    first_seen TEXT,
    last_seen TEXT,
    portal TEXT
)

The key column serves as the primary index, ensuring O(1) average-time complexity for duplicate checks. A background cleanup routine (referenced in the skill documentation) prunes entries older than a configurable TTL—defaulting to 30 days—preventing stale duplicates from blocking legitimate reposts of previously expired positions.

Summary

• Canonical keys are generated using either "{portal}:{native_id}" or a truncated SHA-256 hash of stable fields.
• Duplicate detection relies on a persistent SQLite database (dedup.db) that stores every unique key with metadata.
• Integration points are defined in .claude/skills/job-scraper/SKILL.md and executed via .claude/commands/rank.md.
• Cleanup policies automatically remove entries after 30 days to balance storage growth with the ability to detect fresh reposts.

Frequently Asked Questions

How does the system handle the same job posted on different portals?

The deduplication key always prefixes the portal name, meaning identical jobs on LinkedIn and Indeed receive distinct keys (e.g., linkedin:abc123 vs. indeed:abc123). If you wish to collapse cross-portal duplicates, you would modify _hash_fields to exclude the portal prefix and hash only the content fields, though this risks false positives for genuinely distinct positions at the same company.

What happens if two different jobs generate the same SHA-256 hash?

The probability of a 12-character hexadecimal collision (48 bits) is approximately 1 in 280 trillion. In the unlikely event of a collision, the second job would be incorrectly flagged as a duplicate and skipped. Increasing the truncation length to 16 or 32 characters in _hash_fields reduces this risk further with minimal storage overhead.

Can the deduplication database be inspected manually?

Yes. Since dedup.db is a standard SQLite file, you can query it directly using the sqlite3 command-line tool or any SQLite browser. For example: SELECT * FROM postings WHERE portal = 'linkedin' ORDER BY first_seen DESC LIMIT 10; returns the most recent unique LinkedIn postings.

How often does the system clean up old deduplication entries?

The default TTL is 30 days, after which the cleanup routine removes stale keys. This interval is configurable in the job-scraper skill settings, allowing you to extend retention for long-running searches or shorten it for high-volume, rapidly cycling job markets.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}