How the `/scrape` Command Deduplicates Job Postings in ai-job-search: A Complete Guide

The /scrape command prevents duplicate job postings by cross-referencing new listings against a persistent seen_jobs.json registry and the job_search_tracker.csv application history, ensuring each job is evaluated only once across multiple search sessions.

The MadsLorentzen/ai-job-search repository implements a robust deduplication system that eliminates redundant job postings from multiple sources. This article breaks down how the /scrape skill identifies and filters duplicates using persistent state files and status tracking.

The Three-Stage Deduplication Pipeline

The deduplication logic in .claude/skills/job-scraper/SKILL.md follows a strict workflow that guarantees every job posting is processed exactly once.

Stage 1: Loading the Persistent State

Before executing any search queries, the skill initializes two critical data sources:

• job_scraper/seen_jobs.json: A JSON map that records every URL (or company + title composite key) the scraper has ever encountered. If the file is missing, the system creates it with an empty object: {"seen": {}}.

• job_search_tracker.csv: A CSV log containing jobs the user has already applied to. The skill extracts company + role pairs from this file to prevent resurfacing applications already in progress.

Stage 2: Real-Time Duplicate Detection

After gathering candidates from portal-specific searches (CLI or WebSearch), the skill evaluates each posting against the loaded state. The filtering logic in Step 2 of the workflow checks two conditions:

1. URL or Composite Key Match: If the job's URL or the generated "{company}_{title}" key exists in seen_jobs.json, the entry is discarded.
2. Application Tracker Match: If the company + role pair appears in job_search_tracker.csv, the posting is skipped to avoid re-suggesting applied positions.

Only jobs passing both checks proceed to the candidate presentation stage.

Stage 3: Persisting Records with Status Flags

Regardless of whether a job was presented or skipped, Step 4 writes every entry back to seen_jobs.json. The JSON structure uses the URL (or composite key) as the top-level key and stores:

{
  "seen": {
    "https://example.com/job/123": {
      "title": "Senior Python Developer",
      "company": "TechCorp",
      "url": "https://example.com/job/123",
      "first_seen": "2024-01-15",
      "fit": 85,
      "status": "new"
    }
  }
}

The status field tracks the job lifecycle: new, skipped, evaluated, ranked, or expired. This additive approach ensures that later runs can reliably recognize duplicates even if the posting was previously ignored.

Key Data Structures and File Paths

The deduplication system relies on specific files with distinct roles:

File	Role in Deduplication
.claude/skills/job-scraper/SKILL.md	Defines the loading, skipping, and persisting logic in Steps 1, 2, and 4
job_scraper/seen_jobs.json	Runtime JSON database storing canonical records of all examined postings
job_search_tracker.csv	Runtime CSV preventing resurfacing of jobs already applied to
tools/security_guards.py	Ensures seen_jobs.json remains in .gitignore to protect privacy

Implementation Walkthrough

The following Python pseudocode illustrates the core deduplication logic implemented in the skill:

import json
import csv
import datetime
from pathlib import Path

ROOT = Path(__file__).parents[1]
SEEN_PATH = ROOT / "job_scraper" / "seen_jobs.json"
TRACKER_PATH = ROOT / "job_search_tracker.csv"

def load_state():
    """Load seen_jobs.json and applied jobs from tracker."""
    if not SEEN_PATH.exists():
        SEEN_PATH.write_text(json.dumps({"seen": {}}))
    seen = json.loads(SEEN_PATH.read_text())["seen"]
    
    applied = set()
    if TRACKER_PATH.exists():
        with open(TRACKER_PATH, newline="") as f:
            for row in csv.DictReader(f):
                applied.add((row["company"], row["role"]))
    return seen, applied

def should_skip(job, seen, applied):
    """Check if job exists in seen registry or tracker."""
    key = job["url"] or f"{job['company']}_{job['title']}"
    
    if key in seen:
        return True
    if (job["company"], job["title"]) in applied:
        return True
    return False

def persist_job(job, status, seen):
    """Write job to seen_jobs.json with metadata."""
    key = job["url"] or f"{job['company']}_{job['title']}"
    seen[key] = {
        "title": job["title"],
        "company": job["company"],
        "url": job["url"],
        "first_seen": datetime.date.today().isoformat(),
        "fit": job.get("fit"),
        "status": status
    }
    SEEN_PATH.write_text(json.dumps({"seen": seen}, indent=2))

Security and Data Integrity

The repository includes tools/security_guards.py to protect the deduplication registry. This utility ensures seen_jobs.json is never accidentally committed to version control, preserving the privacy of your job search history and preventing repository bloat from large JSON files.

Summary

• The /scrape command uses a two-tier check against seen_jobs.json and job_search_tracker.csv to eliminate duplicates.
• Jobs are keyed by URL or company+title composite, catching reposts even when links change.
• Every encountered job is persisted with a status flag, creating an additive history that prevents future duplicates.
• The system is privacy-protected by security guards that keep personal job data out of git.

Frequently Asked Questions

What happens if I delete seen_jobs.json?

If seen_jobs.json is missing, the skill automatically creates a new empty file with {"seen": {}}. However, all previous job history is lost, meaning the next scrape may re-present previously seen listings as "new" jobs.

How does the scraper handle reposted jobs with different URLs?

The deduplication falls back to a composite key combining company and title when URLs differ. This catches reposted positions on different job boards or refreshed listings, preventing them from appearing as new opportunities.

What's the difference between 'skipped' and 'expired' status?

A skipped status indicates the job was filtered during the deduplication phase (duplicate or already applied), while expired marks jobs that are no longer available on the source platform. Both statuses prevent future presentation, but expired specifically tracks jobs that were valid but are now closed.

Can I manually add entries to seen_jobs.json?

Yes, you can manually append entries to seen_jobs.json using the same JSON structure. Ensure you use a unique key (URL or company_title format) and include the required fields: title, company, first_seen, and status. The skill will respect these entries on the next run.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}