# How the `/scrape` Command Deduplicates Job Postings in ai-job-search: A Complete Guide

> Discover how the /scrape command in ai-job-search deduplicates job postings using seen_jobs.json and application tracker. Avoid redundant entries and streamline your job search.

- Repository: [Mads Lorentzen/ai-job-search](https://github.com/MadsLorentzen/ai-job-search)
- Tags: how-to-guide
- Published: 2026-07-14

---

**The `/scrape` command prevents duplicate job postings by cross-referencing new listings against a persistent [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json) registry and the `job_search_tracker.csv` application history, ensuring each job is evaluated only once across multiple search sessions.**

The `MadsLorentzen/ai-job-search` repository implements a robust deduplication system that eliminates redundant job postings from multiple sources. This article breaks down how the `/scrape` skill identifies and filters duplicates using persistent state files and status tracking.

## The Three-Stage Deduplication Pipeline

The deduplication logic in [`.claude/skills/job-scraper/SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/.claude/skills/job-scraper/SKILL.md) follows a strict workflow that guarantees every job posting is processed exactly once.

### Stage 1: Loading the Persistent State

Before executing any search queries, the skill initializes two critical data sources:

- **[`job_scraper/seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/job_scraper/seen_jobs.json)**: A JSON map that records every URL (or *company + title* composite key) the scraper has ever encountered. If the file is missing, the system creates it with an empty object: `{"seen": {}}`.

- **`job_search_tracker.csv`**: A CSV log containing jobs the user has already applied to. The skill extracts *company + role* pairs from this file to prevent resurfacing applications already in progress.

### Stage 2: Real-Time Duplicate Detection

After gathering candidates from portal-specific searches (CLI or WebSearch), the skill evaluates each posting against the loaded state. The filtering logic in **Step 2** of the workflow checks two conditions:

1. **URL or Composite Key Match**: If the job's URL or the generated `"{company}_{title}"` key exists in [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json), the entry is discarded.
2. **Application Tracker Match**: If the *company + role* pair appears in `job_search_tracker.csv`, the posting is skipped to avoid re-suggesting applied positions.

Only jobs passing both checks proceed to the candidate presentation stage.

### Stage 3: Persisting Records with Status Flags

Regardless of whether a job was presented or skipped, **Step 4** writes every entry back to [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json). The JSON structure uses the URL (or composite key) as the top-level key and stores:

```json
{
  "seen": {
    "https://example.com/job/123": {
      "title": "Senior Python Developer",
      "company": "TechCorp",
      "url": "https://example.com/job/123",
      "first_seen": "2024-01-15",
      "fit": 85,
      "status": "new"
    }
  }
}

```

The **status** field tracks the job lifecycle: `new`, `skipped`, `evaluated`, `ranked`, or `expired`. This additive approach ensures that later runs can reliably recognize duplicates even if the posting was previously ignored.

## Key Data Structures and File Paths

The deduplication system relies on specific files with distinct roles:

| File | Role in Deduplication |
|------|----------------------|
| [`.claude/skills/job-scraper/SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/.claude/skills/job-scraper/SKILL.md) | Defines the loading, skipping, and persisting logic in Steps 1, 2, and 4 |
| [`job_scraper/seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/job_scraper/seen_jobs.json) | Runtime JSON database storing canonical records of all examined postings |
| `job_search_tracker.csv` | Runtime CSV preventing resurfacing of jobs already applied to |
| [`tools/security_guards.py`](https://github.com/MadsLorentzen/ai-job-search/blob/main/tools/security_guards.py) | Ensures [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json) remains in `.gitignore` to protect privacy |

## Implementation Walkthrough

The following Python pseudocode illustrates the core deduplication logic implemented in the skill:

```python
import json
import csv
import datetime
from pathlib import Path

ROOT = Path(__file__).parents[1]
SEEN_PATH = ROOT / "job_scraper" / "seen_jobs.json"
TRACKER_PATH = ROOT / "job_search_tracker.csv"

def load_state():
    """Load seen_jobs.json and applied jobs from tracker."""
    if not SEEN_PATH.exists():
        SEEN_PATH.write_text(json.dumps({"seen": {}}))
    seen = json.loads(SEEN_PATH.read_text())["seen"]
    
    applied = set()
    if TRACKER_PATH.exists():
        with open(TRACKER_PATH, newline="") as f:
            for row in csv.DictReader(f):
                applied.add((row["company"], row["role"]))
    return seen, applied

def should_skip(job, seen, applied):
    """Check if job exists in seen registry or tracker."""
    key = job["url"] or f"{job['company']}_{job['title']}"
    
    if key in seen:
        return True
    if (job["company"], job["title"]) in applied:
        return True
    return False

def persist_job(job, status, seen):
    """Write job to seen_jobs.json with metadata."""
    key = job["url"] or f"{job['company']}_{job['title']}"
    seen[key] = {
        "title": job["title"],
        "company": job["company"],
        "url": job["url"],
        "first_seen": datetime.date.today().isoformat(),
        "fit": job.get("fit"),
        "status": status
    }
    SEEN_PATH.write_text(json.dumps({"seen": seen}, indent=2))

```

## Security and Data Integrity

The repository includes [`tools/security_guards.py`](https://github.com/MadsLorentzen/ai-job-search/blob/main/tools/security_guards.py) to protect the deduplication registry. This utility ensures [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json) is never accidentally committed to version control, preserving the privacy of your job search history and preventing repository bloat from large JSON files.

## Summary

- The `/scrape` command uses a **two-tier check** against [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json) and `job_search_tracker.csv` to eliminate duplicates.
- Jobs are keyed by **URL or company+title composite**, catching reposts even when links change.
- Every encountered job is **persisted with a status flag**, creating an additive history that prevents future duplicates.
- The system is **privacy-protected** by security guards that keep personal job data out of git.

## Frequently Asked Questions

### What happens if I delete seen_jobs.json?

If [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json) is missing, the skill automatically creates a new empty file with `{"seen": {}}`. However, all previous job history is lost, meaning the next scrape may re-present previously seen listings as "new" jobs.

### How does the scraper handle reposted jobs with different URLs?

The deduplication falls back to a **composite key** combining `company` and `title` when URLs differ. This catches reposted positions on different job boards or refreshed listings, preventing them from appearing as new opportunities.

### What's the difference between 'skipped' and 'expired' status?

A `skipped` status indicates the job was filtered during the deduplication phase (duplicate or already applied), while `expired` marks jobs that are no longer available on the source platform. Both statuses prevent future presentation, but `expired` specifically tracks jobs that were valid but are now closed.

### Can I manually add entries to seen_jobs.json?

Yes, you can manually append entries to [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json) using the same JSON structure. Ensure you use a unique key (URL or company_title format) and include the required fields: `title`, `company`, `first_seen`, and `status`. The skill will respect these entries on the next run.