# How the ai-job-search Job Scraper Avoids Duplicate Job Listings

> Learn how the ai-job-search job scraper effectively avoids duplicate listings using persistent data stores and efficient filtering. Get cleaner job search results.

- Repository: [Mads Lorentzen/ai-job-search](https://github.com/MadsLorentzen/ai-job-search)
- Tags: how-to-guide
- Published: 2026-07-12

---

**The ai-job-search scraper prevents duplicate listings by maintaining two persistent data stores—[`job_scraper/seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/job_scraper/seen_jobs.json) and `job_search_tracker.csv`—and filtering every fetched job against both sources before presenting results to the user.**

The MadsLorentzen/ai-job-search repository implements a robust deduplication system that ensures job seekers never process the same posting twice. By combining a JSON-based seen-jobs registry with a CSV application tracker, the scraper maintains state across sessions and job portals. This dual-layer approach eliminates duplicates based on both unique URLs and deterministic company-title combinations.

## The Dual-Storage Deduplication Architecture

The scraper relies on two complementary data sources to guarantee uniqueness. Both files persist between runs, creating a cumulative memory of every job encountered.

### Persistent JSON Registry ([`job_scraper/seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/job_scraper/seen_jobs.json))

The [`job_scraper/seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/job_scraper/seen_jobs.json) file serves as the primary deduplication database. When the scraper initiates, it loads this file—initializing it with `{ "seen": {} }` if the file does not exist—at lines **L40-L41** of the skill definition. This JSON structure maps unique identifiers (URLs or composite keys) to metadata objects containing the job title, company, first-seen date, and status.

After each search round, the scraper writes **every fetched job** back to this file, regardless of whether it was ultimately shown to the user. Entries receive statuses such as `new`, `skipped`, or `ranked`, ensuring complete historical coverage at lines **L109-L122**.

### Application Tracker CSV (`job_search_tracker.csv`)

The second layer, `job_search_tracker.csv`, tracks jobs the user has already applied to. At the beginning of every run (lines **L41-L42**), the scraper reads this CSV to extract a set of `company|role` pairs. This prevents the system from surfacing jobs the user has already acted upon, even if those jobs were not originally discovered through the scraper itself.

## The Deduplication Check Logic

During **Step 2 – Fetch & Parse**, the scraper evaluates each candidate job against both data sources simultaneously. According to lines **L95-L98**, a job is rejected if it meets any of the following conditions:

- **URL exists** in [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json)
- **Company + title combination** exists in [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json)
- **Company + title combination** exists in `job_search_tracker.csv`

Only jobs passing all three checks are added to the new results list and presented to the user. Because the scraper updates [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json) immediately after processing, subsequent runs automatically treat those entries as "already seen," eliminating duplicates across sessions and even across different portal-search CLIs.

## Implementation Example

The deduplication workflow follows this exact pattern, as implemented in the [`.claude/skills/job-scraper/SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/.claude/skills/job-scraper/SKILL.md) file:

```typescript
import { readFileSync, writeFileSync } from "fs";
import { parse } from "csv-parse/sync";

const SEEN_PATH = "job_scraper/seen_jobs.json";
const TRACKER_PATH = "job_search_tracker.csv";

// Load or initialise seen-jobs map
let seen = { seen: {} };
try {
  seen = JSON.parse(readFileSync(SEEN_PATH, "utf-8"));
} catch {}

const applied = new Set<string>();
// Load applied jobs from tracker (company+title key)
const trackerCsv = readFileSync(TRACKER_PATH, "utf-8");
for (const rec of parse(trackerCsv, { columns: true })) {
  applied.add(`${rec.company}|${rec.title}`);
}

// Helper to decide if a job is duplicate
function isDuplicate(job: { url: string; company: string; title: string }) {
  const key = `${job.url}`;                     // URL-based key
  const ctKey = `${job.company}|${job.title}`; // Company-title key
  return seen.seen[key] || seen.seen[ctKey] || applied.has(ctKey);
}

// Process a batch of scraped jobs
function ingest(jobs: Array<{ url: string; company: string; title: string }>) {
  for (const job of jobs) {
    const dup = isDuplicate(job);
    const entryKey = job.url || `${job.company}|${job.title}`;
    seen.seen[entryKey] = {
      title: job.title,
      company: job.company,
      url: job.url,
      first_seen: new Date().toISOString().split("T")[0],
      fit: "high",
      status: dup ? "skipped" : "new",
    };
  }
  writeFileSync(SEEN_PATH, JSON.stringify(seen, null, 2));
}

```

This implementation mirrors the production logic: loading state from [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json), loading the tracker CSV, skipping duplicates based on URL or company-title matches, and persisting every processed job back to the JSON store.

## Cross-Portal Compatibility

The deduplication logic is baked into the skill definition itself. Any new portal added via the `/add-portal` command automatically inherits the same duplicate-avoidance behavior without requiring code changes. Because the system stores composite keys (company + title) rather than portal-specific IDs, it can detect the same job posted across multiple platforms.

## Summary

- **Dual storage**: The scraper uses [`job_scraper/seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/job_scraper/seen_jobs.json) for historical tracking and `job_search_tracker.csv` for application status.
- **Multi-key matching**: Duplicates are caught by URL or by company-title combination across both files.
- **Persistent state**: Files are loaded at startup (L40-L42) and updated after every batch (L109-L122), ensuring durability across runs.
- **Universal coverage**: The logic applies to all portals automatically, preventing the same job from appearing regardless of which site sourced it.

## Frequently Asked Questions

### How does the scraper handle duplicate detection across different job portals?

The scraper normalizes jobs using the company-title combination as a secondary key, independent of the source URL. When a job is fetched from any portal, the system checks this composite key against [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json) and `job_search_tracker.csv`. If the same job appears on LinkedIn and Indeed with identical company and title values, the second instance is flagged as a duplicate and skipped.

### What happens if a job is reposted with a different URL but the same title?

The scraper catches this scenario through its secondary deduplication key. Even if the URL changes, the `company|title` combination remains in [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json) from the first fetch. During the check at lines L95-L98, the scraper identifies the matching composite key and marks the reposted job as "skipped," preventing it from appearing in the results list.

### Where does the scraper store its deduplication data?

Persistent state is maintained in two locations within the repository: [`job_scraper/seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/job_scraper/seen_jobs.json) stores the complete history of every processed posting with metadata, and `job_search_tracker.csv` records jobs the user has applied to. Both files are read at the start of each execution and updated atomically after processing, ensuring no data loss between runs.

### Can the deduplication logic be customized or disabled?

The deduplication behavior is hardcoded into the skill workflow defined in [`.claude/skills/job-scraper/SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/.claude/skills/job-scraper/SKILL.md). While the current implementation requires the duplicate checks at lines L95-L98 for data integrity, the JSON structure supports custom status values (such as "archived" or "rejected") that influence filtering. To modify the core logic, developers would need to edit the skill definition file directly.