How the AI Job Search Framework Handles Job Scraping: A Multi-Step Pipeline

The AI Job Search Framework implements job scraping through a deterministic pipeline defined in /.claude/skills/job-scraper/SKILL.md that auto-discovers portal CLIs, falls back to web search when necessary, and applies intelligent deduplication to maintain a clean dataset of opportunities.

The MadsLorentzen/ai-job-search repository provides a modular architecture for automated job searching built around self-describing portal skills. The framework's scraping logic operates through the Job-Scraper skill, which orchestrates data collection from multiple job portals while maintaining strict deduplication standards and quality controls.

The Job Scraping Pipeline

The AI Job Search Framework job scraping process follows a deterministic workflow that initializes state, queries multiple sources, and persists results. Each step is documented in the master specification at /.claude/skills/job-scraper/SKILL.md.

Step 0: Load State

The scraper begins by establishing context from persistent storage. It reads job_scraper/seen_jobs.json (creating the file if missing), the application tracker CSV, and search-queries.md to understand the current search strategy and previously seen listings.

Step 1: Search with CLI Discovery

The framework implements a thin-pointer design for portal integration. It automatically discovers every portal skill by scanning .agents/skills/*/SKILL.md files.

During execution:

• Checks for bun (the JavaScript runtime); if missing, skips to WebSearch fallback
• Runs each enabled portal's CLI according to its documented flags
• Limits each call to ~20 results with JSON output
• Applies a 14-day recency filter client-side when portals lack native date filtering

Portal participation is controlled via the enabled toggle in each skill's front-matter.

Step 2: Fetch and Parse Details

For each promising result, the framework extracts full job details through two methods:

• Portal CLI detail command: Retrieves title, company, location, deadline, and description
• WebFetch fallback: Handles 403 responses with browser-like headers for WebSearch results

Duplicate detection runs against both seen_jobs.json and job_search_tracker.csv to prevent re-processing.

Mass-Posting Detection (Step 2.5)

The framework identifies clusters of near-identical listings from the same company—such as identical descriptions across multiple cities—and consolidates them into single entries. This prevents pollution of the dataset with spammed postings while flagging the geographic spread for user awareness.

Quick Fit Assessment (Step 3)

Each posting receives a high, medium, or low fit label based on alignment with the user's core skills. A language-gate override forces a low match if required languages are missing from the candidate profile, ensuring realistic opportunity targeting.

Deduplication and Storage (Step 4)

New and skipped jobs persist to job_scraper/seen_jobs.json using a versioned schema that records:

• Title, company, URL, deadline
• First-seen date and portal source
• Fit rating and status
• Origin type (CLI vs WebSearch)

This additive schema ensures backward compatibility when new fields like rank_score are introduced.

Referral Link Generation (Step 4.5)

For high and medium-fit opportunities, the framework generates two LinkedIn people-search URLs targeting recruiters and peers. These URLs are presented to the user but never fetched programmatically, respecting LinkedIn's terms of service.

Portal Health Validation (Step 4.75)

The system validates each portal's CLI output for degradation signals such as missing company names or empty titles. When a portal appears broken, it performs a bounded sentinel probe and reports results in the final output table, enabling quick remediation.

Results Presentation (Step 5)

The scraper outputs a markdown table ordered by fit score, including:

• Disabled portals for visibility
• Fallback source indicators
• Health diagnostics

The workflow concludes with prompts for further actions like detailed evaluation or the /rank command.

Optional Tracker Updates (Step 6)

If users apply to jobs outside the standard flow, the skill can manually append rows to the tracker CSV. Otherwise, the /rank command handles status synchronization.

Key Design Principles

The framework's scraping architecture follows five core principles defined in the source specification:

1. Thin-Pointer Design

Portal skills are self-describing via SKILL.md files and automatically discovered. New portals integrate through the /add-portal command without requiring modifications to the core scraper code.

2. Deterministic Deduplication

The system checks both seen_jobs.json and the application tracker CSV to eliminate redundant listings across all data sources.

3. Graceful Degradation

When bun is unavailable or a portal CLI fails, the framework seamlessly falls back to generic WebSearch, preserving scraping continuity.

4. Health-Check Feedback

Step 4.75 surfaces silent CLI breakages to users, enabling rapid identification of portals requiring the enabled: false toggle.

5. Extensible Schema

The JSON storage format supports additive versioning, ensuring existing data remains valid as the framework evolves.

Working with the Scraper

Trigger the scraping workflow through the framework's command interface:

# Trigger the complete scraping pipeline

!ai-job-search /scrape

Manually invoke individual portal CLIs according to their documented contracts:

# Example: Direct portal CLI invocation

cd .agents/skills/jobindex-search && bun run cli/src/cli.ts search --format json --limit 20 "data scientist"

The WebSearch fallback triggers automatically when portal CLIs are unavailable or disabled—no explicit user action required.

Summary

• The AI Job Search Framework uses a deterministic multi-step pipeline defined in /.claude/skills/job-scraper/SKILL.md
• Auto-discovery of portal CLIs via .agents/skills/*/SKILL.md enables modular expansion without code changes
• Dual deduplication against job_scraper/seen_jobs.json and job_search_tracker.csv prevents redundant processing
• Mass-posting detection consolidates spammed listings while preserving geographic diversity data
• Graceful degradation to WebSearch ensures continuous operation when bun or portal CLIs fail
• Portal health checks validate CLI output quality and report degradation in real-time

Frequently Asked Questions

What happens if a portal CLI fails during scraping?

When a portal CLI fails or returns degraded data, the framework executes Step 4.75 (Portal Health Check) to perform a bounded sentinel probe. If the portal is confirmed broken, it reports the degradation in the results table and automatically falls back to WebSearch for that specific source. Users can then toggle enabled: false in the portal's SKILL.md front-matter to skip the broken CLI in future runs.

How does the framework prevent duplicate job listings?

The scraper implements deterministic deduplication by checking two sources: the persistent job_scraper/seen_jobs.json file and the job_search_tracker.csv application log. Every fetched job is checked against these stores before processing. Additionally, Step 2.5 detects and consolidates mass-posted listings from the same company to prevent dataset pollution.

What is the "Thin-Pointer Design" principle?

Thin-Pointer Design refers to the framework's architecture where portal skills are self-describing through SKILL.md files and automatically discovered at runtime. New job portals integrate by adding their CLI to .agents/skills/ without modifying the core scraper code in /.claude/skills/job-scraper/SKILL.md. This decoupling allows the ecosystem to expand through simple file additions rather than code changes.

How does the Quick Fit Assessment prioritize job opportunities?

The Quick Fit Assessment (Step 3) assigns high, medium, or low labels based on semantic matching between the job description and the user's core skills. A language-gate override enforces a low rating if the posting requires languages not present in the candidate profile, ensuring the prioritization reflects realistic eligibility criteria rather than keyword density alone.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}