# Job Scraper Skill Orchestration in Claude: A Complete Guide to the MadsLorentzen AI Job Search Pipeline

> Learn how job scraper skill orchestration automates job discovery via a nine-step pipeline. This guide details parallel CLI execution, WebSearch fallback, and data filtering for job opportunities.

- Repository: [Mads Lorentzen/ai-job-search](https://github.com/MadsLorentzen/ai-job-search)
- Tags: how-to-guide
- Published: 2026-08-26

---

**The job scraper skill orchestration implements a nine-step data-driven pipeline defined in [`.claude/skills/job-scraper/SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/.claude/skills/job-scraper/SKILL.md) that automatically discovers portal-specific CLI tools, executes them in parallel with intelligent WebSearch fallback, and applies deduplication and fit-filtering to generate structured, referral-ready job opportunities.**

The [MadsLorentzen/ai-job-search](https://github.com/MadsLorentzen/ai-job-search) repository implements a sophisticated job scraper skill orchestration that transforms raw portal searches into curated opportunities through automated discovery and stateful evaluation. This Claude-based skill architecture enables dynamic integration of new job portals without code modifications while maintaining persistent deduplication across sessions.

## The Nine-Step Orchestration Pipeline

The job scraper skill orchestration follows a strictly ordered execution flow that balances automation with user oversight.

### 1. State Initialization and Query Loading

The pipeline begins by hydrating context from three persistent data sources. It reads [`job_scraper/seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/job_scraper/seen_jobs.json) to establish the deduplication baseline, loads `job_search_tracker.csv` to exclude previously applied positions, and ingests the user-profile-derived search definitions from [`search-queries.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/search-queries.md). This stateful approach ensures that subsequent steps only process genuinely new opportunities.

### 2. Automated Portal Discovery

Rather than maintaining a hardcoded list of sources, the orchestration dynamically discovers available portals by executing a glob search for `.agents/skills/*/SKILL.md`. Each discovered skill directory represents a potential job portal. The scraper examines the front-matter of each [`SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/SKILL.md) to determine if the portal is **enabled**, creating a runtime inventory of active data sources without requiring configuration file edits.

### 3. Parallel Search Execution with CLI and WebSearch Fallback

For every enabled portal, the orchestration extracts the exact `bun run …` invocation specified in that portal's [`SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/SKILL.md). It then leverages the **Agent** tool to execute these commands concurrently, maximizing throughput across multiple job boards simultaneously.

If `bun` is absent from the environment or a portal CLI returns a non-zero exit code, the orchestration **fails soft**—logging the error but continuing execution—and automatically falls back to **WebSearch** for that specific portal. This dual-mechanism approach ensures coverage even when local CLI tools malfunction.

### 4. Content Acquisition and Field Extraction

For each search result identified as promising, the scraper determines the extraction strategy based on the source mechanism. When processing CLI output, it invokes the portal's built-in `detail` command to retrieve full posting data. For WebSearch fallback results, it employs `WebFetch` to retrieve the posting URL and manually scrapes required fields. The orchestration skips any job already present in [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json) or the tracker CSV, preventing redundant processing.

### 5. Mass-Posting Detection and Consolidation

The pipeline analyzes result clusters to identify near-identical listings distributed across multiple platforms. When detected, the orchestration consolidates these duplicates into a single entry and flags the distribution pattern for user awareness, reducing noise from recruitment agencies posting identical roles across dozens of boards.

### 6. Fit Assessment and Language Validation

Each unique posting undergoes a **Quick Fit Assessment** that assigns a high, medium, or low match rating based on core-skill overlap analysis. The orchestration enforces the **Language Gate** defined in [`04-job-evaluation.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/04-job-evaluation.md), automatically downgrading fit ratings for postings that fail language requirement validation. This filtering ensures only culturally and linguistically appropriate opportunities proceed.

### 7. Persistent State Management and Deduplication Storage

The orchestration writes every processed job—whether new or skipped—into [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json) using a rich schema that records the `portal` source, `mechanism` (either `cli` or `websearch`), `fit` level, and processing `status`. This persistent store drives cross-run deduplication and maintains complete provenance for auditability.

### 8. Portal Health Monitoring and Degradation Detection

After completing the primary search phase, the orchestration enters a health-check mode (internally referenced as Step 4.75). It examines each enabled portal for degradation signals such as empty company fields, repeated HTML entities, or malformed JSON output. Suspect portals are probed once with a sentinel query to confirm functionality.

A special **probe-only** mode triggered by `/scrape health` skips normal search steps and executes only this health validation. When degradation is confirmed, the system reports findings and offers to disable the portal by setting `enable: false` in the respective `.agents/skills/{portal}/SKILL.md` front-matter.

### 9. Results Presentation and User Interaction

The final stage formats results into a sorted table organized by fit level, annotating entries with portal status (disabled, WebSearch fallback) and health warnings. The orchestration then presents actionable prompts for further commands such as `/rank` or detailed evaluation, closing the loop between automated discovery and human decision-making.

## Data-Driven Architecture and Extensibility

The job scraper skill orchestration is deliberately **data-driven** to maximize extensibility. Adding support for a new job portal requires only creating a new directory under `.agents/skills/` containing a compliant [`SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/SKILL.md) with the appropriate front-matter and `bun run` commands. The scraper automatically discovers and integrates the new source without modifying core orchestration logic, treating portal skills as plugins rather than dependencies.

## Command Reference for Job Scraper Orchestration

The following commands trigger specific orchestration behaviors:

```text
/scrape

```

Executes the full nine-step pipeline using default categories defined in [`search-queries.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/search-queries.md).

```text
/scrape data science

```

Runs the orchestration focused on a specific domain category, filtering the query catalog before portal execution.

```text
/scrape health

```

Activates **probe-only** mode, skipping job searches to run exclusively the portal health-check Step 4.75 across all enabled portals.

```text
/scrape health jobnet

```

Probes a single specific portal (even if currently disabled) to verify CLI functionality and data quality.

```text

# After seeing "health: jobnet - degraded ..."

# User confirms in SKILL.md front-matter:

enable: false

```

Disables a problematic portal following health-check warnings by editing the portal's [`.agents/skills/jobnet-search/SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/.agents/skills/jobnet-search/SKILL.md) front-matter.

## Summary

- The job scraper skill orchestration operates from [`.claude/skills/job-scraper/SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/.claude/skills/job-scraper/SKILL.md) and utilizes a nine-step pipeline to process job postings.
- Portal discovery is dynamic via glob search of `.agents/skills/*/SKILL.md`, enabling plugin-style extensibility without code changes.
- Parallel execution uses the **Agent** tool to run multiple `bun run` commands simultaneously, with automatic **WebSearch** fallback for failed CLI attempts.
- State persistence in [`job_scraper/seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/job_scraper/seen_jobs.json) drives deduplication and records provenance including `cli` vs `websearch` mechanisms.
- The **Language Gate** from [`04-job-evaluation.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/04-job-evaluation.md) and fit assessment filter opportunities before referral link generation.
- Portal health monitoring detects degradation and offers automated disabling via front-matter toggles.

## Frequently Asked Questions

### How does the job scraper skill orchestration handle adding new job portals?

The orchestration uses a **data-driven discovery** pattern that eliminates the need for code modifications. Simply create a new subdirectory under `.agents/skills/` containing a [`SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/SKILL.md) file with the required front-matter (including `enable: true` or `false`) and the specific `bun run` command for that portal. The next execution of `/scrape` will automatically discover and integrate the new portal into the parallel search execution.

### What happens when a portal CLI fails during job scraper skill orchestration?

The orchestration implements **fail-soft** error handling where non-zero exit codes or missing `bun` binaries are logged but do not abort the entire pipeline. When a CLI failure is detected, the system automatically invokes **WebSearch** as a fallback mechanism for that specific portal, ensuring continuous job discovery even when local tools malfunction. Failed attempts are recorded in the execution logs for diagnostic purposes.

### How does the job scraper prevent duplicate listings across multiple runs?

Deduplication relies on a persistent state file located at [`job_scraper/seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/job_scraper/seen_jobs.json), which stores a unique identifier for every job encountered regardless of source portal. The orchestration checks each newly discovered posting against this store and the `job_search_tracker.csv` file before processing. The schema records whether duplicates originated from `cli` or `websearch` mechanisms, providing complete provenance tracking across sessions.

### What is the probe-only mode in the job scraper orchestration?

Probe-only mode is a specialized health-check execution triggered by `/scrape health` that bypasses the standard job search pipeline. In this mode, the orchestration runs only Step 4.75 (Portal Health Check), sending sentinel queries to each enabled portal to detect degraded output such as empty fields or HTML entity corruption. This mode allows administrators to verify portal integrity without consuming API quotas or processing time on full job searches.