Job Scraper Skill Orchestration in Claude: A Complete Guide to the MadsLorentzen AI Job Search Pipeline

The job scraper skill orchestration implements a nine-step data-driven pipeline defined in .claude/skills/job-scraper/SKILL.md that automatically discovers portal-specific CLI tools, executes them in parallel with intelligent WebSearch fallback, and applies deduplication and fit-filtering to generate structured, referral-ready job opportunities.

The MadsLorentzen/ai-job-search repository implements a sophisticated job scraper skill orchestration that transforms raw portal searches into curated opportunities through automated discovery and stateful evaluation. This Claude-based skill architecture enables dynamic integration of new job portals without code modifications while maintaining persistent deduplication across sessions.

The Nine-Step Orchestration Pipeline

The job scraper skill orchestration follows a strictly ordered execution flow that balances automation with user oversight.

1. State Initialization and Query Loading

The pipeline begins by hydrating context from three persistent data sources. It reads job_scraper/seen_jobs.json to establish the deduplication baseline, loads job_search_tracker.csv to exclude previously applied positions, and ingests the user-profile-derived search definitions from search-queries.md. This stateful approach ensures that subsequent steps only process genuinely new opportunities.

2. Automated Portal Discovery

Rather than maintaining a hardcoded list of sources, the orchestration dynamically discovers available portals by executing a glob search for .agents/skills/*/SKILL.md. Each discovered skill directory represents a potential job portal. The scraper examines the front-matter of each SKILL.md to determine if the portal is enabled, creating a runtime inventory of active data sources without requiring configuration file edits.

3. Parallel Search Execution with CLI and WebSearch Fallback

For every enabled portal, the orchestration extracts the exact bun run … invocation specified in that portal's SKILL.md. It then leverages the Agent tool to execute these commands concurrently, maximizing throughput across multiple job boards simultaneously.

If bun is absent from the environment or a portal CLI returns a non-zero exit code, the orchestration fails soft—logging the error but continuing execution—and automatically falls back to WebSearch for that specific portal. This dual-mechanism approach ensures coverage even when local CLI tools malfunction.

4. Content Acquisition and Field Extraction

For each search result identified as promising, the scraper determines the extraction strategy based on the source mechanism. When processing CLI output, it invokes the portal's built-in detail command to retrieve full posting data. For WebSearch fallback results, it employs WebFetch to retrieve the posting URL and manually scrapes required fields. The orchestration skips any job already present in seen_jobs.json or the tracker CSV, preventing redundant processing.

5. Mass-Posting Detection and Consolidation

The pipeline analyzes result clusters to identify near-identical listings distributed across multiple platforms. When detected, the orchestration consolidates these duplicates into a single entry and flags the distribution pattern for user awareness, reducing noise from recruitment agencies posting identical roles across dozens of boards.

6. Fit Assessment and Language Validation

Each unique posting undergoes a Quick Fit Assessment that assigns a high, medium, or low match rating based on core-skill overlap analysis. The orchestration enforces the Language Gate defined in 04-job-evaluation.md, automatically downgrading fit ratings for postings that fail language requirement validation. This filtering ensures only culturally and linguistically appropriate opportunities proceed.

7. Persistent State Management and Deduplication Storage

The orchestration writes every processed job—whether new or skipped—into seen_jobs.json using a rich schema that records the portal source, mechanism (either cli or websearch), fit level, and processing status. This persistent store drives cross-run deduplication and maintains complete provenance for auditability.

8. Portal Health Monitoring and Degradation Detection

After completing the primary search phase, the orchestration enters a health-check mode (internally referenced as Step 4.75). It examines each enabled portal for degradation signals such as empty company fields, repeated HTML entities, or malformed JSON output. Suspect portals are probed once with a sentinel query to confirm functionality.

A special probe-only mode triggered by /scrape health skips normal search steps and executes only this health validation. When degradation is confirmed, the system reports findings and offers to disable the portal by setting enable: false in the respective .agents/skills/{portal}/SKILL.md front-matter.

9. Results Presentation and User Interaction

The final stage formats results into a sorted table organized by fit level, annotating entries with portal status (disabled, WebSearch fallback) and health warnings. The orchestration then presents actionable prompts for further commands such as /rank or detailed evaluation, closing the loop between automated discovery and human decision-making.

Data-Driven Architecture and Extensibility

The job scraper skill orchestration is deliberately data-driven to maximize extensibility. Adding support for a new job portal requires only creating a new directory under .agents/skills/ containing a compliant SKILL.md with the appropriate front-matter and bun run commands. The scraper automatically discovers and integrates the new source without modifying core orchestration logic, treating portal skills as plugins rather than dependencies.

Command Reference for Job Scraper Orchestration

The following commands trigger specific orchestration behaviors:

/scrape

Executes the full nine-step pipeline using default categories defined in search-queries.md.

/scrape data science

Runs the orchestration focused on a specific domain category, filtering the query catalog before portal execution.

/scrape health

Activates probe-only mode, skipping job searches to run exclusively the portal health-check Step 4.75 across all enabled portals.

/scrape health jobnet

Probes a single specific portal (even if currently disabled) to verify CLI functionality and data quality.

# After seeing "health: jobnet - degraded ..."

# User confirms in SKILL.md front-matter:

enable: false

Disables a problematic portal following health-check warnings by editing the portal's .agents/skills/jobnet-search/SKILL.md front-matter.

Summary

• The job scraper skill orchestration operates from .claude/skills/job-scraper/SKILL.md and utilizes a nine-step pipeline to process job postings.
• Portal discovery is dynamic via glob search of .agents/skills/*/SKILL.md, enabling plugin-style extensibility without code changes.
• Parallel execution uses the Agent tool to run multiple bun run commands simultaneously, with automatic WebSearch fallback for failed CLI attempts.
• State persistence in job_scraper/seen_jobs.json drives deduplication and records provenance including cli vs websearch mechanisms.
• The Language Gate from 04-job-evaluation.md and fit assessment filter opportunities before referral link generation.
• Portal health monitoring detects degradation and offers automated disabling via front-matter toggles.

Frequently Asked Questions

How does the job scraper skill orchestration handle adding new job portals?

The orchestration uses a data-driven discovery pattern that eliminates the need for code modifications. Simply create a new subdirectory under .agents/skills/ containing a SKILL.md file with the required front-matter (including enable: true or false) and the specific bun run command for that portal. The next execution of /scrape will automatically discover and integrate the new portal into the parallel search execution.

What happens when a portal CLI fails during job scraper skill orchestration?

The orchestration implements fail-soft error handling where non-zero exit codes or missing bun binaries are logged but do not abort the entire pipeline. When a CLI failure is detected, the system automatically invokes WebSearch as a fallback mechanism for that specific portal, ensuring continuous job discovery even when local tools malfunction. Failed attempts are recorded in the execution logs for diagnostic purposes.

How does the job scraper prevent duplicate listings across multiple runs?

Deduplication relies on a persistent state file located at job_scraper/seen_jobs.json, which stores a unique identifier for every job encountered regardless of source portal. The orchestration checks each newly discovered posting against this store and the job_search_tracker.csv file before processing. The schema records whether duplicates originated from cli or websearch mechanisms, providing complete provenance tracking across sessions.

What is the probe-only mode in the job scraper orchestration?

Probe-only mode is a specialized health-check execution triggered by /scrape health that bypasses the standard job search pipeline. In this mode, the orchestration runs only Step 4.75 (Portal Health Check), sending sentinel queries to each enabled portal to detect degraded output such as empty fields or HTML entity corruption. This mode allows administrators to verify portal integrity without consuming API quotas or processing time on full job searches.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}