# How /scrape Auto-Discovers Job Portal Skills in the AI Job Search Framework

> Discover how /scrape auto-discovers job portal skills. Learn about dynamic capability scanning, YAML frontmatter extraction, and parallel execution within the AI Job Search framework.

- Repository: [Mads Lorentzen/ai-job-search](https://github.com/MadsLorentzen/ai-job-search)
- Tags: how-to-guide
- Published: 2026-08-29

---

**/scrape** dynamically discovers job portal search capabilities by scanning the `.agents/skills/` directory for [`SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/SKILL.md) files, reading their YAML frontmatter to extract CLI commands and enabled status, and executing valid portals in parallel while falling back to generic WebSearch for unavailable ones.

The AI Job Search framework eliminates manual configuration for job portal integrations by implementing a data-driven discovery system. When invoked, the `/scrape` command automatically locates and executes search capabilities defined in modular skill definitions, ensuring the scraper stays synchronized with newly added portals without requiring code changes or registry updates.

## Scanning the Skills Directory for Portal Definitions

The discovery process initiates with a filesystem glob targeting the pattern `**/.agents/skills/*/SKILL.md`. This pattern captures every portal skill definition within the repository's `.agents/skills/` directory structure, where each subdirectory represents a distinct job portal such as `linkedin-search`.

```typescript
// Discovery logic scans for portal definitions at runtime
const skillPaths = await glob('**/.agents/skills/*/SKILL.md');

for (const path of skillPaths) {
  const skill = parseYamlFrontmatter(path);
  if (skill.enabled === false) continue;
  
  // Execute the verbatim CLI command from frontmatter
  const results = await agent.run(skill.cli, searchArgs);
}

```

Because the scanner reads these definitions at runtime, any portal added via the `/add-portal` scaffolding command is recognized immediately without requiring framework restarts or configuration reloads.

## Parsing SKILL.md Frontmatter and Validation

Each [`SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/SKILL.md) file contains YAML frontmatter that controls portal activation and execution. The frontmatter specifies an `enabled` boolean flag (defaulting to `true`) and the exact CLI invocation string that `/scrape` must execute.

Portals marked with `enabled: false` are excluded from the current scrape operation. For active portals, the `cli` field provides the verbatim command—typically formatted as `bun run .agents/skills/{portal-name}/cli/src/cli.ts search`. The `/scrape` implementation uses this string exactly as written, never inferring or modifying flags, ensuring consistent behavior across different portal implementations.

```yaml
---
name: linkedin-search
description: Search LinkedIn job listings
enabled: true
cli: "bun run .agents/skills/linkedin-search/cli/src/cli.ts search"
---

```

## Executing Portal CLIs with the JSON Contract

For each enabled portal, `/scrape` executes the extracted CLI command via the `Agent` tool, passing search query arguments as parameters. These commands run in parallel to maximize throughput across multiple job portals.

Every portal CLI must output JSON when invoked with the `--format json` flag. The output must include five mandatory fields: `title`, `company`, `location`, `date`, and `url`. This contract is enforced by the test suite in [`tests/test_scrape_contract.py`](https://github.com/MadsLorentzen/ai-job-search/blob/main/tests/test_scrape_contract.py), which verifies that each portal CLI emits the required schema. The test uses a regex pattern to assert that search outputs contain these specific fields, maintaining data integrity across the aggregation pipeline.

## Fallback to WebSearch for Resilient Operation

The framework implements graceful degradation when portal-specific skills encounter issues. If a portal lacks a [`SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/SKILL.md) definition, if the extracted CLI command fails during execution, or if the `bun` runtime is not installed on the system, `/scrape` automatically falls back to a generic WebSearch implementation for that specific portal.

This fallback mechanism ensures continuous job search coverage even when individual portal integrations are incomplete, encounter technical errors, or run in environments missing required dependencies.

## Extending the Framework with New Portals

The auto-discovery architecture enables zero-configuration extension of the scraping capabilities. When developers create a new portal skill, placing a [`SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/SKILL.md) file in `.agents/skills/{new-portal}/` makes it immediately available to `/scrape` on the next invocation.

As documented in [`.claude/skills/job-scraper/SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/.claude/skills/job-scraper/SKILL.md), the system treats portal skills as data rather than compiled code. This design guarantees that the scraper stays in sync with newly scaffolded portal skills without requiring updates to a central registry or routing table.

## Summary

- **/scrape** uses glob patterns to scan `.agents/skills/*/SKILL.md` files for portal definitions at runtime.
- Each [`SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/SKILL.md) contains YAML frontmatter with an `enabled` flag and verbatim CLI command strings extracted verbatim for execution.
- Enabled portals execute in parallel via the `Agent` tool, with CLIs outputting JSON containing `title`, `company`, `location`, `date`, and `url`.
- The system falls back to generic WebSearch when portal CLIs are missing, disabled, or fail to execute.
- New portals are auto-discovered immediately upon file creation without registry updates or application restarts.

## Frequently Asked Questions

### What file pattern does /scrape use to find job portal skills?

`/scrape` searches for files matching the pattern `**/.agents/skills/*/SKILL.md` within the repository. This glob pattern captures all portal skill definitions located in subdirectories under `.agents/skills/`, where each subdirectory represents a specific job portal integration such as `linkedin-search`.

### How does the framework determine which portals to skip during execution?

The framework parses the YAML frontmatter in each [`SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/SKILL.md) file and evaluates the `enabled` boolean field. If `enabled` is explicitly set to `false`, the portal is excluded from the current scrape operation. When the field is omitted, it defaults to `true`, ensuring portals are active unless explicitly disabled.

### What data format must portal CLIs output to satisfy the scrape contract?

Each portal CLI must output JSON containing exactly five fields: `title`, `company`, `location`, `date`, and `url`. The CLI receives the `--format json` flag to trigger structured output mode. The test suite in [`tests/test_scrape_contract.py`](https://github.com/MadsLorentzen/ai-job-search/blob/main/tests/test_scrape_contract.py) validates that all portal CLIs adhere to this contract before integration.

### What happens if a portal CLI is not installed or fails during execution?

When a portal CLI fails to execute, returns non-zero exit codes, or when the `bun` runtime is unavailable on the host system, `/scrape` automatically falls back to a generic WebSearch implementation for that specific portal. This ensures the job search remains comprehensive even when specific portal integrations encounter technical difficulties or environment constraints.