# How Data Scraping for Job Listings Works in ai-job-search: CLI Architecture and Implementation

> Discover how ai-job-search uses a CLI architecture for dynamic data scraping of job listings. Learn about its modular pipeline and implementation for efficient data acquisition.

- Repository: [Mads Lorentzen/ai-job-search](https://github.com/MadsLorentzen/ai-job-search)
- Tags: architecture
- Published: 2026-07-25

---

**The ai-job-search framework implements a modular, CLI-driven pipeline that discovers portal skills dynamically, executes board-specific searches, and persists results to JSON for deduplication and downstream processing.**

The MadsLorentzen/ai-job-search repository automates job search workflows through a sophisticated data acquisition system. This solution handles data scraping for job listings across multiple portals using a portable, extensible architecture that isolates scraping logic within self-contained command-line modules.

## Portal Skill Discovery via File System Walking

When the `/scrape` command executes, it initiates an automatic discovery process that walks the directory structure `.agents/skills/*/SKILL.md`. The orchestration logic, defined in [`.claude/skills/job-scraper/SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/.claude/skills/job-scraper/SKILL.md), identifies every portal skill that declares an enabled CLI interface. This file-based discovery mechanism allows the core framework to locate and invoke scraping capabilities without hardcoding portal-specific logic. Each skill descriptor contains metadata that enables the orchestrator to determine which job boards are available for querying.

## CLI-Driven Search Implementation

Each portal skill provides a dedicated command-line interface that understands how to query its target job board. For example, the LinkedIn portal ships a TypeScript CLI with an entry point in [`search.ts`](https://github.com/MadsLorentzen/ai-job-search/blob/main/search.ts), which orchestrates the entire request lifecycle from URL construction to result formatting.

### Dynamic URL Construction with Parameter Mapping

The CLI builds search URLs by translating user-provided options into portal-specific query parameters. In [`.agents/skills/linkedin-search/cli/src/commands/search.ts`](https://github.com/MadsLorentzen/ai-job-search/blob/main/.agents/skills/linkedin-search/cli/src/commands/search.ts), the `buildUrl` function constructs the request using standard `URLSearchParams`:

```typescript
function buildUrl(opts: SearchOpts): string {
  const params = new URLSearchParams()
  if (opts.query)   params.set('keywords', opts.query)
  if (opts.location) params.set('location', opts.location)
  const tpr = jobageToTPR(opts.jobage)     // translates “job age” to LinkedIn’s TPR filter
  if (tpr) params.set('f_TPR', tpr)
  const wt = workTypeFlag(opts.remote)     // remote / hybrid / onsite filter
  if (wt) params.set('f_WT', wt)
  params.set('start', String((opts.page - 1) * 10))
  return `${SEARCH_URL}?${params.toString()}`
}

```

This approach handles pagination via the `start` parameter (calculated as `(page - 1) * 10`), job age filters through `f_TPR`, and work arrangement preferences through `f_WT`.

### HTML Retrieval and Structured Parsing

After constructing the URL, the CLI retrieves raw HTML using `htmlFetch` and converts it into structured data. The `parseJobCards` function from [`helpers.ts`](https://github.com/MadsLorentzen/ai-job-search/blob/main/helpers.ts) processes the HTML into a list of **JobCard** objects containing standardized fields like `id`, `title`, `company`, `location`, and `date`. This abstraction layer ensures that downstream components receive uniform data structures regardless of the source portal's markup.

### Flexible Output Rendering

The CLI supports multiple output formats controlled by the `--format` flag. The `renderTable` function generates pretty-printed ASCII tables for terminal display, while alternative branches in `runSearch` handle JSON serialization or plain-text list output. This flexibility allows integration with both human interfaces and automated pipelines:

```typescript
function renderTable(cards: JobCard[]): string {
  if (cards.length === 0) return "No results."
  const rows = cards.map(c => {
    const title = (c.title || "").slice(0, 42).padEnd(42)
    const company = (c.company || "—").slice(0, 26).padEnd(26)
    const loc = (c.location || "—").slice(0, 24).padEnd(24)
    const date = c.date || "—"
    return `${c.id.padEnd(11)} ${title} ${company} ${loc} ${date}`
  })
  const header =
    "ID".padEnd(11) + " " +
    "TITLE".padEnd(42) + " " +
    "COMPANY".padEnd(26) + " " +
    "LOCATION".padEnd(24) + " DATE"
  return [header, "-".repeat(header.length), ...rows].join("\n")
}

```

## State Persistence and Deduplication

Following a successful scrape, the orchestrator persists new postings to [`job_scraper/seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/job_scraper/seen_jobs.json). This file maintains a `"seen"` map that tracks previously encountered job IDs:

```python

# Executed by the core scraper after all portals return results

with open('job_scraper/seen_jobs.json', 'r+') as f:
    data = json.load(f)
    for posting in new_postings:
        data['seen'][posting['id']] = posting
    f.seek(0)
    json.dump(data, f, indent=2)

```

Downstream commands such as `/rank` and `/apply` reference this state file to eliminate duplicate processing and maintain awareness of each posting's lifecycle status.

## Health Checks and Extensibility

The architecture includes a health monitoring mechanism accessible via `/scrape health`. In this mode, the system skips standard search execution and instead sends sentinel requests to each installed portal CLI. This verifies that scraping endpoints remain accessible and that parsing logic can still extract valid results from target sites.

Adding support for additional job boards requires only creating a new folder under `.agents/skills/` containing a [`SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/SKILL.md) descriptor and a compatible CLI implementation. The core `/scrape` workflow automatically incorporates these new modules without requiring changes to the orchestration logic.

## Summary

- **Discovery-driven architecture**: The system automatically detects scraping capabilities by scanning `.agents/skills/*/SKILL.md` files, eliminating manual registration steps.
- **Isolated CLI modules**: Each portal implements a standalone CLI (such as the LinkedIn TypeScript client in [`search.ts`](https://github.com/MadsLorentzen/ai-job-search/blob/main/search.ts)) that handles URL construction, HTML fetching, and result parsing.
- **Structured data pipeline**: Raw HTML transforms into uniform **JobCard** objects via `parseJobCards`, enabling consistent downstream processing across heterogeneous job boards.
- **JSON-backed state management**: The [`job_scraper/seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/job_scraper/seen_jobs.json) file provides durable storage for deduplication and lifecycle tracking across `/scrape`, `/rank`, and `/apply` commands.
- **Zero-configuration extensibility**: New portals integrate by conforming to the skill directory convention, supporting immediate inclusion in the data scraping workflow.

## Frequently Asked Questions

### How does the scraper discover new job portals automatically?

The `/scrape` command walks the `.agents/skills/*/SKILL.md` directory structure and automatically identifies any portal skill that declares an enabled CLI interface. This file-based discovery mechanism, orchestrated according to [`.claude/skills/job-scraper/SKILL.md`](https://github.com/MadsLorentzen/ai-job-search/blob/main/.claude/skills/job-scraper/SKILL.md), requires no manual registration or configuration changes to the core framework.

### What file stores previously scraped job listings to prevent duplicates?

The system persists scraped data to [`job_scraper/seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/job_scraper/seen_jobs.json), which contains a `"seen"` object mapping job IDs to posting details. This state file enables deduplication across command invocations and provides the data foundation for subsequent `/rank` and `/apply` operations.

### How does the LinkedIn scraper handle pagination and filters?

The LinkedIn CLI in [`.agents/skills/linkedin-search/cli/src/commands/search.ts`](https://github.com/MadsLorentzen/ai-job-search/blob/main/.agents/skills/linkedin-search/cli/src/commands/search.ts) constructs search URLs using `URLSearchParams` with specific filters: `f_TPR` for job age ranges (translated via `jobageToTPR`), `f_WT` for remote work preferences (translated via `workTypeFlag`), and `start` for pagination calculated as `(page - 1) * 10`.

### What is the purpose of the `/scrape health` command?

The health check mode executes sentinel requests against each installed portal CLI to verify that scraping endpoints remain accessible and that HTML parsing logic can still extract valid **JobCard** data. This probes the entire data acquisition pipeline without triggering full search workflows or modifying the [`seen_jobs.json`](https://github.com/MadsLorentzen/ai-job-search/blob/main/seen_jobs.json) state.