How Data Scraping for Job Listings Works in ai-job-search: CLI Architecture and Implementation

The ai-job-search framework implements a modular, CLI-driven pipeline that discovers portal skills dynamically, executes board-specific searches, and persists results to JSON for deduplication and downstream processing.

The MadsLorentzen/ai-job-search repository automates job search workflows through a sophisticated data acquisition system. This solution handles data scraping for job listings across multiple portals using a portable, extensible architecture that isolates scraping logic within self-contained command-line modules.

Portal Skill Discovery via File System Walking

When the /scrape command executes, it initiates an automatic discovery process that walks the directory structure .agents/skills/*/SKILL.md. The orchestration logic, defined in .claude/skills/job-scraper/SKILL.md, identifies every portal skill that declares an enabled CLI interface. This file-based discovery mechanism allows the core framework to locate and invoke scraping capabilities without hardcoding portal-specific logic. Each skill descriptor contains metadata that enables the orchestrator to determine which job boards are available for querying.

CLI-Driven Search Implementation

Each portal skill provides a dedicated command-line interface that understands how to query its target job board. For example, the LinkedIn portal ships a TypeScript CLI with an entry point in search.ts, which orchestrates the entire request lifecycle from URL construction to result formatting.

Dynamic URL Construction with Parameter Mapping

The CLI builds search URLs by translating user-provided options into portal-specific query parameters. In .agents/skills/linkedin-search/cli/src/commands/search.ts, the buildUrl function constructs the request using standard URLSearchParams:

function buildUrl(opts: SearchOpts): string {
  const params = new URLSearchParams()
  if (opts.query)   params.set('keywords', opts.query)
  if (opts.location) params.set('location', opts.location)
  const tpr = jobageToTPR(opts.jobage)     // translates “job age” to LinkedIn’s TPR filter
  if (tpr) params.set('f_TPR', tpr)
  const wt = workTypeFlag(opts.remote)     // remote / hybrid / onsite filter
  if (wt) params.set('f_WT', wt)
  params.set('start', String((opts.page - 1) * 10))
  return `${SEARCH_URL}?${params.toString()}`
}

This approach handles pagination via the start parameter (calculated as (page - 1) * 10), job age filters through f_TPR, and work arrangement preferences through f_WT.

HTML Retrieval and Structured Parsing

After constructing the URL, the CLI retrieves raw HTML using htmlFetch and converts it into structured data. The parseJobCards function from helpers.ts processes the HTML into a list of JobCard objects containing standardized fields like id, title, company, location, and date. This abstraction layer ensures that downstream components receive uniform data structures regardless of the source portal's markup.

Flexible Output Rendering

The CLI supports multiple output formats controlled by the --format flag. The renderTable function generates pretty-printed ASCII tables for terminal display, while alternative branches in runSearch handle JSON serialization or plain-text list output. This flexibility allows integration with both human interfaces and automated pipelines:

function renderTable(cards: JobCard[]): string {
  if (cards.length === 0) return "No results."
  const rows = cards.map(c => {
    const title = (c.title || "").slice(0, 42).padEnd(42)
    const company = (c.company || "—").slice(0, 26).padEnd(26)
    const loc = (c.location || "—").slice(0, 24).padEnd(24)
    const date = c.date || "—"
    return `${c.id.padEnd(11)} ${title} ${company} ${loc} ${date}`
  })
  const header =
    "ID".padEnd(11) + " " +
    "TITLE".padEnd(42) + " " +
    "COMPANY".padEnd(26) + " " +
    "LOCATION".padEnd(24) + " DATE"
  return [header, "-".repeat(header.length), ...rows].join("\n")
}

State Persistence and Deduplication

Following a successful scrape, the orchestrator persists new postings to job_scraper/seen_jobs.json. This file maintains a "seen" map that tracks previously encountered job IDs:

# Executed by the core scraper after all portals return results

with open('job_scraper/seen_jobs.json', 'r+') as f:
    data = json.load(f)
    for posting in new_postings:
        data['seen'][posting['id']] = posting
    f.seek(0)
    json.dump(data, f, indent=2)

Downstream commands such as /rank and /apply reference this state file to eliminate duplicate processing and maintain awareness of each posting's lifecycle status.

Health Checks and Extensibility

The architecture includes a health monitoring mechanism accessible via /scrape health. In this mode, the system skips standard search execution and instead sends sentinel requests to each installed portal CLI. This verifies that scraping endpoints remain accessible and that parsing logic can still extract valid results from target sites.

Adding support for additional job boards requires only creating a new folder under .agents/skills/ containing a SKILL.md descriptor and a compatible CLI implementation. The core /scrape workflow automatically incorporates these new modules without requiring changes to the orchestration logic.

Summary

• Discovery-driven architecture: The system automatically detects scraping capabilities by scanning .agents/skills/*/SKILL.md files, eliminating manual registration steps.
• Isolated CLI modules: Each portal implements a standalone CLI (such as the LinkedIn TypeScript client in search.ts) that handles URL construction, HTML fetching, and result parsing.
• Structured data pipeline: Raw HTML transforms into uniform JobCard objects via parseJobCards, enabling consistent downstream processing across heterogeneous job boards.
• JSON-backed state management: The job_scraper/seen_jobs.json file provides durable storage for deduplication and lifecycle tracking across /scrape, /rank, and /apply commands.
• Zero-configuration extensibility: New portals integrate by conforming to the skill directory convention, supporting immediate inclusion in the data scraping workflow.

Frequently Asked Questions

How does the scraper discover new job portals automatically?

The /scrape command walks the .agents/skills/*/SKILL.md directory structure and automatically identifies any portal skill that declares an enabled CLI interface. This file-based discovery mechanism, orchestrated according to .claude/skills/job-scraper/SKILL.md, requires no manual registration or configuration changes to the core framework.

What file stores previously scraped job listings to prevent duplicates?

The system persists scraped data to job_scraper/seen_jobs.json, which contains a "seen" object mapping job IDs to posting details. This state file enables deduplication across command invocations and provides the data foundation for subsequent /rank and /apply operations.

How does the LinkedIn scraper handle pagination and filters?

The LinkedIn CLI in .agents/skills/linkedin-search/cli/src/commands/search.ts constructs search URLs using URLSearchParams with specific filters: f_TPR for job age ranges (translated via jobageToTPR), f_WT for remote work preferences (translated via workTypeFlag), and start for pagination calculated as (page - 1) * 10.

What is the purpose of the /scrape health command?

The health check mode executes sentinel requests against each installed portal CLI to verify that scraping endpoints remain accessible and that HTML parsing logic can still extract valid JobCard data. This probes the entire data acquisition pipeline without triggering full search workflows or modifying the seen_jobs.json state.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}