Job Scraper Parallelization Strategy: Multi-Agent Concurrency in AI-Job-Search

The job scraper uses Claude Code's parallel-agent runtime to spawn concurrent agents for each job portal, executing portal-specific CLI tools simultaneously and aggregating results into a shared JSON state file.

The MadsLorentzen/ai-job-search repository implements a sophisticated job aggregation system that searches across multiple Danish and international job portals simultaneously. Rather than processing portals sequentially, the scraper employs an agent-based parallelization strategy that significantly reduces search latency by running all portal queries concurrently through Claude Code's built-in parallel-agent runtime.

Agent-Based Parallelization Strategy

The parallelization strategy centers on Claude Code's parallel-agent runtime, which orchestrates multiple independent agents rather than traditional thread-based or process-based concurrency. According to the job-scraper skill definition in .claude/skills/job-scraper/SKILL.md, the system is designed to "scrape jobs" across multiple portals concurrently.

For each configured portal, the scraper spawns a dedicated Claude Code agent that executes the corresponding portal-specific CLI tool:

• jobbank-search
• jobdanmark-search
• jobindex-search
• jobnet-search
• linkedin-search
• freehire-search

Each agent runs independently, executing its respective CLI implementation found in .agents/skills/{portal-name}/cli/src/cli.ts.

Parallel Execution Flow

The scraper initiates concurrent searches using a Promise-based pattern that launches all portal agents simultaneously. As documented in .claude/commands/rank.md, "parallel agents fetch each posting"—the same mechanism powers the /scrape command.

Each parallel agent executes the search command using Bun:

// Launch one agent per portal – each runs its CLI `search` command
await Promise.all(
  portals.map((tool) =>
    // Claude Code creates a parallel agent to run the CLI
    runAgent({
      command: `bun run .agents/skills/${tool}/cli/src/cli.ts search`,
      // each agent writes its output to `job_scraper/seen_jobs.json`
      outputPath: "job_scraper/seen_jobs.json",
    })
  )
);

This approach ensures that slow-responding portals do not block fast ones, as each agent operates on its own timeline.

Shared State and Deduplication

All parallel agents write their results to a single shared state file: job_scraper/seen_jobs.json. This file acts as a concurrent collection point where each agent appends discovered job postings during its execution.

After all agents complete via Promise.all, the scraper performs a deduplication phase to merge results:

// After all agents finish, the scraper deduplicates the JSON entries
deduplicateSeenJobs();

The deduplication process filters duplicate postings that may appear across multiple portals (e.g., the same job listed on both LinkedIn and Jobindex), presenting the user with a unified, unique list of opportunities.

Portal-Specific Implementation Details

Each portal runs its own search logic within the parallel agent framework. The CLI implementations reside in:

• .agents/skills/jobbank-search/cli/src/cli.ts
• .agents/skills/jobdanmark-search/cli/src/cli.ts
• .agents/skills/jobindex-search/cli/src/cli.ts
• .agents/skills/jobnet-search/cli/src/cli.ts
• .agents/skills/linkedin-search/cli/src/cli.ts
• .agents/skills/freehire-search/cli/src/cli.ts

These modules handle portal-specific authentication, HTML parsing, and API rate limiting, while the parent scraper manages the orchestration layer.

Summary

• Agent-based concurrency: The scraper uses Claude Code's parallel-agent runtime rather than traditional threads or processes to execute searches concurrently.
• One agent per portal: Each job portal (Jobindex, LinkedIn, Jobnet, etc.) receives its own dedicated agent that runs the portal-specific CLI independently.
• Shared state architecture: All agents write results to job_scraper/seen_jobs.json, creating a centralized data collection point.
• Post-processing deduplication: After parallel execution completes, the system deduplicates entries to eliminate cross-portal job posting duplicates.
• Performance benefits: The parallelization strategy prevents slow portals from blocking fast ones, significantly reducing total search time across all sources.

Frequently Asked Questions

What job portals does the parallel scraper support?

The scraper supports six major portals: Jobbank, Jobdanmark, Jobindex, Jobnet, LinkedIn, and Freehire. Each portal has its own dedicated agent and CLI implementation located in .agents/skills/{portal-name}-search/cli/src/cli.ts, allowing the system to handle each site's specific search parameters and response formats concurrently.

How does the scraper prevent duplicate job postings across portals?

After all parallel agents complete their execution, the scraper runs a deduplicateSeenJobs() function against the shared job_scraper/seen_jobs.json file. This process identifies and removes duplicate entries that may appear on multiple platforms (such as the same employer posting on both LinkedIn and Jobnet), ensuring the final results contain only unique opportunities.

Is the parallelization handled by Node.js or external tools?

The parallelization is handled by Claude Code's built-in parallel-agent runtime, not native Node.js worker threads or child processes. The scraper simply launches agents using the runtime's API, and Claude Code manages the concurrent execution, resource allocation, and result collection automatically.

Where are intermediate results stored during parallel execution?

During execution, each agent writes its discovered job postings to job_scraper/seen_jobs.json located in the repository root. This JSON file serves as a shared state container that persists data across the parallel agent boundary, allowing the system to aggregate results from all portals before performing final deduplication and presentation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}