How the Portal Scanner Differentiates API Discovery from Playwright Verification in Career-Ops

The portal scanner separates lightweight HTTP/JSON API discovery from resource-intensive Playwright browser verification, using provider modules for the former and an opt-in --verify flag to trigger headless Chromium validation.

The santifer/career-ops repository implements a two-phase job scanning architecture that cleanly separates fast, token-free API queries from expensive browser automation. This design allows users to rapidly discover job postings via lightweight HTTP requests while optionally validating those URLs through full browser rendering only when necessary.

API Discovery Phase: Zero-Token HTTP Fetching

The first phase relies entirely on provider modules that communicate with Applicant Tracking Systems (ATS) and job boards using standard HTTP requests and JSON parsing. This "zero-Claude" mode requires no browser instances and consumes minimal resources.

Provider Loading and Resolution

The scanner dynamically loads provider implementations based on the portals.yml configuration. In scan.mjs, the loadProviders() function (lines 55-84) reads the configuration and prepares the provider registry, while resolveProvider() (lines 87-119) matches each company entry to its specific provider implementation (e.g., providers/greenhouse.mjs or providers/lever.mjs).

Each provider implements a standard interface with detect() and fetch() methods. The resolution logic examines the company entry's provider field to determine which module handles that specific ATS.

Fetching Job Listings via HTTP

Once resolved, the scanner calls provider.fetch(entry, ctx) (lines 124-131 in scan.mjs) to retrieve job listings. This method returns an array of objects containing title, url, company, location, and other metadata:

// Inside scan.mjs – provider fetch (API only)
const jobs = await provider.fetch(company, ctx);   // ← HTTP/JSON, no Playwright

The underlying HTTP logic lives in providers/_http.mjs, which provides a minimal fetch wrapper for interacting with ATS APIs. This phase completes entirely without launching a browser, making it suitable for frequent, scheduled scans.

Playwright Verification Phase: Browser-Based Validation

The second phase occurs only when the user explicitly requests it via the --verify CLI flag. This phase launches a headless Chromium browser to verify that discovered URLs are still live and contain active "Apply" controls.

The --verify CLI Flag

The scanner parses CLI arguments at lines 100-106 in scan.mjs, extracting the verify boolean flag. When present, this triggers the verifyOffers() function (lines 779-885) after the initial API discovery completes:

// Inside scan.mjs – verification (Playwright)
if (verify && newOffers.length > 0) {
  const result = await verifyOffers(newOffers, {
    headedFallback,
    throttleBaseMs,
    rediscover,
  });
  // result.verified contains only live URLs
}

Headless Chromium and Liveness Checks

The verifyOffers() function launches Playwright's chromium.launch({headless:true}) and iterates through each discovered URL. It calls checkUrlLiveness(page, url) from liveness-browser.mjs to categorize postings as active, expired, no-apply-control, or invalid.

Expired or invalid items are excluded from pipeline.md but recorded in scan-history.tsv for deduplication purposes. This prevents the scanner from re-processing dead links in future runs while maintaining a record of verification attempts.

How the Two Phases Interact

The scanner maintains strict separation between discovery and verification. The API phase produces a list of candidate URLs, while the verification phase filters that list based on actual browser rendering. This pipeline ensures that:

• Fast-path execution: Running node scan.mjs without flags performs only API discovery, writing new offers directly to pipeline.md
• Thorough validation: Running node scan.mjs --verify adds the browser confirmation step, ensuring only live, apply-able jobs reach the pipeline

# 1️⃣ Run a pure-API scan (no browser)

node scan.mjs               # only http requests → new offers go to pipeline.md

# 2️⃣ Run a scan that also verifies each URL with Playwright

node scan.mjs --verify      # fetch via API, then open each URL in headless Chromium

Summary

• API discovery is the default, lightweight mode that uses provider modules (providers/greenhouse.mjs, providers/lever.mjs) to fetch job data via HTTP/JSON without browser overhead.
• Playwright verification is opt-in via the --verify flag and executes only after API discovery, using headless Chromium to confirm URL liveness and the presence of apply controls.
• Source locations: Provider loading occurs in scan.mjs lines 55-119, fetching at lines 124-131, and verification logic spans lines 779-885 with helper functions in liveness-browser.mjs.
• Output handling: Verified active jobs go to pipeline.md, while expired entries are logged to scan-history.tsv for deduplication.

Frequently Asked Questions

What triggers Playwright verification in the career-ops scanner?

Playwright verification only triggers when you pass the --verify flag to the CLI. Without this flag, the scanner runs in API-only mode, performing HTTP requests through provider modules without launching any browser instances.

Which files handle the HTTP-based API discovery?

API discovery logic resides in scan.mjs (lines 55-131) for provider loading and resolution, with individual provider implementations located in providers/*.mjs files (such as greenhouse.mjs and lever.mjs). The shared HTTP utilities live in providers/_http.mjs.

How does the scanner determine if a job URL is still active?

During verification, the scanner uses checkUrlLiveness() from liveness-browser.mjs to load each URL in a headless Chromium browser via Playwright. It checks for the presence of apply controls and valid page content, categorizing results as active, expired, no-apply-control, or invalid.

Can I run the scanner without any browser automation?

Yes. Running node scan.mjs without the --verify flag executes only the API discovery phase. This uses zero browser instances and relies solely on HTTP requests to ATS APIs, making it suitable for automated schedules or environments without Playwright dependencies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}