# How the Portal Scanner Differentiates API Discovery from Playwright Verification in Career-Ops

> Learn how the portal scanner distinguishes API discovery from Playwright verification. It uses provider modules for lightweight API scans and an opt-in flag for headless Chromium validation.

- Repository: [Santiago Fernández de Valderrama/career-ops](https://github.com/santifer/career-ops)
- Tags: architecture
- Published: 2026-06-13

---

**The portal scanner separates lightweight HTTP/JSON API discovery from resource-intensive Playwright browser verification, using provider modules for the former and an opt-in `--verify` flag to trigger headless Chromium validation.**

The `santifer/career-ops` repository implements a two-phase job scanning architecture that cleanly separates fast, token-free API queries from expensive browser automation. This design allows users to rapidly discover job postings via lightweight HTTP requests while optionally validating those URLs through full browser rendering only when necessary.

## API Discovery Phase: Zero-Token HTTP Fetching

The first phase relies entirely on provider modules that communicate with Applicant Tracking Systems (ATS) and job boards using standard HTTP requests and JSON parsing. This "zero-Claude" mode requires no browser instances and consumes minimal resources.

### Provider Loading and Resolution

The scanner dynamically loads provider implementations based on the [`portals.yml`](https://github.com/santifer/career-ops/blob/main/portals.yml) configuration. In `scan.mjs`, the `loadProviders()` function (lines 55-84) reads the configuration and prepares the provider registry, while `resolveProvider()` (lines 87-119) matches each company entry to its specific provider implementation (e.g., `providers/greenhouse.mjs` or `providers/lever.mjs`).

Each provider implements a standard interface with `detect()` and `fetch()` methods. The resolution logic examines the company entry's `provider` field to determine which module handles that specific ATS.

### Fetching Job Listings via HTTP

Once resolved, the scanner calls `provider.fetch(entry, ctx)` (lines 124-131 in `scan.mjs`) to retrieve job listings. This method returns an array of objects containing `title`, `url`, `company`, `location`, and other metadata:

```javascript
// Inside scan.mjs – provider fetch (API only)
const jobs = await provider.fetch(company, ctx);   // ← HTTP/JSON, no Playwright

```

The underlying HTTP logic lives in `providers/_http.mjs`, which provides a minimal fetch wrapper for interacting with ATS APIs. This phase completes entirely without launching a browser, making it suitable for frequent, scheduled scans.

## Playwright Verification Phase: Browser-Based Validation

The second phase occurs only when the user explicitly requests it via the `--verify` CLI flag. This phase launches a headless Chromium browser to verify that discovered URLs are still live and contain active "Apply" controls.

### The `--verify` CLI Flag

The scanner parses CLI arguments at lines 100-106 in `scan.mjs`, extracting the `verify` boolean flag. When present, this triggers the `verifyOffers()` function (lines 779-885) after the initial API discovery completes:

```javascript
// Inside scan.mjs – verification (Playwright)
if (verify && newOffers.length > 0) {
  const result = await verifyOffers(newOffers, {
    headedFallback,
    throttleBaseMs,
    rediscover,
  });
  // result.verified contains only live URLs
}

```

### Headless Chromium and Liveness Checks

The `verifyOffers()` function launches Playwright's `chromium.launch({headless:true})` and iterates through each discovered URL. It calls `checkUrlLiveness(page, url)` from `liveness-browser.mjs` to categorize postings as **active**, **expired**, **no-apply-control**, or **invalid**.

Expired or invalid items are excluded from [`pipeline.md`](https://github.com/santifer/career-ops/blob/main/pipeline.md) but recorded in `scan-history.tsv` for deduplication purposes. This prevents the scanner from re-processing dead links in future runs while maintaining a record of verification attempts.

## How the Two Phases Interact

The scanner maintains strict separation between discovery and verification. The API phase produces a list of candidate URLs, while the verification phase filters that list based on actual browser rendering. This pipeline ensures that:

- **Fast-path execution**: Running `node scan.mjs` without flags performs only API discovery, writing new offers directly to [`pipeline.md`](https://github.com/santifer/career-ops/blob/main/pipeline.md)
- **Thorough validation**: Running `node scan.mjs --verify` adds the browser confirmation step, ensuring only live, apply-able jobs reach the pipeline

```bash

# 1️⃣ Run a pure-API scan (no browser)

node scan.mjs               # only http requests → new offers go to pipeline.md

# 2️⃣ Run a scan that also verifies each URL with Playwright

node scan.mjs --verify      # fetch via API, then open each URL in headless Chromium

```

## Summary

- **API discovery** is the default, lightweight mode that uses provider modules (`providers/greenhouse.mjs`, `providers/lever.mjs`) to fetch job data via HTTP/JSON without browser overhead.
- **Playwright verification** is opt-in via the `--verify` flag and executes only after API discovery, using headless Chromium to confirm URL liveness and the presence of apply controls.
- **Source locations**: Provider loading occurs in `scan.mjs` lines 55-119, fetching at lines 124-131, and verification logic spans lines 779-885 with helper functions in `liveness-browser.mjs`.
- **Output handling**: Verified active jobs go to [`pipeline.md`](https://github.com/santifer/career-ops/blob/main/pipeline.md), while expired entries are logged to `scan-history.tsv` for deduplication.

## Frequently Asked Questions

### What triggers Playwright verification in the career-ops scanner?

Playwright verification only triggers when you pass the `--verify` flag to the CLI. Without this flag, the scanner runs in API-only mode, performing HTTP requests through provider modules without launching any browser instances.

### Which files handle the HTTP-based API discovery?

API discovery logic resides in `scan.mjs` (lines 55-131) for provider loading and resolution, with individual provider implementations located in `providers/*.mjs` files (such as `greenhouse.mjs` and `lever.mjs`). The shared HTTP utilities live in `providers/_http.mjs`.

### How does the scanner determine if a job URL is still active?

During verification, the scanner uses `checkUrlLiveness()` from `liveness-browser.mjs` to load each URL in a headless Chromium browser via Playwright. It checks for the presence of apply controls and valid page content, categorizing results as active, expired, no-apply-control, or invalid.

### Can I run the scanner without any browser automation?

Yes. Running `node scan.mjs` without the `--verify` flag executes only the API discovery phase. This uses zero browser instances and relies solely on HTTP requests to ATS APIs, making it suitable for automated schedules or environments without Playwright dependencies.