# How Liveness Checking Detects and Filters Ghost Jobs in Career-Ops

> Learn how Career-Ops uses liveness checking with API probes and Playwright to detect and filter ghost jobs, ensuring accurate evaluation before processing.

- Repository: [Santiago Fernández de Valderrama/career-ops](https://github.com/santifer/career-ops)
- Tags: how-to-guide
- Published: 2026-08-19

---

**Career-Ops uses a two-rung liveness detection system that combines zero-token ATS API probes with Playwright browser inspection to identify and discard expired or nonexistent job postings before they enter the evaluation pipeline.**

The `santifer/career-ops` repository implements a robust defense against "ghost" job postings—expired or invalid listings that clutter search results—through a sophisticated liveness checking mechanism. This system validates posting URLs against known Applicant Tracking Systems (ATS) using lightweight HTTP requests before escalating to full browser automation. By filtering these dead links early, the pipeline ensures that downstream **evaluation** and **scanning** modes only process actionable opportunities.

## The Two-Rung Liveness Detection Architecture

The liveness system operates on a cost-optimization gradient. It first attempts a cheap, deterministic API call; only when that returns ambiguous results does it launch a resource-intensive browser instance. This architecture minimizes cloud compute costs while maximizing accuracy against the diverse landscape of job board implementations.

## Rung One: Zero-Token ATS API Probing

The first detection layer lives in `liveness-api.mjs` and targets known ATS providers including **Greenhouse**, **Lever**, **Ashby**, and **Workday**. The `checkLivenessViaApi` function builds fixed-host JSON endpoints for specific job postings and issues `GET` requests with short timeouts.

The resolution chain flows through `resolveAtsApi` → `classifyAshbyBoard` for provider-specific body parsing. Status code interpretation follows strict rules:

- **404/410**: The posting is **expired** (`expired` result)
- **200**: The posting is **active** (for org-level APIs like Ashby, the response body requires additional parsing to confirm the specific job is listed)
- **429, 5xx, network errors**: Returns `null`, triggering the browser fallback

This approach requires zero authentication tokens and can process hundreds of URLs per second without browser overhead.

## Rung Two: Playwright Browser Inspection

When the API layer returns `null`, `check-liveness.mjs` escalates to `liveness-browser.mjs`, launching a headless Chromium instance via Playwright. The `checkUrlLivenessWithFallback` function orchestrates this process with several defensive layers.

### Egress Security Guards

Before any network request executes, `rejectPrivateOrInvalid` combined with DNS validation via `validateUrlSecurity` blocks private IP ranges and malformed URLs. This prevents the headless browser from accessing internal network resources.

### Content Classification

The `classifyLiveness` function in `liveness-core.mjs` examines HTTP status, body text, and visible "apply" controls after navigation. It also polls same-origin child frames (common with **iCIMS** embeds) to capture delayed JavaScript-rendered content.

Specific heuristics detect ghost jobs:

- **Hard-expired banners**: Text patterns like "job is no longer available" or French "offre n'est plus disponible" → `expired`
- **Listing-page patterns**: Strings like "0 jobs found" indicating the specific posting vanished from the company board → `expired`
- **Anti-bot challenges**: Cloudflare "Just a moment…" pages → `uncertain` (never treated as expired to avoid false positives)
- **Job ID loss**: Redirects that strip the original job identifier → `uncertain`
- **Insufficient content**: Pages with fewer than 300 characters and no apply button → `expired`

The browser uses `BROWSER_LIKE_USER_AGENT` from `user-agent.mjs` to minimize bot challenges during initial requests.

## Fallback Handling for Uncertain States

When headless requests encounter anti-bot walls, `checkUrlLivenessWithFallback` optionally retries using a headed browser instance via `createHeadedPageProvider`. This headed result supersedes the uncertain headless classification, but the system enforces a strict rule: headed fallback **never** flips an `uncertain` state to `expired`. This prevents aggressive filtering when human-interactive pages behave differently than automated ones.

## Integration with the Career-Ops Pipeline

The `check-liveness.mjs` CLI serves as the entry point for both manual and programmatic validation. It aggregates results across all input URLs, counting active, expired, and uncertain states, then exits with a non-zero status if any posting is expired or uncertain. This ensures that downstream `scan` or `evaluate` commands discard ghost jobs before they contaminate the dataset.

```bash
node check-liveness.mjs https://jobs.greenhouse.io/company/1234 https://jobs.lever.co/company/slug/5678

```

For programmatic integration within custom scripts:

```js
import { checkLivenessViaApi } from './liveness-api.mjs';
import { checkUrlLivenessWithFallback } from './liveness-browser.mjs';
import { chromium } from 'playwright';

async function isLive(url) {
  const apiResult = await checkLivenessViaApi(url);
  if (apiResult) return apiResult.result === 'active';
  const browser = await chromium.launch({ headless: true });
  const page = await newLivenessPage(browser);
  const result = await checkUrlLivenessWithFallback(page, url, { getHeadedPage: createHeadedPageProvider(chromium) });
  await browser.close();
  return result.result === 'active';
}

```

Security-conscious scripts can reuse the egress guards independently:

```js
import { rejectPrivateOrInvalid, validateUrlSecurity } from './liveness-browser.mjs';

async function safeFetch(url) {
  const guard = rejectPrivateOrInvalid(url);
  if (guard) throw new Error(`Blocked URL: ${guard.reason}`);
  await validateUrlSecurity(url);            // DNS-level guard
  return fetch(url);                         // now safe to issue external request
}

```

## Summary

- **Two-rung architecture** minimizes compute costs by preferring API calls over browser automation
- **ATS API probing** in `liveness-api.mjs` handles Greenhouse, Lever, Workday, and Ashby via zero-token endpoints
- **Playwright inspection** in `liveness-browser.mjs` validates JavaScript-rendered pages and iCIMS embeds
- **Egress guards** (`rejectPrivateOrInvalid`, `validateUrlSecurity`) block private network access before request execution
- **Classification heuristics** in `liveness-core.mjs` distinguish between expired postings, bot challenges, and valid listings
- **Headed fallback** resolves anti-bot walls without introducing false-positive expirations

## Frequently Asked Questions

### What constitutes a "ghost" job in Career-Ops?

A ghost job refers to job postings that appear in search results but point to expired, removed, or nonexistent positions. According to the Career-Ops source code, these manifest as hard-expired pages with removal banners, listings showing "0 jobs found," or URLs that return 404/410 status codes from ATS APIs.

### How does the system handle rate limiting or API errors?

When `liveness-api.mjs` encounters rate limits (429) or server errors (5xx), it returns `null` rather than marking the job expired. This triggers the second-rung Playwright inspection in `checkUrlLivenessWithFallback`, ensuring transient API failures do not result in false-positive ghost classifications.

### Why does liveness checking use both API and browser methods?

The dual approach optimizes for both speed and accuracy. API calls in `checkLivenessViaApi` are computationally cheap and handle standardized ATS responses instantly, while Playwright in `liveness-browser.mjs` handles edge cases like single-page applications, iCIMS embeds, and JavaScript-rendered content that APIs cannot see.

### What happens when a job posting triggers an anti-bot challenge?

When `classifyLiveness` detects anti-bot patterns like Cloudflare "Just a moment…" pages, it marks the result as `uncertain`. The system may then invoke `createHeadedPageProvider` to retry with a headed browser, but even if that fails, the status remains `uncertain` rather than flipping to `expired`, preventing valid but protected listings from being filtered as ghosts.