How Liveness Checking Detects and Filters Ghost Jobs in Career-Ops

Career-Ops uses a two-rung liveness detection system that combines zero-token ATS API probes with Playwright browser inspection to identify and discard expired or nonexistent job postings before they enter the evaluation pipeline.

The santifer/career-ops repository implements a robust defense against "ghost" job postings—expired or invalid listings that clutter search results—through a sophisticated liveness checking mechanism. This system validates posting URLs against known Applicant Tracking Systems (ATS) using lightweight HTTP requests before escalating to full browser automation. By filtering these dead links early, the pipeline ensures that downstream evaluation and scanning modes only process actionable opportunities.

The Two-Rung Liveness Detection Architecture

The liveness system operates on a cost-optimization gradient. It first attempts a cheap, deterministic API call; only when that returns ambiguous results does it launch a resource-intensive browser instance. This architecture minimizes cloud compute costs while maximizing accuracy against the diverse landscape of job board implementations.

Rung One: Zero-Token ATS API Probing

The first detection layer lives in liveness-api.mjs and targets known ATS providers including Greenhouse, Lever, Ashby, and Workday. The checkLivenessViaApi function builds fixed-host JSON endpoints for specific job postings and issues GET requests with short timeouts.

The resolution chain flows through resolveAtsApi → classifyAshbyBoard for provider-specific body parsing. Status code interpretation follows strict rules:

• 404/410: The posting is expired (expired result)
• 200: The posting is active (for org-level APIs like Ashby, the response body requires additional parsing to confirm the specific job is listed)
• 429, 5xx, network errors: Returns null, triggering the browser fallback

This approach requires zero authentication tokens and can process hundreds of URLs per second without browser overhead.

Rung Two: Playwright Browser Inspection

When the API layer returns null, check-liveness.mjs escalates to liveness-browser.mjs, launching a headless Chromium instance via Playwright. The checkUrlLivenessWithFallback function orchestrates this process with several defensive layers.

Egress Security Guards

Before any network request executes, rejectPrivateOrInvalid combined with DNS validation via validateUrlSecurity blocks private IP ranges and malformed URLs. This prevents the headless browser from accessing internal network resources.

Content Classification

The classifyLiveness function in liveness-core.mjs examines HTTP status, body text, and visible "apply" controls after navigation. It also polls same-origin child frames (common with iCIMS embeds) to capture delayed JavaScript-rendered content.

Specific heuristics detect ghost jobs:

• Hard-expired banners: Text patterns like "job is no longer available" or French "offre n'est plus disponible" → expired
• Listing-page patterns: Strings like "0 jobs found" indicating the specific posting vanished from the company board → expired
• Anti-bot challenges: Cloudflare "Just a moment…" pages → uncertain (never treated as expired to avoid false positives)
• Job ID loss: Redirects that strip the original job identifier → uncertain
• Insufficient content: Pages with fewer than 300 characters and no apply button → expired

The browser uses BROWSER_LIKE_USER_AGENT from user-agent.mjs to minimize bot challenges during initial requests.

Fallback Handling for Uncertain States

When headless requests encounter anti-bot walls, checkUrlLivenessWithFallback optionally retries using a headed browser instance via createHeadedPageProvider. This headed result supersedes the uncertain headless classification, but the system enforces a strict rule: headed fallback never flips an uncertain state to expired. This prevents aggressive filtering when human-interactive pages behave differently than automated ones.

Integration with the Career-Ops Pipeline

The check-liveness.mjs CLI serves as the entry point for both manual and programmatic validation. It aggregates results across all input URLs, counting active, expired, and uncertain states, then exits with a non-zero status if any posting is expired or uncertain. This ensures that downstream scan or evaluate commands discard ghost jobs before they contaminate the dataset.

node check-liveness.mjs https://jobs.greenhouse.io/company/1234 https://jobs.lever.co/company/slug/5678

For programmatic integration within custom scripts:

import { checkLivenessViaApi } from './liveness-api.mjs';
import { checkUrlLivenessWithFallback } from './liveness-browser.mjs';
import { chromium } from 'playwright';

async function isLive(url) {
  const apiResult = await checkLivenessViaApi(url);
  if (apiResult) return apiResult.result === 'active';
  const browser = await chromium.launch({ headless: true });
  const page = await newLivenessPage(browser);
  const result = await checkUrlLivenessWithFallback(page, url, { getHeadedPage: createHeadedPageProvider(chromium) });
  await browser.close();
  return result.result === 'active';
}

Security-conscious scripts can reuse the egress guards independently:

import { rejectPrivateOrInvalid, validateUrlSecurity } from './liveness-browser.mjs';

async function safeFetch(url) {
  const guard = rejectPrivateOrInvalid(url);
  if (guard) throw new Error(`Blocked URL: ${guard.reason}`);
  await validateUrlSecurity(url);            // DNS-level guard
  return fetch(url);                         // now safe to issue external request
}

Summary

• Two-rung architecture minimizes compute costs by preferring API calls over browser automation
• ATS API probing in liveness-api.mjs handles Greenhouse, Lever, Workday, and Ashby via zero-token endpoints
• Playwright inspection in liveness-browser.mjs validates JavaScript-rendered pages and iCIMS embeds
• Egress guards (rejectPrivateOrInvalid, validateUrlSecurity) block private network access before request execution
• Classification heuristics in liveness-core.mjs distinguish between expired postings, bot challenges, and valid listings
• Headed fallback resolves anti-bot walls without introducing false-positive expirations

Frequently Asked Questions

What constitutes a "ghost" job in Career-Ops?

A ghost job refers to job postings that appear in search results but point to expired, removed, or nonexistent positions. According to the Career-Ops source code, these manifest as hard-expired pages with removal banners, listings showing "0 jobs found," or URLs that return 404/410 status codes from ATS APIs.

How does the system handle rate limiting or API errors?

When liveness-api.mjs encounters rate limits (429) or server errors (5xx), it returns null rather than marking the job expired. This triggers the second-rung Playwright inspection in checkUrlLivenessWithFallback, ensuring transient API failures do not result in false-positive ghost classifications.

Why does liveness checking use both API and browser methods?

The dual approach optimizes for both speed and accuracy. API calls in checkLivenessViaApi are computationally cheap and handle standardized ATS responses instantly, while Playwright in liveness-browser.mjs handles edge cases like single-page applications, iCIMS embeds, and JavaScript-rendered content that APIs cannot see.

What happens when a job posting triggers an anti-bot challenge?

When classifyLiveness detects anti-bot patterns like Cloudflare "Just a moment…" pages, it marks the result as uncertain. The system may then invoke createHeadedPageProvider to retry with a headed browser, but even if that fails, the status remains uncertain rather than flipping to expired, preventing valid but protected listings from being filtered as ghosts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}