How the Career-Ops Liveness Gate Prevents Evaluation of Dead Job Postings

The Career-Ops liveness gate employs a three-layer validation system—fast ATS API checks, Playwright browser automation, and strict content classification rules—to eliminate dead job postings before they consume processing resources.

The liveness gate in the Career-Ops open-source repository acts as a protective filter between job posting URLs and the evaluation pipeline. By orchestrating lightweight API validation with headless browser inspection and linguistic pattern analysis, the system ensures only active, accessible opportunities proceed to downstream analysis. This multi-stage approach minimizes false positives from bot protection services while definitively identifying expired listings.

Three-Layer Validation Architecture

The gate processes every URL through a cascading sequence defined in check-liveness.mjs. Each layer provides increasing scrutiny, with early exit points to optimize performance.

Layer 1: Fast ATS API Check

The first line of defense calls checkLivenessViaApi(url) (lines 81‑85 of check-liveness.mjs), implemented in liveness-api.mjs. This zero-token API check queries applicant tracking systems for definitive status verdicts. When the API returns active or expired, the process terminates immediately, avoiding any browser overhead.

Layer 2: Playwright Browser Verification

When the API yields inconclusive results, the system launches a headless Chromium instance via newLivenessPage (lines 45‑50 in liveness-browser.mjs). The checkUrlLiveness function (lines 42‑50) navigates to the target URL and executes the classification logic against the rendered DOM.

Layer 3: Content Classification Rules

The classifyLiveness function in liveness-core.mjs (lines 16‑20) applies a strict rule set to the page content. This logic distinguishes between genuinely expired postings and temporary failures or anti-bot challenges.

Classification Rules That Identify Dead Postings

The liveness-core.mjs module implements granular heuristics to prevent false negatives while catching diverse expiration patterns.

HTTP Status Code Evaluation

The classifier first examines response codes:

• 404 or 410 status codes trigger an immediate expired verdict with the http_gone classification (lines 20‑22).
• 403, 429, or 503 codes produce uncertain results (lines 39‑41), as these indicate anti-bot walls or rate limiting rather than removed content.
• 5xx server errors ≥ 500 yield uncertain (lines 43‑48) to prevent filtering temporarily unavailable jobs during maintenance windows.

Expired Content Patterns

Beyond status codes, the system analyzes page content for definitive expiration signals:

• Query parameter flags: URLs containing error=true are flagged as expired (lines 51‑53).
• Hard-expired banners: A comprehensive whitelist of language-specific phrases—including English ("job is no longer available"), French ("offre n'est plus disponible"), and German ("diese stelle ist bereits besetzt")—is normalized and matched against the page text (lines 17‑49). Any match marks the posting expired (lines 56‑58).
• Content-length guard: Pages with fewer than 300 characters of body text are considered expired (lines 86‑88), catching empty or placeholder pages.
• Listing-page detection: Generic search-result pages are classified as expired (lines 81‑84) to prevent evaluating index pages instead of specific job details.

Bot Challenge and Redirect Handling

To avoid misclassifying protected pages as dead:

• Bot-challenge detection: Patterns like "just a moment" or "cloudflare" (lines 28‑30) produce uncertain results, never expired.
• Job-ID mismatch: If the original URL contains a job identifier that disappears after redirects, the result is uncertain (lines 68‑75), preventing false expirations caused by portal migrations.

Active Posting Confirmation

The system requires positive evidence of liveness:

• Apply-control detection: Visible application buttons labeled "Apply", "Postuler", "Aplikuj", or similar variants (extracted in lines 78‑94) trigger an active verdict (line 77).
• Fallback logic: Any page passing all checks but lacking apply controls is labeled uncertain (line 90), ensuring conservative evaluation.

Running the Liveness Gate

Command-Line Interface

Execute standalone URL checks using the CLI entry point:

node check-liveness.mjs https://example.com/job/123

The output displays a concise verdict:

• ✅ active for live postings
• ❌ expired for dead postings

Programmatic Integration

Import the browser module to embed liveness checks in custom scripts:

import { checkUrlLivenessWithFallback } from './liveness-browser.mjs';
import { chromium } from 'playwright';

async function isLive(url) {
  const browser = await chromium.launch({ headless: true });
  const page = await import('./liveness-browser.mjs')
    .then(m => m.newLivenessPage(browser));
  const result = await checkUrlLivenessWithFallback(page, url, { 
    getHeadedPage: null 
  });
  await browser.close();
  return result.result === 'active';
}

Scanner Integration

When using the main scanner with the --verify flag, scan.mjs invokes checkUrlLiveness via check-liveness.mjs to filter dead postings before they enter the evaluation pipeline:

node scan.mjs --verify https://jobs.example.com/position/abc

Summary

• The liveness gate implements a three-stage pipeline: API check → browser verification → content classification.
• check-liveness.mjs orchestrates the validation flow, calling checkLivenessViaApi via liveness-api.mjs before falling back to Playwright.
• liveness-core.mjs contains the classifyLiveness rules that distinguish expired postings from bot-protected or temporarily unavailable pages.
• Strict HTTP status handling treats 404/410 as expired while marking 403, 429, 503, and 5xx errors as uncertain to avoid false positives.
• Linguistic pattern matching detects language-specific expiration phrases across English, French, German, and other languages.
• The apply-control detection requires visible application buttons to confirm active status, ensuring only genuine opportunities proceed.

Frequently Asked Questions

How does Career-Ops handle job postings protected by Cloudflare or bot detection?

The classifyLiveness function in liveness-core.mjs detects bot-challenge patterns like "just a moment" or "cloudflare" (lines 28‑30) and returns an uncertain status rather than expired. This prevents the system from incorrectly filtering live postings that are merely protected by anti-bot measures. These URLs can be retried later or flagged for manual review.

What happens when a job posting returns a 500 error?

HTTP 5xx status codes ≥ 500 are treated as uncertain (lines 43‑48 of liveness-core.mjs) because they typically indicate temporary server issues rather than permanent removal. This conservative approach prevents the liveness gate from discarding valid opportunities during temporary outages. The system assumes the posting may still be live until a definitive expiration signal appears.

Can the liveness gate detect expired postings in non-English languages?

Yes. The classifyLiveness function maintains a large whitelist of language-specific expiration phrases covering English ("job is no longer available"), French ("offre n'est plus disponible"), German ("diese stelle ist bereits besetzt"), and others (lines 17‑49). These patterns are normalized and matched against page content, enabling accurate detection across international job boards.

Why does the liveness gate require an apply button to confirm active status?

The apply-control detection logic (lines 78‑94 of liveness-core.mjs) searches for visible application elements like "Apply", "Postuler", or "Aplikuj" to confirm the posting accepts candidates. This positive confirmation prevents the system from marking generic content pages or listing indexes as active. If no apply control is found after passing other checks, the posting receives an uncertain classification (line 90), ensuring only actionable opportunities enter the pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}