How `check-table-freshness.mjs` Validates Staleness of Jurisdiction Data Tables in santifer/career-ops

check-table-freshness.mjs validates jurisdiction data freshness by scanning YAML tables for as_of verification dates and next_effective change dates, flagging rows as expired when legal changes have arrived but tables weren't updated, and marking rows as review-due when their verification date exceeds a configurable age threshold (default 12 months).

The check-table-freshness.mjs script in the santifer/career-ops repository is a pure-Node utility designed to prevent stale legal and wage data from propagating through job-search pipelines. It operates on jurisdiction-specific YAML tables stored in the templates/ directory, applying two distinct freshness rules to identify outdated records. This article explains the complete validation mechanism, from file discovery to exit code handling, based on the actual implementation in the repository.

Two-Rule Freshness Validation

The core validation logic in checkFreshness (lines 82-88) applies two complementary checks to every row containing temporal metadata.

Expired Rows: Detecting Missed Legal Changes

The expired rows rule targets a specific failure mode: when a jurisdiction has pre-announced a legal change via next_effective, but the table's as_of verification date predates that change.

A row is flagged as expired when:

• next_effective exists and today ≥ next_effective
• The row's as_of date is earlier than next_effective

This condition signals that the announced change has taken effect, yet the table contains no post-change verification. The implementation appears in the core loop at lines 31-34:

// Simplified representation of the expiration check logic
// Lines 31-34 in check-table-freshness.mjs
if (row.next_effective && today >= parseDate(row.next_effective)) {
  if (parseDate(row.as_of) < parseDate(row.next_effective)) {
    findings.push({ type: 'expired', ...rowLocation });
  }
}

This check is critical for compliance-sensitive data like minimum wage rates, overtime thresholds, or tax brackets where jurisdictions publish effective dates in advance.

Review-Due Rows: Enforcing Periodic Re-verification

The review-due rows rule ensures active data doesn't stagnate. A row is flagged when its as_of verification date exceeds a configurable maximum age.

The threshold calculation uses addMonthsUTC (lines 14-22) to compute a cutoff date by subtracting the threshold from today's UTC date:

// Date manipulation utility at lines 14-22
function addMonthsUTC(date, months) {
  const result = new Date(Date.UTC(date.getUTCFullYear(), date.getUTCMonth() + months, date.getUTCDate()));
  return result;
}

const cutoffDate = addMonthsUTC(new Date(), -maxAgeMonths); // Default: -12

The review-due check applies this cutoff in the loop at lines 42-46:

// Lines 42-46: Age-based freshness check
if (parseDate(row.as_of) < cutoffDate) {
  findings.push({ type: 'review-due', ...rowLocation });
}

Three-Phase Processing Pipeline

The script executes validation through a structured pipeline defined in loadTables, extractRows, and checkFreshness.

Phase 1: Table Discovery

loadTables (lines 58-78) recursively traverses templates/*.yml, parsing each file and filtering for jurisdiction tables—defined as files containing at least one row with an as_of field. Non-conforming files are silently skipped.

Phase 2: Row Extraction

extractRows (lines 25-76) normalizes two YAML shapes found in the wild:

• Top-level arrays: [{as_of: '2024-01-01', ...}, ...]
• Key-mapped objects: {CA: {as_of: '2024-01-01', ...}, NY: {...}}

Each extracted row carries metadata: container name, array index or object key, and a flag for missing as_of (which generates warnings but not findings).

Phase 3: Freshness Assessment

checkFreshness orchestrates the two-rule validation, producing a structured result object:

// Result structure at lines 82-88
{
  scannedTables: 15,
  checkedRows: 247,
  findings: [
    { type: 'expired', file: 'templates/wage/ca.yml', path: 'rows[3]' },
    { type: 'review-due', file: 'templates/tax/ny.yml', path: 'rows[0]' }
  ],
  warnings: [
    { file: 'templates/benefits/tx.yml', reason: 'missing_as_of', path: 'rows[5]' }
  ]
}

Configuration and CLI Options

The script supports flexible threshold configuration through multiple channels, resolved in loadConfigMaxAge (lines 87-106).

Configuration Precedence (Highest to Lowest)

1. CLI flag: --max-age-months 6
2. Profile config: config/profile.yml → table_freshness.max_age_months
3. Default: 12 months

# Override review threshold for quarterly compliance audits

node check-table-freshness.mjs --max-age-months 3

# Generate human-readable summary instead of JSON

node check-table-freshness.mjs --summary

Quality Assurance Features

Built-In Self-Test

Running with --self-test executes runSelfTest (starting at line 55), an in-file fixture suite that verifies:

• Row extraction from both YAML shapes
• parseDate handling of ISO and legacy formats
• Correct classification of expired vs. review-due findings

# CI pipeline integration

node check-table-freshness.mjs --self-test || exit 1

Exit Code Semantics

The script provides CI-friendly exit codes (lines 13-14):

Exit Code	Condition
1	One or more expired findings present
0	No expired findings (review-due alone doesn't fail)

This distinction allows pipelines to hard-fail on compliance violations while tolerating routine maintenance reminders.

Date Parsing Precision

parseDate (lines 4-10) normalizes input formats to UTC midnight, ensuring deterministic comparison regardless of execution timezone:

// Lines 4-10: Date parsing implementation
function parseDate(input) {
  // Handles '2024-01-15', '2024-01-15T00:00:00Z', '01/15/2024'
  const parsed = new Date(input);
  return Date.UTC(parsed.getUTCFullYear(), parsed.getUTCMonth(), parsed.getUTCDate());
}

All threshold calculations maintain UTC alignment through addMonthsUTC, preventing daylight saving time anomalies from skewing month-boundary comparisons.

Summary

• Two-rule validation: check-table-freshness.mjs detects expired rows (missed next_effective changes) and review-due rows (as_of exceeds age threshold)
• Configurable threshold: Default 12-month maximum age, overridable via --max-age-months or config/profile.yml
• Three-phase pipeline: Discovery (loadTables, lines 58-78) → Extraction (extractRows, lines 25-76) → Assessment (checkFreshness, lines 82-88)
• CI integration: Exit code 1 on expired findings; --self-test mode for unit verification; --summary for human review
• Timezone safety: All date operations use UTC-aligned functions parseDate and addMonthsUTC

Frequently Asked Questions

What triggers an "expired" finding versus a "review-due" finding?

An expired finding occurs when a row has a next_effective date that has passed, but the as_of verification date predates it—indicating the legal change arrived without table update. A review-due finding occurs when the as_of date alone exceeds the maximum age threshold, regardless of whether next_effective is present. Expired findings fail CI; review-due findings do not.

How does the script handle different YAML table structures?

extractRows (lines 25-76) automatically detects both top-level arrays and keyed objects, flattening each into a uniform row stream with path metadata. This allows the same validation logic to process files like wages.yml (array of states) and tax-brackets.yml (keyed by jurisdiction code) without configuration changes.

What's the difference between findings and warnings in the output?

Findings (expired, review-due) represent actionable staleness conditions on valid rows. Warnings indicate data quality issues that prevent assessment: missing as_of fields, unparseable dates, or malformed YAML structures. Warnings don't affect exit codes but appear in JSON output for remediation tracking.

Why does the default threshold use 12 months rather than a shorter period?

The 12-month default balances practical maintenance burden against legal data volatility—most jurisdictions don't update wage or tax parameters more than annually. Operations requiring tighter compliance (e.g., quarterly payroll systems) can override via --max-age-months 3 without modifying source.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}