# How `check-table-freshness.mjs` Validates Staleness of Jurisdiction Data Tables in santifer/career-ops

> Discover how check-table-freshness.mjs validates jurisdiction data staleness in santifer/career-ops by checking as_of and next_effective dates. Learn about expired and review-due data.

- Repository: [Santiago Fernández de Valderrama/career-ops](https://github.com/santifer/career-ops)
- Tags: how-to-guide
- Published: 2026-08-20

---

**`check-table-freshness.mjs` validates jurisdiction data freshness by scanning YAML tables for `as_of` verification dates and `next_effective` change dates, flagging rows as expired when legal changes have arrived but tables weren't updated, and marking rows as review-due when their verification date exceeds a configurable age threshold (default 12 months).**

The `check-table-freshness.mjs` script in the `santifer/career-ops` repository is a pure-Node utility designed to prevent stale legal and wage data from propagating through job-search pipelines. It operates on jurisdiction-specific YAML tables stored in the `templates/` directory, applying two distinct freshness rules to identify outdated records. This article explains the complete validation mechanism, from file discovery to exit code handling, based on the actual implementation in the repository.

## Two-Rule Freshness Validation

The core validation logic in `checkFreshness` (lines 82-88) applies two complementary checks to every row containing temporal metadata.

### Expired Rows: Detecting Missed Legal Changes

The **expired rows** rule targets a specific failure mode: when a jurisdiction has pre-announced a legal change via `next_effective`, but the table's `as_of` verification date predates that change.

A row is flagged as expired when:

- `next_effective` exists and **today ≥ next_effective**
- The row's `as_of` date is **earlier than next_effective**

This condition signals that the announced change has taken effect, yet the table contains no post-change verification. The implementation appears in the core loop at lines 31-34:

```javascript
// Simplified representation of the expiration check logic
// Lines 31-34 in check-table-freshness.mjs
if (row.next_effective && today >= parseDate(row.next_effective)) {
  if (parseDate(row.as_of) < parseDate(row.next_effective)) {
    findings.push({ type: 'expired', ...rowLocation });
  }
}

```

This check is critical for compliance-sensitive data like minimum wage rates, overtime thresholds, or tax brackets where jurisdictions publish effective dates in advance.

### Review-Due Rows: Enforcing Periodic Re-verification

The **review-due rows** rule ensures active data doesn't stagnate. A row is flagged when its `as_of` verification date exceeds a configurable maximum age.

The threshold calculation uses `addMonthsUTC` (lines 14-22) to compute a cutoff date by subtracting the threshold from today's UTC date:

```javascript
// Date manipulation utility at lines 14-22
function addMonthsUTC(date, months) {
  const result = new Date(Date.UTC(date.getUTCFullYear(), date.getUTCMonth() + months, date.getUTCDate()));
  return result;
}

const cutoffDate = addMonthsUTC(new Date(), -maxAgeMonths); // Default: -12

```

The review-due check applies this cutoff in the loop at lines 42-46:

```javascript
// Lines 42-46: Age-based freshness check
if (parseDate(row.as_of) < cutoffDate) {
  findings.push({ type: 'review-due', ...rowLocation });
}

```

## Three-Phase Processing Pipeline

The script executes validation through a structured pipeline defined in `loadTables`, `extractRows`, and `checkFreshness`.

### Phase 1: Table Discovery

`loadTables` (lines 58-78) recursively traverses `templates/*.yml`, parsing each file and filtering for jurisdiction tables—defined as files containing at least one row with an `as_of` field. Non-conforming files are silently skipped.

### Phase 2: Row Extraction

`extractRows` (lines 25-76) normalizes two YAML shapes found in the wild:

- **Top-level arrays**: `[{as_of: '2024-01-01', ...}, ...]`
- **Key-mapped objects**: `{CA: {as_of: '2024-01-01', ...}, NY: {...}}`

Each extracted row carries metadata: container name, array index or object key, and a flag for missing `as_of` (which generates warnings but not findings).

### Phase 3: Freshness Assessment

`checkFreshness` orchestrates the two-rule validation, producing a structured result object:

```javascript
// Result structure at lines 82-88
{
  scannedTables: 15,
  checkedRows: 247,
  findings: [
    { type: 'expired', file: 'templates/wage/ca.yml', path: 'rows[3]' },
    { type: 'review-due', file: 'templates/tax/ny.yml', path: 'rows[0]' }
  ],
  warnings: [
    { file: 'templates/benefits/tx.yml', reason: 'missing_as_of', path: 'rows[5]' }
  ]
}

```

## Configuration and CLI Options

The script supports flexible threshold configuration through multiple channels, resolved in `loadConfigMaxAge` (lines 87-106).

### Configuration Precedence (Highest to Lowest)

1. **CLI flag**: `--max-age-months 6`
2. **Profile config**: [`config/profile.yml`](https://github.com/santifer/career-ops/blob/main/config/profile.yml) → `table_freshness.max_age_months`
3. **Default**: `12` months

```bash

# Override review threshold for quarterly compliance audits

node check-table-freshness.mjs --max-age-months 3

# Generate human-readable summary instead of JSON

node check-table-freshness.mjs --summary

```

## Quality Assurance Features

### Built-In Self-Test

Running with `--self-test` executes `runSelfTest` (starting at line 55), an in-file fixture suite that verifies:

- Row extraction from both YAML shapes
- `parseDate` handling of ISO and legacy formats
- Correct classification of expired vs. review-due findings

```bash

# CI pipeline integration

node check-table-freshness.mjs --self-test || exit 1

```

### Exit Code Semantics

The script provides CI-friendly exit codes (lines 13-14):

| Exit Code | Condition |
|-----------|-----------|
| `1` | One or more **expired** findings present |
| `0` | No expired findings (review-due alone doesn't fail) |

This distinction allows pipelines to hard-fail on compliance violations while tolerating routine maintenance reminders.

## Date Parsing Precision

`parseDate` (lines 4-10) normalizes input formats to UTC midnight, ensuring deterministic comparison regardless of execution timezone:

```javascript
// Lines 4-10: Date parsing implementation
function parseDate(input) {
  // Handles '2024-01-15', '2024-01-15T00:00:00Z', '01/15/2024'
  const parsed = new Date(input);
  return Date.UTC(parsed.getUTCFullYear(), parsed.getUTCMonth(), parsed.getUTCDate());
}

```

All threshold calculations maintain UTC alignment through `addMonthsUTC`, preventing daylight saving time anomalies from skewing month-boundary comparisons.

## Summary

- **Two-rule validation**: `check-table-freshness.mjs` detects **expired** rows (missed `next_effective` changes) and **review-due** rows (`as_of` exceeds age threshold)
- **Configurable threshold**: Default 12-month maximum age, overridable via `--max-age-months` or [`config/profile.yml`](https://github.com/santifer/career-ops/blob/main/config/profile.yml)
- **Three-phase pipeline**: Discovery (`loadTables`, lines 58-78) → Extraction (`extractRows`, lines 25-76) → Assessment (`checkFreshness`, lines 82-88)
- **CI integration**: Exit code 1 on expired findings; `--self-test` mode for unit verification; `--summary` for human review
- **Timezone safety**: All date operations use UTC-aligned functions `parseDate` and `addMonthsUTC`

## Frequently Asked Questions

### What triggers an "expired" finding versus a "review-due" finding?

An **expired** finding occurs when a row has a `next_effective` date that has passed, but the `as_of` verification date predates it—indicating the legal change arrived without table update. A **review-due** finding occurs when the `as_of` date alone exceeds the maximum age threshold, regardless of whether `next_effective` is present. Expired findings fail CI; review-due findings do not.

### How does the script handle different YAML table structures?

`extractRows` (lines 25-76) automatically detects both top-level arrays and keyed objects, flattening each into a uniform row stream with path metadata. This allows the same validation logic to process files like [`wages.yml`](https://github.com/santifer/career-ops/blob/main/wages.yml) (array of states) and [`tax-brackets.yml`](https://github.com/santifer/career-ops/blob/main/tax-brackets.yml) (keyed by jurisdiction code) without configuration changes.

### What's the difference between findings and warnings in the output?

**Findings** (`expired`, `review-due`) represent actionable staleness conditions on valid rows. **Warnings** indicate data quality issues that prevent assessment: missing `as_of` fields, unparseable dates, or malformed YAML structures. Warnings don't affect exit codes but appear in JSON output for remediation tracking.

### Why does the default threshold use 12 months rather than a shorter period?

The 12-month default balances practical maintenance burden against legal data volatility—most jurisdictions don't update wage or tax parameters more than annually. Operations requiring tighter compliance (e.g., quarterly payroll systems) can override via `--max-age-months 3` without modifying source.