How to Debug Pipeline Issues Using verify-pipeline.mjs Health Checks

Run node verify-pipeline.mjs to validate the Career-Ops pipeline integrity, detecting canonical status violations, duplicate entries, broken report links, and stale reservation files before they corrupt downstream operations.

The verify-pipeline.mjs script in the santifer/career-ops repository serves as the central health-check utility that validates the integrity of the Career-Ops pipeline before you run any evaluation or batch operation. By inspecting the tracker markdown, TSV additions directory, report links, and status values, this script provides a concise checklist of errors and warnings that pinpoint exactly which part of the pipeline is broken.

What verify-pipeline.mjs Validates

The script performs eight distinct validation checks across the Career-Ops data pipeline. Each check targets a specific structural requirement to ensure consistency between the tracker file and supporting resources.

Canonical Status Values

All status cells must contain one of eight canonical values: evaluated, applied, responded, interview, offer, rejected, discarded, or skip. The script also rejects aliases that don't map to these values, as well as bold markdown or dates inside status cells. This validation appears in verify-pipeline.mjs:39-53 and references the templates/states.yml file for the authoritative status list.

Duplicate Entry Detection

The script detects multiple rows sharing the same company and role combination, using case-insensitive comparison with symbols stripped. This prevents data corruption in downstream analytics. The duplicate detection logic resides in verify-pipeline.mjs:13-28.

Report Link Integrity

Every markdown link in the report column must point to an existing file. The validator tries both the tracker-relative path and the repository root to accommodate legacy link formats. This check is implemented in verify-pipeline.mjs:30-45.

Score Format Validation

Scores must match the pattern X.XX/5, N/A, or DUP. The script separately flags bold markup (**) inside score cells as warnings. See the validation logic in verify-pipeline.mjs:48-56 and verify-pipeline.mjs:82-90.

Row Structure Verification

Every pipeline row must contain at least nine pipe-delimited columns, conforming to the TSV format used by merge-tracker.mjs. Malformed rows trigger immediate errors at verify-pipeline.mjs:59-68.

Pending TSV Detection

The batch/tracker-additions/ directory must be empty before running pipeline operations. Any leftover TSV files indicate that merge-tracker.mjs has not been run to integrate pending changes. This check appears in verify-pipeline.mjs:71-79.

Stale Reservation Cleanup

The script automatically removes *-RESERVED.md sentinel files older than 4 hours that were left behind by crashed reserve-report-num.mjs runs. This prevents reservation conflicts when generating new report numbers. The cleanup logic is located at verify-pipeline.mjs:93-108.

How to Run the Health Check

Execute the validator from the repository root to receive a color-coded status report:

node verify-pipeline.mjs

The script outputs green ✅ for passing checks, yellow ⚠️ for warnings, and red ❌ for errors, followed by a summary line:

• 🟢 Pipeline is clean! – All validations passed
• 🟡 Pipeline OK with warnings – Non-critical issues detected
• 🔴 Pipeline has errors – Critical failures requiring immediate attention

The exit code is 1 when errors are present, enabling CI pipelines to abort automatically:

if ! node verify-pipeline.mjs; then
  echo "❗ Pipeline validation failed – fix the above errors before proceeding."
  exit 1
fi

Common Pipeline Issues and Fixes

Fixing Non-Canonical Status Values

When the script reports ❌ #12: Non-canonical status "Evaluada", open applications.md (or data/applications.md) and locate the offending row. Replace the value with the canonical equivalent evaluated, or consult the ALIASES map in the source code for acceptable alternatives. Re-run node verify-pipeline.mjs to confirm the error clears.

Processing Pending TSV Files

If the health check flags pending TSVs in batch/tracker-additions/, merge them using the official pipeline tool:

# List pending files

ls batch/tracker-additions/*.tsv

# Merge into the tracker

node merge-tracker.mjs

# Verify clearance

node verify-pipeline.mjs

Manual Reservation Cleanup

While the script auto-removes stale *-RESERVED.md files after 4 hours, you can force immediate cleanup:

find reports/ -name '*-RESERVED.md' -type f -mtime +4h -delete
node verify-pipeline.mjs

Summary

• verify-pipeline.mjs validates eight critical pipeline components including canonical statuses, duplicate detection, report links, and pending TSVs.
• The script references templates/states.yml for authoritative status definitions and inspects applications.md, batch/tracker-additions/, and reports/ directories.
• Exit code 1 signals errors, enabling integration with CI/CD workflows and pre-commit hooks.
• Automatic cleanup of stale reservation sentinels prevents conflicts with reserve-report-num.mjs.
• Running this health check before merge-tracker.mjs or generate-pdf.mjs prevents downstream crashes and data corruption.

Frequently Asked Questions

What exit code does verify-pipeline.mjs return on failure?

The script exits with status 1 when errors are detected, and 0 when the pipeline is clean or only contains warnings. This allows shell scripts and CI pipelines to halt execution automatically when structural issues are found.

How does the script handle legacy report links?

The validator attempts to resolve report links using both tracker-relative paths and repository root paths. This dual-resolution strategy accommodates legacy link formats while enforcing that all referenced files must exist in the reports/ directory.

Can I run verify-pipeline.mjs before merge-tracker.mjs?

Yes, and you should. The script specifically checks that batch/tracker-additions/ is empty, which means it detects when merge-tracker.mjs needs to run. If pending TSVs exist, complete the merge process before proceeding with PDF generation or other batch operations.

Where does the script get the list of valid status values?

The canonical status list is derived from templates/states.yml, which defines the eight allowed values: evaluated, applied, responded, interview, offer, rejected, discarded, and skip. The script also maintains an internal ALIASES map to handle common variations while rejecting bold markdown or dates embedded in status cells.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}