How the Career Ops Pipeline Integrity System Ensures Data Consistency: Merge, Dedup, and Normalize Statuses

The Career Ops pipeline integrity system guarantees data consistency by acquiring an exclusive file-system lock during batch merges, detecting duplicates through exact and fuzzy matching while preserving the highest score, and rewriting non-canonical statuses into a strict set of eight canonical states defined in templates/states.yml.

The santifer/career-ops repository manages every evaluated job offer inside a central tracker file, applications.md. Batch jobs write new rows to batch/tracker-additions/*.tsv, which the pipeline integrity system then folds into the main tracker through a sequential merge-dedup-normalize workflow. This design ensures that downstream scripts for PDF generation, reporting, and follow-up cadence always consume a deduplicated, ordered, and semantically clean dataset.

Merge and Deduplicate: How merge-tracker.mjs Enforces Consistency

Exclusive Lock Acquisition and Atomic Writes

The merge script begins by calling acquireTrackerLock(), which creates a temporary lock directory, writes an owner.json file containing the current PID, and implements retry logic with back-off to prevent concurrent writes. If a stale lock is detected, the routine recovers automatically before returning a release() function. This critical section, found in merge-tracker.mjs at lines 19–34, ensures that no two batch jobs can corrupt applications.md by writing simultaneously.

Once changes are computed, the script commits them via writeFileAtomic(). The helper writes to a random-named temporary file in the same directory and calls renameSync to replace the original tracker, guaranteeing an all-or-nothing update that downstream readers never observe mid-write.

Parser Flexibility and Report-Link Normalization

The script loads the existing tracker with readFileSync(APPS_FILE, 'utf-8') and iterates over every addition file in batch/tracker-additions/. Each candidate row is parsed by parseTsvContent() (lines 65–104), which heuristically detects nine-column TSV, eight-column TSV, or pipe-delimited markdown rows regardless of column order variations.

Before deduplication runs, normalizeReportLink() (lines 31–35) delegates to tracker-links.mjs to rewrite any reports/... paths relative to the tracker directory. This compensates for repository layout differences between the data/ and root-level tracker placements and prevents broken report references.

A typical batch addition looks like this:

42	2026-06-12	Acme Corp	Senior Backend Engineer	4.5/5	Applied	✅	reports/042-acme-corp-2026-06-12.md	First evaluation

You can run the merge with:

node career-ops/merge-tracker.mjs       # normal run

node career-ops/merge-tracker.mjs --dry-run   # preview only

Three-Layer Duplicate Detection

Duplicate detection combines three strategies implemented near lines 36–41:

1. extractReportNum() identifies exact report-number collisions.
2. An existing row is flagged if both the row number and the normalized company name match the incoming entry.
3. roleFuzzyMatch() tokenizes the role title, removes stop words, excludes baseline tokens, and computes a Jaccard similarity ratio; a value of at least 0.6 signals a duplicate.

Because the same job re-posted under slightly different titles will still exceed the Jaccard threshold, this fuzzy layer catches semantic duplicates that exact string comparison would miss.

Score-Aware Update Logic

When a duplicate is found, the merge script compares numerical scores using parseScore(). If the incoming score is strictly higher than the stored score, the existing row is overwritten with the newer data; otherwise the candidate is skipped and a counter is incremented.

This rule ensures that manual evaluations or later batch imports never downgrade a previously recorded assessment. New rows receive a sequentially computed entryNum and are spliced into the markdown table immediately after the header, preserving the tracker’s sort order.

Canonical Status Normalization with normalize-statuses.mjs

After merging, the normalize-statuses.mjs script sanitizes every status cell in applications.md to guarantee uniform semantics. It reads the tracker, scans each row, and rewrites non-canonical values into one of the eight canonical states defined in templates/states.yml.

Mapping Aliases and Stripping Markup

The normalizeStatus() function, located at lines 28–84, first strips markdown bold markers, lower-cases the raw string, and then matches it against a hard-coded list of English canonical statuses and Spanish aliases. This collapses variants such as **Applied**, aplicado, or Aplicado 2024-01-01 into the single canonical value Applied.

Special Cases and Column Migration

Special handling at lines 26–35 detects DUPLICADO or REPOST annotations and moves them into the notes column rather than treating them as statuses. Dates embedded inside status strings are removed, while em-dash or empty cells are normalized to Discarded. These rules prevent free-text artifacts from leaking into the state machine that downstream tooling relies on.

Backup and In-Place Rewrite

Before overwriting, the script creates a .bak copy via copyFileSync and then writes the cleaned tracker back to applications.md. The line reconstruction logic at lines 41–44 rebuilds each markdown table row with canonical statuses and cleaned score fields, ensuring the file remains valid markdown.

Run the normalizer with:

node career-ops/normalize-statuses.mjs          # clean the tracker in-place

node career-ops/normalize-statuses.mjs --dry-run   # show changes without writing

Typical console output looks like:

#42: "aplicado 2024-01-01" → "Applied"
📊 7 statuses normalized
✅ Written to applications.md (backup: applications.md.bak)

End-to-End Pipeline Integrity Guarantees

Running merge-tracker.mjs and normalize-statuses.mjs together closes the loop on several data-consistency risks addressed by the pipeline integrity system. The exclusive lock plus atomic rename eliminates lost updates from concurrent batch jobs.

Multi-layer deduplication—exact report number, row-plus-company matching, and fuzzy role comparison—prevents redundant entries without blocking legitimate reposts. Score-aware updates protect historical evaluations from accidental downgrade.

Finally, status canonicalization removes language drift, markup noise, and misplaced metadata so that every downstream consumer reads the same eight canonical states.

Summary

• Exclusive locking: acquireTrackerLock() in merge-tracker.mjs prevents concurrent writes by creating a PID-backed mutex with stale-lock recovery.
• Atomic writes: writeFileAtomic() commits changes via a temporary file and renameSync, ensuring readers never see a partially updated tracker.
• Three-layer deduplication: Exact report number, normalized company plus row number, and Jaccard-based fuzzy role matching catch duplicates while allowing distinct reposts.
• Score-preserving updates: Incoming rows overwrite existing ones only when parseScore() returns a strictly higher value, preventing downgrade.
• Status canonicalization: normalize-statuses.mjs maps aliases, Spanish variants, and markdown artifacts to the eight canonical states defined in templates/states.yml.
• Auditability: Both scripts emit detailed console output and support --dry-run for safe previews.

Frequently Asked Questions

How does merge-tracker.mjs prevent two batch jobs from corrupting applications.md?

Before reading or writing, the script calls acquireTrackerLock() (lines 19–34), which creates a lock directory and records the owner PID. If another process holds the lock, the script retries with back-off and can recover stale locks. The exclusive critical section, paired with an atomic renameSync write, prevents simultaneous modifications.

What happens when a merged row has the same report number but a lower score than the existing entry?

The merge script uses parseScore() to compare the incoming score against the existing one. It overwrites the row only when the new score is strictly higher; otherwise the candidate is skipped. This rule prevents any batch import from downgrading a previously recorded evaluation.

Which non-canonical status values does normalize-statuses.mjs handle?

normalize-statuses.mjs strips markdown bold, lower-cases strings, and maps both English and Spanish aliases to one of eight canonical states defined in templates/states.yml. It also moves DUPLICADO or REPOST annotations into the notes column, removes dates from statuses, and treats empty cells as Discarded.

Can I preview merge or normalization changes without modifying applications.md?

Yes. Both scripts accept a --dry-run flag that previews additions, updates, or status rewrites in the console without executing the atomic write or creating a .bak file. This allows safe inspection before any permanent change to applications.md.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}