# How the Career Ops Pipeline Integrity System Ensures Data Consistency: Merge, Dedup, and Normalize Statuses > Learn how the Career Ops pipeline integrity system ensures data consistency with merge, dedup, and normalize statuses. Discover its robust data management techniques. - Repository: [Santiago Fernández de Valderrama/career-ops](https://github.com/santifer/career-ops) - Tags: internals - Published: 2026-06-12 --- **The Career Ops pipeline integrity system guarantees data consistency by acquiring an exclusive file-system lock during batch merges, detecting duplicates through exact and fuzzy matching while preserving the highest score, and rewriting non-canonical statuses into a strict set of eight canonical states defined in [`templates/states.yml`](https://github.com/santifer/career-ops/blob/main/templates/states.yml).** The `santifer/career-ops` repository manages every evaluated job offer inside a central tracker file, [`applications.md`](https://github.com/santifer/career-ops/blob/main/applications.md). Batch jobs write new rows to `batch/tracker-additions/*.tsv`, which the pipeline integrity system then folds into the main tracker through a sequential merge-dedup-normalize workflow. This design ensures that downstream scripts for PDF generation, reporting, and follow-up cadence always consume a deduplicated, ordered, and semantically clean dataset. ## Merge and Deduplicate: How `merge-tracker.mjs` Enforces Consistency ### Exclusive Lock Acquisition and Atomic Writes The merge script begins by calling `acquireTrackerLock()`, which creates a temporary lock directory, writes an [`owner.json`](https://github.com/santifer/career-ops/blob/main/owner.json) file containing the current PID, and implements retry logic with back-off to prevent concurrent writes. If a stale lock is detected, the routine recovers automatically before returning a `release()` function. This critical section, found in `merge-tracker.mjs` at lines 19–34, ensures that no two batch jobs can corrupt [`applications.md`](https://github.com/santifer/career-ops/blob/main/applications.md) by writing simultaneously. Once changes are computed, the script commits them via `writeFileAtomic()`. The helper writes to a random-named temporary file in the same directory and calls `renameSync` to replace the original tracker, guaranteeing an all-or-nothing update that downstream readers never observe mid-write. ### Parser Flexibility and Report-Link Normalization The script loads the existing tracker with `readFileSync(APPS_FILE, 'utf-8')` and iterates over every addition file in `batch/tracker-additions/`. Each candidate row is parsed by `parseTsvContent()` (lines 65–104), which heuristically detects nine-column TSV, eight-column TSV, or pipe-delimited markdown rows regardless of column order variations. Before deduplication runs, `normalizeReportLink()` (lines 31–35) delegates to `tracker-links.mjs` to rewrite any `reports/...` paths relative to the tracker directory. This compensates for repository layout differences between the `data/` and root-level tracker placements and prevents broken report references. A typical batch addition looks like this: ```tsv 42 2026-06-12 Acme Corp Senior Backend Engineer 4.5/5 Applied ✅ reports/042-acme-corp-2026-06-12.md First evaluation ``` You can run the merge with: ```bash node career-ops/merge-tracker.mjs # normal run node career-ops/merge-tracker.mjs --dry-run # preview only ``` ### Three-Layer Duplicate Detection Duplicate detection combines three strategies implemented near lines 36–41: 1. `extractReportNum()` identifies exact report-number collisions. 2. An existing row is flagged if both the row number and the normalized company name match the incoming entry. 3. `roleFuzzyMatch()` tokenizes the role title, removes stop words, excludes baseline tokens, and computes a Jaccard similarity ratio; a value of at least 0.6 signals a duplicate. Because the same job re-posted under slightly different titles will still exceed the Jaccard threshold, this fuzzy layer catches semantic duplicates that exact string comparison would miss. ### Score-Aware Update Logic When a duplicate is found, the merge script compares numerical scores using `parseScore()`. If the incoming score is strictly higher than the stored score, the existing row is overwritten with the newer data; otherwise the candidate is skipped and a counter is incremented. This rule ensures that manual evaluations or later batch imports never downgrade a previously recorded assessment. New rows receive a sequentially computed `entryNum` and are spliced into the markdown table immediately after the header, preserving the tracker’s sort order. ## Canonical Status Normalization with `normalize-statuses.mjs` After merging, the `normalize-statuses.mjs` script sanitizes every status cell in [`applications.md`](https://github.com/santifer/career-ops/blob/main/applications.md) to guarantee uniform semantics. It reads the tracker, scans each row, and rewrites non-canonical values into one of the eight canonical states defined in [`templates/states.yml`](https://github.com/santifer/career-ops/blob/main/templates/states.yml). ### Mapping Aliases and Stripping Markup The `normalizeStatus()` function, located at lines 28–84, first strips markdown bold markers, lower-cases the raw string, and then matches it against a hard-coded list of English canonical statuses and Spanish aliases. This collapses variants such as `**Applied**`, `aplicado`, or `Aplicado 2024-01-01` into the single canonical value `Applied`. ### Special Cases and Column Migration Special handling at lines 26–35 detects `DUPLICADO` or `REPOST` annotations and moves them into the notes column rather than treating them as statuses. Dates embedded inside status strings are removed, while em-dash or empty cells are normalized to `Discarded`. These rules prevent free-text artifacts from leaking into the state machine that downstream tooling relies on. ### Backup and In-Place Rewrite Before overwriting, the script creates a `.bak` copy via `copyFileSync` and then writes the cleaned tracker back to [`applications.md`](https://github.com/santifer/career-ops/blob/main/applications.md). The line reconstruction logic at lines 41–44 rebuilds each markdown table row with canonical statuses and cleaned score fields, ensuring the file remains valid markdown. Run the normalizer with: ```bash node career-ops/normalize-statuses.mjs # clean the tracker in-place node career-ops/normalize-statuses.mjs --dry-run # show changes without writing ``` Typical console output looks like: ``` #42: "aplicado 2024-01-01" → "Applied" 📊 7 statuses normalized ✅ Written to applications.md (backup: applications.md.bak) ``` ## End-to-End Pipeline Integrity Guarantees Running `merge-tracker.mjs` and `normalize-statuses.mjs` together closes the loop on several data-consistency risks addressed by the pipeline integrity system. The exclusive lock plus atomic rename eliminates lost updates from concurrent batch jobs. Multi-layer deduplication—exact report number, row-plus-company matching, and fuzzy role comparison—prevents redundant entries without blocking legitimate reposts. Score-aware updates protect historical evaluations from accidental downgrade. Finally, status canonicalization removes language drift, markup noise, and misplaced metadata so that every downstream consumer reads the same eight canonical states. ## Summary - **Exclusive locking:** `acquireTrackerLock()` in `merge-tracker.mjs` prevents concurrent writes by creating a PID-backed mutex with stale-lock recovery. - **Atomic writes:** `writeFileAtomic()` commits changes via a temporary file and `renameSync`, ensuring readers never see a partially updated tracker. - **Three-layer deduplication:** Exact report number, normalized company plus row number, and Jaccard-based fuzzy role matching catch duplicates while allowing distinct reposts. - **Score-preserving updates:** Incoming rows overwrite existing ones only when `parseScore()` returns a strictly higher value, preventing downgrade. - **Status canonicalization:** `normalize-statuses.mjs` maps aliases, Spanish variants, and markdown artifacts to the eight canonical states defined in [`templates/states.yml`](https://github.com/santifer/career-ops/blob/main/templates/states.yml). - **Auditability:** Both scripts emit detailed console output and support `--dry-run` for safe previews. ## Frequently Asked Questions ### How does `merge-tracker.mjs` prevent two batch jobs from corrupting [`applications.md`](https://github.com/santifer/career-ops/blob/main/applications.md)? Before reading or writing, the script calls `acquireTrackerLock()` (lines 19–34), which creates a lock directory and records the owner PID. If another process holds the lock, the script retries with back-off and can recover stale locks. The exclusive critical section, paired with an atomic `renameSync` write, prevents simultaneous modifications. ### What happens when a merged row has the same report number but a lower score than the existing entry? The merge script uses `parseScore()` to compare the incoming score against the existing one. It overwrites the row only when the new score is strictly higher; otherwise the candidate is skipped. This rule prevents any batch import from downgrading a previously recorded evaluation. ### Which non-canonical status values does `normalize-statuses.mjs` handle? `normalize-statuses.mjs` strips markdown bold, lower-cases strings, and maps both English and Spanish aliases to one of eight canonical states defined in [`templates/states.yml`](https://github.com/santifer/career-ops/blob/main/templates/states.yml). It also moves `DUPLICADO` or `REPOST` annotations into the notes column, removes dates from statuses, and treats empty cells as `Discarded`. ### Can I preview merge or normalization changes without modifying [`applications.md`](https://github.com/santifer/career-ops/blob/main/applications.md)? Yes. Both scripts accept a `--dry-run` flag that previews additions, updates, or status rewrites in the console without executing the atomic write or creating a `.bak` file. This allows safe inspection before any permanent change to [`applications.md`](https://github.com/santifer/career-ops/blob/main/applications.md).