How merge-tracker.mjs Handles TSV Tracker Additions and Link Normalization in Career Ops

The merge-tracker.mjs script processes pending TSV files from batch/tracker-additions/, normalizes report links to be relative to the tracker directory using normalizeReportLink(), detects duplicates via report numbers and fuzzy matching, and atomically merges valid entries into applications.md while skipping or updating existing records based on score comparison.

The merge-tracker.mjs utility in the santifer/career-ops repository automates the integration of batch-generated TSV tracker additions into the master applications tracker. This defensive pipeline supports multiple column layouts, validates canonical status values against a predefined set, and ensures that Markdown report links remain functional whether the tracker resides at the repository root or under data/ (addressing issue #760).

Discovering Pending TSV Files

The script begins by scanning the dedicated additions directory for pending batch files. Using readdirSync on batch/tracker-additions/, it filters for files ending with the .tsv extension:

const tsvFiles = readdirSync(ADDITIONS_DIR).filter(f => f.endsWith('.tsv'));

If no files are detected, the script exits early with a confirmation message (✅ No pending additions to merge.). When files are present, the script acquires an exclusive filesystem lock to prevent race conditions during the merge process.

Parsing TSV Content with Multiple Format Support

The parseTsvContent(content, filename) function (lines 65–86) handles three distinct input formats flexibly, allowing contributors to submit data in whatever format they generated.

Handling Pipe-Delimited Markdown Tables

When the content starts with a pipe character (|), the parser recognizes it as a Markdown table row:

if (content.startsWith('|')) {
  // Split on pipe delimiter and trim whitespace
  const columns = content.split('|').map(c => c.trim()).filter(c => c);
}

This detection occurs at lines 77–84, enabling direct copying from rendered Markdown tables into TSV files.

Tab-Separated Value Processing

For standard TSV files, the parser splits on tab characters:

const columns = content.split('\t');

The function gracefully handles malformed rows by emitting warnings and returning null (lines 79–82 and 99–101), preventing a single bad row from crashing the entire batch.

Column Order Detection and Validation

The script uses heuristics on columns 4 and 5 to determine whether they represent status-score or score-status ordering (lines 103–126). Once detected, the status value undergoes normalization through validateStatus() (lines 26–54), which ensures only canonical values from the predefined set (lines 24–44) are accepted. Scores are extracted and later parsed by parseScore() (lines 48–51) for numerical comparison.

Normalizing Report Links for Cross-Directory Compatibility

A critical function of the merger is ensuring report links work regardless of the tracker's location. Raw TSV entries contain root-relative paths (e.g., reports/123-acme-senior-engineer.md), but the tracker may reside in data/applications.md or ./applications.md.

The script imports the normalization utility from tracker-links.mjs:

import { normalizeReportLink as normalizeLink } from './tracker-links.mjs';

const normalizeReportLink = (reportField) => normalizeLink(reportField, TRACKER_DIR, REPORTS_ROOT);

During the merge loop, each addition's report field is rewritten before duplicate detection:

addition.report = normalizeReportLink(addition.report);

This transformation (lines 31–35) converts root-relative paths to absolute paths, then re-relativizes them to TRACKER_DIR. The result is a properly relative link like [123](../reports/123-acme-senior-engineer.md) when the tracker lives in data/, or [123](reports/123-acme-senior-engineer.md) when at the repo root.

Duplicate Detection and Scoring Logic

After normalization, the script implements a three-tier strategy to identify existing entries (lines 36–70).

Three-Tier Duplicate Finding Strategy

1. Exact report number match: Extracts the number from the Markdown link using extractReportNum() and searches for existing rows with identical numbers (lines 39–48).

2. Exact tracker-row number and company match: Guards against numbering mismatches across different companies by checking both the row number and normalized company name (lines 50–61).

3. Fuzzy company-role match: Uses roleFuzzyMatch() (lines 17–41) after normalizing company names via normalizeCompany() (line 57) to catch entries where the report number may have changed but the company and role combination represents the same application.

Score-Based Merge Decisions

When duplicates are found, the script compares scores using parseScore():

• Higher score: Triggers an update (lines 72–84), rewriting the row while preserving the existing status and PDF flag, and appending a note indicating the re-evaluation.
• Lower or equal score: The addition is skipped (lines 85–87) to prevent stale data from overwriting fresher evaluations.

For entirely new entries, the script inserts the row using either the TSV-provided number or the next sequential available number (lines 88–99). All actions are logged with emojis (➕ Add, 🔄 Update, ⏭️ Skip) for immediate visual feedback.

Atomic Writes and File Management

When not running in --dry-run mode, the script writes the final tracker content using writeFileAtomic() (lines 86–96), preventing partial writes that could corrupt the tracker during system interruptions.

Successfully processed TSV files are moved to batch/tracker-additions/merged/ (lines 19–24), ensuring they are not re-processed in subsequent runs. This housekeeping maintains the integrity of the batch processing queue.

Summary

• Discovery: Scans batch/tracker-additions/ for .tsv files and exits early if none exist.
• Parsing: Handles pipe-delimited Markdown, tab-separated values, and variable column ordering while validating canonical status values.
• Normalization: Rewrites report links using normalizeReportLink() to ensure they remain functional relative to the tracker's location.
• Deduplication: Employs three strategies—report number matching, tracker-row matching, and fuzzy company-role matching—to identify existing entries.
• Merging: Updates existing entries when the new score is higher, adds new entries when no match exists, and skips when the score is lower or equal.
• Safety: Uses atomic file writes and exclusive locks to prevent data corruption during concurrent operations.

Frequently Asked Questions

How does merge-tracker.mjs handle different TSV column formats?

The parseTsvContent() function automatically detects whether the input uses pipe-delimited Markdown tables (starting with |) or standard tab-separated values. It then applies heuristics to columns 4 and 5 to determine if they represent status-score or score-status ordering, ensuring flexibility regardless of how the TSV was generated.

What happens if a TSV file contains an invalid status value?

The script calls validateStatus() (lines 26–54) which normalizes input against the canonical status set defined in the source code. Invalid status values trigger validation errors before the merge proceeds, preventing malformed data from entering the master tracker.

How does the script prevent duplicate application entries?

The script checks for duplicates using three methods: exact report number matching via extractReportNum(), exact tracker-row number combined with normalized company name, and fuzzy matching of company-role pairs using roleFuzzyMatch(). When duplicates are found, it compares scores using parseScore() and only updates if the new entry has a higher score.

Why are report links normalized, and how does it work?

Report links are normalized to ensure they work correctly whether the tracker file is at the repository root or under data/. The normalizeReportLink() function (utilizing tracker-links.mjs) converts root-relative paths to paths relative to the tracker's directory, addressing issue #760 and ensuring clickable links in all repository layouts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}