What Is Lineage in WhichLLM’s Ranking System? Recency-Aware Scoring for Model Families

Lineage in WhichLLM’s ranking system is a family-aware generational ordering that applies a decaying multiplicative factor to frozen benchmark scores, ensuring newer model generations are not unfairly penalized by missing current data.

WhichLLM uses lineage to solve a specific leaderboard problem: when only outdated benchmark snapshots exist, a new model release might have no current scores to compete with older, heavily-tested versions. This mechanism injects recency awareness directly into the scoring pipeline by analyzing model family generations. Understanding lineage is critical for interpreting WhichLLM rankings and contributing new family mappings.

How Lineage Works in WhichLLM’s Ranking System

The lineage heuristic operates across four distinct stages inside the WhichLLM codebase. Each stage transforms static model metadata into an adjustable recency score that prevents stale benchmarks from dominating the leaderboard.

Static Model Family Tables

The lineage mapping lives in src/whichllm/data/lineage.py. This file defines MODEL_LINEAGE_VERSIONS, a dictionary that assigns generation indices to known model families such as qwen, llama, and deepseek. Each entry contains regex patterns paired with an integer index, where a higher index represents a newer generation.

Regex Pre-compilation in the Ranker

At import time, src/whichllm/engine/ranker.py compiles these patterns into a fast-lookup table stored in _LINEAGE_REGEX. Lines 28–33 iterate over every family and compile each pattern:

_LINEAGE_REGEX = {
    family: [(re.compile(pat), idx) for pat, idx in entries]
    for family, entries in MODEL_LINEAGE_VERSIONS.items()
}

This pre-compilation ensures that family detection adds minimal overhead during the ranking process.

Recency Factor Calculation

When a model has only frozen scores—for example, from the Open LLM Leaderboard v2—the _lineage_recency_factor() function in src/whichllm/models/benchmark.py computes a multiplier. According to lines 86–102, the logic measures how many generations separate a model from the newest entry in its family:

gens_old = max(0, max_idx - idx)          # generations older than newest

factor = max(0.55, 1.0 - 0.12 * gens_old) # each older gen ≈‑12%

The resulting factor is 1.0 for the newest generation and decays linearly to a hard floor of 0.55 for the oldest tracked releases.

Application to Frozen Benchmark Scores

The _apply_lineage_recency_demotion function (lines 108–128 of the same file) applies this factor exclusively to frozen-score entries during the benchmark merging stage:

factor = _lineage_recency_factor(k)
out[k] = round(v * factor, 1)

By multiplying frozen values before they are combined with current scores, WhichLLM prevents a stale high score from outranking a newer model that has not yet appeared in frozen leaderboards. As documented in docs/scoring.md under the "Generation lineage" section, this design gives newer generations a small bonus and older generations a small penalty.

Practical Code Examples for WhichLLM Lineage

You can interact with the lineage system directly using the private helper functions exposed in the WhichLLM Python package.

Querying the Lineage Factor for a Model ID

To see the exact recency multiplier assigned to a specific model:

from whichllm.models.benchmark import _lineage_recency_factor

model_id = "meta-llama-3.2-70b"
factor = _lineage_recency_factor(model_id)
print(f"Lineage factor for {model_id!r}: {factor}")

# → 0.88   (two generations older than the newest llama entry)

Adding a New Model Family

Contributors can extend lineage support by editing src/whichllm/data/lineage.py and appending to MODEL_LINEAGE_VERSIONS:

# src/whichllm/data/lineage.py – add a new family

MODEL_LINEAGE_VERSIONS["mynewmodel"] = [
    (r"mynewmodel-v5", 5),   # newest generation

    (r"mynewmodel-v4", 4),
    (r"mynewmodel-v3", 3),
]

The ranker will automatically pick up the new family on the next import because _LINEAGE_REGEX is built at import time.

Inspecting the Compiled Regex Table

For debugging or auditing purposes, inspect the compiled patterns that the ranker uses:

from whichllm.engine.ranker import _LINEAGE_REGEX

for fam, patterns in _LINEAGE_REGEX.items():
    print(f"{fam}: {[p.pattern for p, _ in patterns]}")

Demonstrating Lineage Demotion on Frozen Scores

This example shows how an older generation’s frozen score is demoted while a newer generation remains unchanged:

from whichllm.models.benchmark import _apply_lineage_recency_demotion

combined = {"llama-3.1-8b": 85.0, "llama-2-7b": 90.0}
frozen   = {"llama-3.1-8b": 85.0, "llama-2-7b": 90.0}
current  = {}                     # no current evidence

adjusted = _apply_lineage_recency_demotion(combined, frozen, current)
print(adjusted)

# {'llama-3.1-8b': 85.0, 'llama-2-7b': 77.0}

# ^ older generation (llama‑2) received a ~0.12*1 = 0.88 factor → 90×0.88≈77

Summary

• Lineage is a recency heuristic that prevents stale frozen benchmarks from distorting WhichLLM rankings.
• The static family tables are maintained in src/whichllm/data/lineage.py via MODEL_LINEAGE_VERSIONS.
• src/whichllm/engine/ranker.py pre-compiles regexes at import time for efficient model detection.
• src/whichllm/models/benchmark.py calculates the recency factor via _lineage_recency_factor() and applies it through _apply_lineage_recency_demotion.
• The factor ranges from 1.0 for the newest generation down to a minimum of 0.55 for the oldest tracked generation.

Frequently Asked Questions

What does lineage mean in WhichLLM?

In WhichLLM, lineage refers to a family-aware generational ordering that tracks how old a model release is relative to the newest version in its family. It is used solely to adjust frozen benchmark scores so that newer hardware or architecture improvements are reflected fairly in the ranking pipeline.

How does WhichLLM calculate the lineage recency factor?

The WhichLLM source code calculates the factor in src/whichllm/models/benchmark.py by finding a model’s generation index and subtracting it from the family’s maximum index. The formula factor = max(0.55, 1.0 - 0.12 * gens_old) applies approximately a 12% penalty per older generation, capped at a floor of 0.55.

Which models are affected by lineage adjustments?

Only models backed exclusively by frozen benchmark data receive lineage adjustments. If current, live benchmark evidence exists for a model, WhichLLM does not apply the recency demotion because the score already reflects up-to-date performance.

How can I add a new model family to WhichLLM’s lineage system?

You can extend MODEL_LINEAGE_VERSIONS inside src/whichllm/data/lineage.py with regex patterns and ascending generation indices. Because src/whichllm/engine/ranker.py rebuilds _LINEAGE_REGEX at import time, the ranker automatically recognizes the new family on the next application startup without additional wiring.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}