How to Configure Relevance Scoring in Headroom: BM2SScorer vs EmbeddingScorer

Headroom delegates relevance decisions to objects implementing the RelevanceScorer protocol defined in headroom/relevance/base.py, automatically selecting EmbeddingScorer for semantic matching when fastembed is present or falling back to the lexical BM2SScorer for zero-dependency environments.

The chopratejas/headroom library uses a pluggable relevance scoring system to determine which document sections best match a query before compression. Configuring relevance scoring allows you to choose between fast lexical matching (BM25-style) and dense semantic embeddings, depending on your runtime constraints and accuracy requirements. You can explicitly instantiate either scorer or rely on the automatic HybridScorer detection mechanism provided in headroom/relevance/hybrid.py.

Understanding Headroom's Relevance Scoring Architecture

Headroom's compression pipeline relies on a scorer to rank document fragments before trimming or rewriting them. The architecture centers on a protocol-based design that enables swappable implementations.

The RelevanceScorer Protocol

All scorers inherit from the abstract base class defined in headroom/relevance/base.py. This protocol standardizes how relevance scores are calculated across different algorithms, ensuring the Headroom client can interact with any scorer implementation uniformly.

Available Scorer Implementations

Headroom ships with two concrete implementations:

• EmbeddingScorer – A semantic scorer that generates dense embeddings using the fastembed library and the BAAI/bge-small-en-v1.5 model (approximately 30MB). It captures synonyms and paraphrases through vector similarity.
• BM2SScorer – A lexical scorer implementing BM25-style term weighting. It operates using pure Python standard libraries with no external model downloads, making it ideal for constrained environments.

Automatic Scorer Selection with HybridScorer

When you instantiate Headroom without specifying a scorer, the library uses HybridScorer to automatically detect available dependencies.

As implemented in headroom/relevance/embedding.py, the EmbeddingScorer.is_available() method attempts to import fastembed and returns True only if the package is present. HybridScorer checks this condition and defaults to EmbeddingScorer when available, otherwise falling back to BM2SScorer.

from headroom.relevance import HybridScorer

# Automatically selects EmbeddingScorer if fastembed is installed,

# otherwise uses BM2SScorer

scorer = HybridScorer()

Explicit Configuration Methods

For production deployments requiring deterministic behavior, instantiate your preferred scorer explicitly and pass it to the Headroom client.

Installing Optional Dependencies

To enable semantic scoring, install the relevance extras:

pip install "headroom[relevance]"

This command pulls in fastembed and its ONNX runtime dependencies.

Instantiating Specific Scorers

Force a specific scorer implementation regardless of installed packages:

from headroom import Headroom
from headroom.relevance import EmbeddingScorer, BM2SScorer

# Force semantic scoring with embeddings

headroom_semantic = Headroom(relevance_scorer=EmbeddingScorer())

# Force lexical BM25-style scoring

headroom_lexical = Headroom(relevance_scorer=BM2SScorer())

Tuning Scorer Parameters

Both scorers expose constructor arguments for fine-tuning behavior.

EmbeddingScorer Configuration

Override the default model or caching behavior:

from headroom.relevance import EmbeddingScorer

# Use a custom embedding model

scorer = EmbeddingScorer(
    model_name="BAAI/bge-base-en-v1.5",
    cache_model=True
)

BM2SScorer Hyperparameters

Adjust the BM25 algorithm parameters:

from headroom.relevance import BM2SScorer

# Tune k1 and b parameters for specific document collections

scorer = BM2SScorer(k1=1.5, b=0.8)

Practical Implementation Examples

Semantic Scoring with EmbeddingScorer

Score individual fragments using dense vector similarity:

from headroom.relevance import EmbeddingScorer

scorer = EmbeddingScorer()
result = scorer.score(
    '{"status": "failed", "error": "connection refused"}',
    "show me the errors"
)

print(result.score)   # 0.73 (example)

print(result.reason)  # "Embedding: semantic similarity 0.73"

Batch Processing

Process multiple document fragments efficiently:

items = [
    '{"status": "failed"}',
    '{"status": "success"}',
    '{"error": "timeout"}',
]

scorer = EmbeddingScorer()
batch = scorer.score_batch(items, "what errors happened?")

for r in batch:
    print(r.score, r.reason)

Lexical Scoring with BM2SScorer

Use term-frequency based matching without external dependencies:

from headroom.relevance import BM2SScorer

scorer = BM2SScorer(k1=1.2, b=0.75)
result = scorer.score(
    '{"status": "failed", "error": "connection refused"}',
    "show me the errors"
)

print(result.score)   # 0.45 (example, based on term overlap)

Runtime Selection with HybridScorer

Verify which scorer was selected at runtime:

from headroom.relevance import HybridScorer

scorer = HybridScorer()
print(type(scorer))  # <class 'headroom.relevance.embedding.EmbeddingScorer'>

Summary

• Headroom uses the RelevanceScorer protocol defined in headroom/relevance/base.py to standardize relevance calculations.
• EmbeddingScorer requires pip install "headroom[relevance]" and provides semantic matching via the BAAI/bge-small-en-v1.5 model.
• BM2SScorer provides fast, dependency-free lexical matching suitable for resource-constrained environments.
• HybridScorer automatically selects EmbeddingScorer when fastembed is available, falling back to BM2SScorer otherwise.
• Pass explicit scorer instances to the Headroom constructor via the relevance_scorer parameter for deterministic behavior.

Frequently Asked Questions

How do I force Headroom to use lexical scoring even if fastembed is installed?

Instantiate BM2SScorer explicitly and pass it to the Headroom constructor:

from headroom import Headroom
from headroom.relevance import BM2SScorer

headroom = Headroom(relevance_scorer=BM2SScorer())

This bypasses the HybridScorer automatic selection logic and guarantees the use of BM25-style term matching regardless of optional dependencies.

Can I use a different embedding model with EmbeddingScorer?

Yes. The EmbeddingScorer constructor accepts a model_name parameter to specify any model supported by fastembed. For example:

scorer = EmbeddingScorer(model_name="BAAI/bge-base-en-v1.5")

The default model is BAAI/bge-small-en-v1.5, which balances accuracy and size at approximately 30MB.

What are the performance trade-offs between BM2SScorer and EmbeddingScorer?

BM2SScorer operates entirely in Python with no model loading overhead, making it faster for short queries and suitable for environments without GPU or significant RAM. EmbeddingScorer incurs a one-time model download (approximately 30MB) and higher memory usage, but provides superior accuracy for semantic queries involving synonyms, paraphrases, or conceptually related terms that lack lexical overlap.

Where is the automatic scorer selection logic implemented?

The automatic fallback mechanism resides in headroom/relevance/hybrid.py. This module implements HybridScorer, which checks EmbeddingScorer.is_available() (defined in headroom/relevance/embedding.py) to determine whether the semantic dependencies are importable before defaulting to the pure-Python BM2SScorer.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}