Headroom Relevance Scoring Methods: BM25, Embedding, and Hybrid (BM2S) Explained

Headroom provides three interchangeable relevance scorers—BM25 for exact keyword matches, Embedding for semantic similarity, and Hybrid (BM2S) for adaptive fusion—implemented in headroom/relevance/ with a common protocol defined in base.py.

The chopratejas/headroom repository implements a "smart crusher" compression pipeline powered by pluggable relevance scoring. These Headroom relevance scoring methods enable context-aware filtering by ranking JSON log items against a user query using either classical keyword matching, neural embeddings, or an intelligent combination of both.

Architecture of the Relevance Scoring Pipeline

All three scoring strategies share a unified interface defined in headroom/relevance/base.py, ensuring any transform can swap scorers via the create_scorer factory without changing consumer code.

Base Protocol (headroom/relevance/base.py)

The foundation consists of two core abstractions:

• RelevanceScore – A dataclass clamping scores to [0, 1] while providing a human-readable reason string and a list of matched_terms for observability.
• RelevanceScorer – An abstract class requiring score(item, context) and score_batch(items, context) implementations. Subclasses inherit default_batch_score, a helper that falls back to per-item iteration when no optimized batch path exists.

This protocol ensures that whether you use BM25, Embedding, or Hybrid scoring, the output shape remains identical.

BM25 Keyword Scoring (headroom/relevance/bm25.py)

The BM25Scorer provides zero-dependency keyword matching optimized for exact identifiers.

Tokenization strategy: A regex extracts UUIDs, numeric IDs (4+ digits), and alphanumeric tokens while preserving case-insensitive terms.

IDF computation: Standard BM25 inverse document frequency using log((N - n + 0.5) / (n + 0.5) + 1) with a safe fallback to 0.0 when doc_freq == 0.

Scoring and normalization: The _bm25_score method applies term scores weighted by corpus IDF. Raw scores are normalized against a max_score default of 10.0 and boosted for long matches—UUIDs of 8+ characters receive additional weight to prioritize precise identifier lookups.

Embedding Scoring (headroom/relevance/embedding.py)

The EmbeddingScorer enables semantic similarity through ONNX-backed neural networks.

Dependencies: Requires fastembed (ONNX runtime) and numpy, installable via pip install headroom[relevance].

Model configuration: Defaults to BAAI/bge-small-en-v1.5 (33M parameters, 384-dimensional, int8-quantized) loaded via fastembed.TextEmbedding.

Encoding and similarity: The self._encode method batches all items plus the context into a single call to embed, returning a NumPy matrix. Cosine similarity is computed between context and item vectors, then clamped to [0, 1].

Cross-language parity: The batch API mirrors Headroom’s Rust implementation to ensure consistent behavior across language bindings.

Hybrid Adaptive Fusion (headroom/relevance/hybrid.py)

The HybridScorer implements BM2S (BM25 + Embedding) scoring, automatically weighting keyword and semantic signals based on query characteristics.

Composition: Internally maintains a BM25Scorer instance and optionally an EmbeddingScorer.

Adaptive alpha calculation: The _compute_alpha method inspects the context for specific patterns:

• UUID detected → α ≥ 0.85 (heavy BM25 weighting)
• ≥ 2 numeric IDs → α ≥ 0.75
• 1 numeric ID → α ≥ 0.65
• Hostname or email pattern → α ≥ 0.60
• General semantic query → uses base_alpha (default 0.5)

Fusion formula: Final score = α * bm25_score + (1 - α) * embedding_score.

Graceful degradation: If fastembed is unavailable, the scorer boosts BM25 scores (minimum 0.3 for any match, +0.2 bonus for ≥ 2 matches) to prevent relevance collapse.

Choosing Between Scoring Methods

Select the appropriate scorer based on your query characteristics and deployment constraints:

• BM25Scorer – Use for exact-match lookups involving UUIDs, numeric IDs, or precise technical terms. Zero external dependencies make it ideal for constrained environments.
• EmbeddingScorer – Deploy when handling paraphrased queries, synonyms, or fuzzy semantic matching. Requires ONNX runtime but provides superior recall for natural language.
• HybridScorer – The general-purpose default. Automatically detects identifier-heavy queries (UUIDs, IDs) and shifts weight toward BM25, while leveraging embeddings for conceptual similarity. Best for mixed workloads where queries may contain both precise identifiers and descriptive terms.

Practical Implementation Examples

Instantiate any scorer through the factory function in headroom/relevance/__init__.py:

from headroom.relevance import create_scorer, RelevanceScore

# BM25: Zero-dependency keyword matching

bm25 = create_scorer("bm25")
items = [
    '{"id":"550e8400-e29b-41d4-a716-446655440000","name":"Alice"}',
    '{"id":"123e4567-e89b-12d3-a456-426614174000","name":"Bob"}',
]
context = "find record 550e8400-e29b-41d4-a716-446655440000"
bm25_scores = bm25.score_batch(items, context)
print([s.score for s in bm25_scores])   # → [>0.8, <0.2]

# Embedding: Semantic similarity via fastembed

emb = create_scorer("embedding")
items = [
    '{"status":"failed","error":"connection refused"}',
    '{"status":"success","data":[1,2,3]}',
]
context = "show me the errors"
emb_scores = emb.score_batch(items, context)
print([s.score for s in emb_scores])    # → higher for the first item

# Hybrid: Adaptive fusion with automatic alpha selection

hybrid = create_scorer("hybrid", adaptive=True)
items = [
    '{"id":"12345","message":"User login failed"}',
    '{"id":"67890","message":"All systems operational"}',
]

# ID-heavy query triggers α ≈ 0.85 (BM25 dominates)

ctx_id = "find user 12345"

# Semantic query uses base_alpha ≈ 0.5 (balanced fusion)

ctx_semantic = "show recent login errors"

print(hybrid.score_batch(items, ctx_id)[0].score)      # Boosted BM25 match

print(hybrid.score_batch(items, ctx_semantic)[0].score)  # Blended semantic+keyword

All scorers return RelevanceScore objects containing score, reason, and matched_terms, enabling detailed logging and debugging throughout the compression pipeline.

Summary

• BM25 scoring in headroom/relevance/bm25.py provides zero-dependency keyword matching with IDF weighting and UUID-specific boosts.
• Embedding scoring in headroom/relevance/embedding.py leverages fastembed and BAAI/bge-small-en-v1.5 for 384-dimensional semantic similarity via cosine distance.
• Hybrid (BM2S) scoring in headroom/relevance/hybrid.py adaptively fuses both signals using query-pattern detection (UUIDs, IDs, emails) to dynamically adjust the alpha weighting.
• Unified protocol in headroom/relevance/base.py ensures all scorers implement score() and score_batch() methods, consumable via create_scorer() from headroom/relevance/__init__.py.

Frequently Asked Questions

What is the default alpha value in Headroom's Hybrid scorer?

The HybridScorer uses a base_alpha of 0.5 when no specific query patterns are detected. However, when adaptive=True, the _compute_alpha method automatically increases this weight toward BM25 (up to 0.85) if the context contains UUIDs, numeric IDs, hostnames, or email addresses.

Does the BM25 scorer require any external dependencies?

No. The BM25Scorer is implemented in pure Python with zero external dependencies in headroom/relevance/bm25.py. It uses standard library modules for regex tokenization and mathematical operations, making it suitable for lightweight deployments where ONNX or PyTorch cannot be installed.

How does Headroom handle missing embedding models in the Hybrid scorer?

According to headroom/relevance/hybrid.py, the HybridScorer implements graceful degradation. If the EmbeddingScorer fails to initialize (missing fastembed), the system falls back to BM25-only scoring with automatic boosting: any match receives a minimum score of 0.3, and matches containing two or more terms receive an additional 0.2 lift, preventing relevance collapse.

Which embedding model does Headroom use for semantic scoring?

The EmbeddingScorer defaults to BAAI/bge-small-en-v1.5 via the fastembed library. This model provides 384-dimensional embeddings with int8 quantization (33M parameters), offering a balance between accuracy and inference speed suitable for log analysis workloads.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}