# How LLM Model Quality is Scored in llmfit: A Technical Deep Dive into the Benchmarking Engine

> Discover how llmfit scores LLM model quality using regex rubrics and composite metrics. Learn about accuracy, inference speed, and automated routing in this technical deep dive.

- Repository: [Alex Jones/llmfit](https://github.com/AlexsJones/llmfit)
- Tags: deep-dive
- Published: 2026-08-19

---

**llmfit scores LLM model quality using a regex-based rubric system defined in YAML configuration files, calculating composite metrics from pattern-matching accuracy and inference speed to drive automated model routing decisions.**

The llmfit repository implements a sophisticated quality-benchmark subsystem that transforms subjective LLM output evaluation into quantitative metrics. Located primarily in [`llmfit-core/src/quality.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/quality.rs), this system enables developers to compare models objectively and automatically select the best performer for specific roles such as coding, research, or general tasks.

## The Three-Stage Quality Scoring Pipeline

The quality evaluation process follows a structured pipeline that separates configuration, inference, and scoring concerns. This architecture ensures reproducible benchmarks across different model providers and endpoints.

### Stage 1: Loading Role-Based Test Suites

Benchmarking begins with declarative configuration loaded from an embedded YAML file ([`benchmarks.yaml`](https://github.com/AlexsJones/llmfit/blob/main/benchmarks.yaml)). This file defines **roles** (e.g., *general*, *coding*, *research*), each containing one or more **quality tests**. Every test specifies a prompt and a vector of `ScoringRule` objects that determine how responses are evaluated.

The core data structures parsing this configuration appear in lines 31-56 of [`quality.rs`](https://github.com/AlexsJones/llmfit/blob/main/quality.rs):

- `QualityConfig`: The root configuration container
- `RoleDef`: Defines a specific use case and its associated tests
- `QualityTestDef`: Contains the prompt and scoring rules for individual test cases
- `ScoringRule`: Specifies regex patterns, weights, case sensitivity, and negation logic

### Stage 2: Executing Inference and Capturing Responses

Once configured, llmfit dispatches prompts to the target model through provider-specific adapters. The system supports both local Ollama instances and OpenAI-compatible APIs via functions defined in lines 75-92:

- `quality_ollama_generate`: Interfaces with Ollama's generate endpoint
- `quality_openai_chat`: Communicates with OpenAI-style chat completion endpoints

Both functions return an `InferenceResponse` struct containing the raw text output and timing metadata, including tokens-per-second metrics critical for the speed component of the final score.

### Stage 3: Evaluating Responses and Calculating Composite Scores

The `evaluate_response` function drives the actual quality calculation. This implementation iterates over each `ScoringRule` in the test definition, applying regex pattern matching with configurable case sensitivity. When `negate` is set to true, matches subtract points rather than adding them.

Raw quality scores are clamped to the range **0-10**. Simultaneously, speed scores derive from the `tok_per_sec` field and normalize using the formula `(tok_per_sec / 3).min(10)`, capping performance at 10x the baseline threshold.

The **composite score** blends these dimensions using a configurable `speed_weight` parameter (default 1.0):

```rust
composite = (quality * 2.0 + speed_norm * speed_weight) / (2.0 + speed_weight)

```

This weighting prioritizes response accuracy while incorporating latency characteristics essential for production deployment decisions.

## Aggregation and Routing Logic

Individual test results aggregate into per-role averages represented by `RoleScore` structs. The `run_all_tests` function (lines 69-78 and 108-140 in [`quality.rs`](https://github.com/AlexsJones/llmfit/blob/main/quality.rs)) handles this aggregation, computing mean scores across all tests within a role before assembling the final `ModelQualityResult`.

Beyond simple reporting, llmfit leverages these scores for intelligent model selection. The `compute_routing` function generates a routing matrix mapping each role to its optimal model, while `compute_runner_ups` identifies secondary options. These matrices enable dynamic request routing based on empirical quality data rather than heuristics.

## Implementation Examples

### Running a Complete Benchmark

The following example demonstrates benchmarking a local Ollama model against the default quality configuration:

```rust
// Load the built-in quality configuration (embedded benchmarks.yaml)
let cfg = llmfit_core::quality::default_quality_config();

// Run a quality benchmark against an Ollama endpoint
let result = llmfit_core::quality::bench_quality_ollama(
    "http://localhost:11434",   // Ollama base URL
    "llama3:8b",                // model name
    &cfg,
    None,                       // optional role filter
).expect("benchmark failed");

// Display results
result.display();               // Human-readable table
result.display_json();          // JSON output

```

### Direct Response Evaluation

For custom testing outside the full benchmark suite, evaluate individual responses against specific scoring rules:

```rust
use llmfit_core::quality::{ScoringRule, evaluate_response};

let rules = vec![
    ScoringRule { 
        pattern: "hello".into(), 
        weight: 3, 
        negate: false, 
        case_insensitive: true 
    },
    ScoringRule { 
        pattern: "error".into(), 
        weight: -5, 
        negate: false, 
        case_insensitive: false 
    },
];

let score = evaluate_response("Hello world!", &rules);   // Returns 3.0
println!("Quality score: {}", score);

```

### Generating Routing Matrices

After benchmarking multiple models, compute optimal assignments:

```rust
let all_results = vec![result1, result2, result3];
let routing = llmfit_core::quality::compute_routing(&all_results);
let runner_ups = llmfit_core::quality::compute_runner_ups(&all_results);

llmfit_core::quality::display_routing_matrix(&routing, &runner_ups);

```

## Key Source Files

Understanding llmfit's quality scoring requires familiarity with these components:

- **[`llmfit-core/src/quality.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/quality.rs)**: Core implementation containing `ScoringRule` definitions, `evaluate_response` logic, benchmark runners, and composite calculations
- **[`llmfit-core/data/benchmarks.yaml`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/data/benchmarks.yaml)**: Embedded declarative configuration defining test prompts and scoring rubrics
- **[`llmfit-core/src/bench.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/bench.rs)**: Shared HTTP response types (`ChatCompletionResponse`, `OllamaGenResponse`) consumed by the quality module
- **[`llmfit-core/src/analysis.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/analysis.rs)**: Integration point connecting quality results with deployment fit analysis

## Summary

- **Regex-based evaluation**: llmfit uses `ScoringRule` structs with configurable patterns, weights, and negation logic to objectively grade model outputs
- **Composite scoring**: Final metrics combine quality (0-10) and normalized speed scores using a weighted formula that balances accuracy against performance
- **Role-specific benchmarking**: The [`benchmarks.yaml`](https://github.com/AlexsJones/llmfit/blob/main/benchmarks.yaml) configuration enables targeted evaluation across distinct use cases like coding and research
- **Automated routing**: Aggregate `ModelQualityResult` data powers the `compute_routing` matrix for intelligent model selection
- **Provider agnostic**: Unified interfaces support both Ollama local instances and OpenAI-compatible APIs through `quality_ollama_generate` and `quality_openai_chat`

## Frequently Asked Questions

### How does llmfit handle case sensitivity in quality scoring?

Each `ScoringRule` includes a `case_insensitive` boolean field. When set to true, the regex matcher applies patterns without regard to letter casing. This configuration is defined in the YAML test definitions and parsed into the `ScoringRule` struct in [`llmfit-core/src/quality.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/quality.rs).

### Can I adjust the importance of speed versus quality in the final score?

Yes. The composite calculation uses a `speed_weight` parameter that defaults to 1.0 but is fully configurable. Increasing this value prioritizes inference speed, while decreasing it emphasizes response quality. The formula `(quality * 2.0 + speed_norm * speed_weight) / (2.0 + speed_weight)` ensures quality maintains baseline importance regardless of speed weighting.

### What file defines the test prompts and scoring criteria?

The [`llmfit-core/data/benchmarks.yaml`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/data/benchmarks.yaml) file contains the declarative definitions of roles, prompts, and scoring rules. This file is embedded into the binary at compile time and loaded via `default_quality_config()`, allowing the benchmark suite to travel with the compiled artifact without external file dependencies.

### How are results aggregated when a role contains multiple tests?

The `run_all_tests` function (lines 108-140 in [`quality.rs`](https://github.com/AlexsJones/llmfit/blob/main/quality.rs)) collects individual test scores into `RoleScore` structs, calculating arithmetic means for quality, speed, and composite metrics across all tests assigned to that role. These role averages then populate the final `ModelQualityResult` for comprehensive model comparison.