How LLM Model Quality is Scored in llmfit: A Technical Deep Dive into the Benchmarking Engine

llmfit scores LLM model quality using a regex-based rubric system defined in YAML configuration files, calculating composite metrics from pattern-matching accuracy and inference speed to drive automated model routing decisions.

The llmfit repository implements a sophisticated quality-benchmark subsystem that transforms subjective LLM output evaluation into quantitative metrics. Located primarily in llmfit-core/src/quality.rs, this system enables developers to compare models objectively and automatically select the best performer for specific roles such as coding, research, or general tasks.

The Three-Stage Quality Scoring Pipeline

The quality evaluation process follows a structured pipeline that separates configuration, inference, and scoring concerns. This architecture ensures reproducible benchmarks across different model providers and endpoints.

Stage 1: Loading Role-Based Test Suites

Benchmarking begins with declarative configuration loaded from an embedded YAML file (benchmarks.yaml). This file defines roles (e.g., general, coding, research), each containing one or more quality tests. Every test specifies a prompt and a vector of ScoringRule objects that determine how responses are evaluated.

The core data structures parsing this configuration appear in lines 31-56 of quality.rs:

• QualityConfig: The root configuration container
• RoleDef: Defines a specific use case and its associated tests
• QualityTestDef: Contains the prompt and scoring rules for individual test cases
• ScoringRule: Specifies regex patterns, weights, case sensitivity, and negation logic

Stage 2: Executing Inference and Capturing Responses

Once configured, llmfit dispatches prompts to the target model through provider-specific adapters. The system supports both local Ollama instances and OpenAI-compatible APIs via functions defined in lines 75-92:

• quality_ollama_generate: Interfaces with Ollama's generate endpoint
• quality_openai_chat: Communicates with OpenAI-style chat completion endpoints

Both functions return an InferenceResponse struct containing the raw text output and timing metadata, including tokens-per-second metrics critical for the speed component of the final score.

Stage 3: Evaluating Responses and Calculating Composite Scores

The evaluate_response function drives the actual quality calculation. This implementation iterates over each ScoringRule in the test definition, applying regex pattern matching with configurable case sensitivity. When negate is set to true, matches subtract points rather than adding them.

Raw quality scores are clamped to the range 0-10. Simultaneously, speed scores derive from the tok_per_sec field and normalize using the formula (tok_per_sec / 3).min(10), capping performance at 10x the baseline threshold.

The composite score blends these dimensions using a configurable speed_weight parameter (default 1.0):

composite = (quality * 2.0 + speed_norm * speed_weight) / (2.0 + speed_weight)

This weighting prioritizes response accuracy while incorporating latency characteristics essential for production deployment decisions.

Aggregation and Routing Logic

Individual test results aggregate into per-role averages represented by RoleScore structs. The run_all_tests function (lines 69-78 and 108-140 in quality.rs) handles this aggregation, computing mean scores across all tests within a role before assembling the final ModelQualityResult.

Beyond simple reporting, llmfit leverages these scores for intelligent model selection. The compute_routing function generates a routing matrix mapping each role to its optimal model, while compute_runner_ups identifies secondary options. These matrices enable dynamic request routing based on empirical quality data rather than heuristics.

Implementation Examples

Running a Complete Benchmark

The following example demonstrates benchmarking a local Ollama model against the default quality configuration:

// Load the built-in quality configuration (embedded benchmarks.yaml)
let cfg = llmfit_core::quality::default_quality_config();

// Run a quality benchmark against an Ollama endpoint
let result = llmfit_core::quality::bench_quality_ollama(
    "http://localhost:11434",   // Ollama base URL
    "llama3:8b",                // model name
    &cfg,
    None,                       // optional role filter
).expect("benchmark failed");

// Display results
result.display();               // Human-readable table
result.display_json();          // JSON output

Direct Response Evaluation

For custom testing outside the full benchmark suite, evaluate individual responses against specific scoring rules:

use llmfit_core::quality::{ScoringRule, evaluate_response};

let rules = vec![
    ScoringRule { 
        pattern: "hello".into(), 
        weight: 3, 
        negate: false, 
        case_insensitive: true 
    },
    ScoringRule { 
        pattern: "error".into(), 
        weight: -5, 
        negate: false, 
        case_insensitive: false 
    },
];

let score = evaluate_response("Hello world!", &rules);   // Returns 3.0
println!("Quality score: {}", score);

Generating Routing Matrices

After benchmarking multiple models, compute optimal assignments:

let all_results = vec![result1, result2, result3];
let routing = llmfit_core::quality::compute_routing(&all_results);
let runner_ups = llmfit_core::quality::compute_runner_ups(&all_results);

llmfit_core::quality::display_routing_matrix(&routing, &runner_ups);

Key Source Files

Understanding llmfit's quality scoring requires familiarity with these components:

• llmfit-core/src/quality.rs: Core implementation containing ScoringRule definitions, evaluate_response logic, benchmark runners, and composite calculations
• llmfit-core/data/benchmarks.yaml: Embedded declarative configuration defining test prompts and scoring rubrics
• llmfit-core/src/bench.rs: Shared HTTP response types (ChatCompletionResponse, OllamaGenResponse) consumed by the quality module
• llmfit-core/src/analysis.rs: Integration point connecting quality results with deployment fit analysis

Summary

• Regex-based evaluation: llmfit uses ScoringRule structs with configurable patterns, weights, and negation logic to objectively grade model outputs
• Composite scoring: Final metrics combine quality (0-10) and normalized speed scores using a weighted formula that balances accuracy against performance
• Role-specific benchmarking: The benchmarks.yaml configuration enables targeted evaluation across distinct use cases like coding and research
• Automated routing: Aggregate ModelQualityResult data powers the compute_routing matrix for intelligent model selection
• Provider agnostic: Unified interfaces support both Ollama local instances and OpenAI-compatible APIs through quality_ollama_generate and quality_openai_chat

Frequently Asked Questions

How does llmfit handle case sensitivity in quality scoring?

Each ScoringRule includes a case_insensitive boolean field. When set to true, the regex matcher applies patterns without regard to letter casing. This configuration is defined in the YAML test definitions and parsed into the ScoringRule struct in llmfit-core/src/quality.rs.

Can I adjust the importance of speed versus quality in the final score?

Yes. The composite calculation uses a speed_weight parameter that defaults to 1.0 but is fully configurable. Increasing this value prioritizes inference speed, while decreasing it emphasizes response quality. The formula (quality * 2.0 + speed_norm * speed_weight) / (2.0 + speed_weight) ensures quality maintains baseline importance regardless of speed weighting.

What file defines the test prompts and scoring criteria?

The llmfit-core/data/benchmarks.yaml file contains the declarative definitions of roles, prompts, and scoring rules. This file is embedded into the binary at compile time and loaded via default_quality_config(), allowing the benchmark suite to travel with the compiled artifact without external file dependencies.

How are results aggregated when a role contains multiple tests?

The run_all_tests function (lines 108-140 in quality.rs) collects individual test scores into RoleScore structs, calculating arithmetic means for quality, speed, and composite metrics across all tests assigned to that role. These role averages then populate the final ModelQualityResult for comprehensive model comparison.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}