How llmfit's Benchmark and Share Feature Generates tok/s Measurements

llmfit calculates tokens-per-second (tok/s) by combining local inference benchmarking with a crowdsourced community cache, allowing real hardware performance to override theoretical estimates.

The llmfit ecosystem, developed in the AlexsJones/llmfit repository, solves a critical problem in LLM deployment: theoretical performance metrics rarely match reality. Its benchmark and share feature closes this gap by measuring actual generation speed on your hardware, uploading the results to a community dataset, and using those real-world measurements to improve model selection accuracy.

How tok/s Is Calculated During Benchmarking

The bench.rs module in llmfit-core performs the heavy lifting of turning inference calls into measurable tok/s values.

Provider-Specific Timing Strategies

When benchmarking against Ollama, llmfit reads the eval_duration field (in nanoseconds) directly from the API response:

// llmfit-core/src/bench.rs lines 82-90
let tps = if let Some(eval_duration) = response.eval_duration {
    // Ollama provides precise token-level timing
    response.eval_count as f64 / (eval_duration as f64 / 1e9)
} else {
    // Fallback to wall-clock timing for providers without token timers
    output_tokens as f64 / total_wall_secs
};

For providers like vLLM or MLX that lack granular token timers, llmfit falls back to wall-clock timing: total output tokens divided by elapsed seconds. This ensures consistent tok/s reporting across all supported backends.

Aggregating Multiple Runs

Benchmarking isn't a single-shot measurement. The BENCH_PROMPTS constant defines multiple inference calls, each producing a BenchRun struct. The BenchSummary::from_runs method (lines 47-71) aggregates these into a reliable average:

// From bench.rs - aggregation logic
let summary = BenchSummary::from_runs(&runs);
// summary.avg_tps contains the mean tok/s across all runs

This statistical approach smooths out noise from thermal throttling, background processes, and warmup effects.

The Share Pipeline: From Local Measurement to Community Data

The share.rs module transforms private benchmark results into public community knowledge.

Building the Submission Payload

The build_submission helper (lines 61-70) packages the benchmark summary into a ResultPayload struct:

// llmfit-core/src/share.rs
let payload = ResultPayload {
    avg_tps: round2(summary.avg_tps),  // Rounded to 2 decimal places
    // ... hardware specs, model config, etc.
};

The ResultPayload struct (lines 92-96) explicitly stores avg_tps as the canonical tok/s measurement for this hardware-model combination.

GitHub API Integration

Submitted results travel through the GitHub REST API, merging into the repository's data/community/ directory. A build.rs script embeds these into community_benchmarks.json, compiling the entire dataset into the binary at build time. This design means every llmfit installation carries the full community knowledge base—no network queries needed during analysis.

Consuming Community tok/s in Model Fitting

The real power emerges when benchmarked tok/s values feed back into model selection.

The MeasuredTpsIndex Cache

llmfit-core/src/benchmarks.rs builds an efficient lookup structure. The MeasuredTpsIndex::from_rows method (lines 64-77) extracts tok_s_out from each community record, indexing by (model, quantization) tuple and sorting values for fast median calculation.

Hardware-Aware Lookup

The measured_tps_for function (lines 25-31) queries this cache:

use llmfit_core::benchmarks::measured_tps_for;

if let Some(measured) = measured_tps_for(&system_specs, "llama3.1:8b", "q4_k_m") {
    // measured.tok_s contains the community median for matching hardware
}

Critical matching criteria include CPU architecture, GPU model, and memory configuration—ensuring a MacBook Pro M3 doesn't inherit measurements from an RTX 4090 workstation.

Overriding Theoretical Estimates

In fit.rs (lines 62-66), the ModelFit struct contains:

pub measured_tps: Option<MeasuredTps>,

When measured_tps_for returns a value, the fit engine prefers this real measurement over formula-based estimates. The UI renders this with a "✓" badge (lines 900-904 in llmfit-tui/src/tui_ui.rs), signaling verified performance data.

Complete Workflow Example

Here's how the components orchestrate from command to usable data:

# 1. Benchmark and share

llmfit bench llama3.1:8b --share

Internal pipeline:

Step	File	Action
Inference	bench.rs	Run BENCH_PROMPTS against Ollama/vLLM/MLX
Calculate	bench.rs	tps = eval_count / (eval_duration / 1e9)
Aggregate	bench.rs	BenchSummary::from_runs(&runs) → avg_tps
Package	share.rs	build_submission() → ResultPayload.avg_tps
Upload	share.rs	GitHub REST API → data/community/
Index	benchmarks.rs	MeasuredTpsIndex::from_rows() at build time
Query	benchmarks.rs	measured_tps_for(&specs, model, quant)
Display	tui_ui.rs	"✓ Measured: 127.4 tok/s"

Practical Code Examples

Retrieve Measured tok/s Programmatically

use llmfit_core::benchmarks::measured_tps_for;
use llmfit_core::hardware::SystemSpecs;

fn check_community_speed(model: &str, quant: &str) {
    let specs = SystemSpecs::detect();
    
    match measured_tps_for(&specs, model, quant) {
        Some(measured) => {
            println!("Median community tok/s: {:.1}", measured.tok_s);
            println!("Based on {} benchmark submissions", measured.sample_count);
        }
        None => println!("No community data for this configuration"),
    }
}

Inspect Within a Fit Analysis

use llmfit_core::fit::ModelFit;

let fit = ModelFit::analyze(&model_config, &detected_specs);

if let Some(mt) = &fit.measured_tps {
    println!("⚡ Verified performance: {:.1} tok/s", mt.tok_s);
    println!("   (override: theoretical estimate was {:.1} tok/s)", 
             fit.estimated_tps);
} else {
    println!("ℹ Using estimated tok/s: {:.1}", fit.estimated_tps);
}

Key Implementation Files

File	Responsibility	Critical Lines
llmfit-core/src/bench.rs	Inference execution, tok/s calculation	82-90 (Ollama timing), 47-71 (aggregation)
llmfit-core/src/share.rs	Payload construction, GitHub upload	61-70 (build_submission), 92-96 (ResultPayload)
llmfit-core/src/benchmarks.rs	Community cache indexing, lookup	25-31 (measured_tps_for), 64-77 (index building)
llmfit-core/src/fit.rs	Integration into model scoring	62-66 (ModelFit.measured_tps)
llmfit-tui/src/tui_ui.rs	Visual indicator of measured data	900-904 (badge rendering)

Summary

• Benchmarking (bench.rs) produces tok/s through provider-aware timing—nanosecond-precision for Ollama, wall-clock fallback for others
• Sharing (share.rs) uploads avg_tps to a GitHub-backed community dataset compiled into every binary
• Caching (benchmarks.rs) indexes measurements by hardware-model-quantization for median-based retrieval
• Fitting (fit.rs) prefers measured tok/s over estimates, surfaced in the TUI with a verification badge
• Result: Users see real-world performance numbers fromhardware configurations matching their own, not marketing specifications

Frequently Asked Questions

How accurate is llmfit's tok/s measurement compared to manual timing?

The Ollama path achieves nanosecond precision using eval_duration from the provider's native token counter. The fallback wall-clock method introduces potential variance from OS scheduling and thermal management, which llmfit mitigates by averaging multiple runs through BenchSummary::from_runs. For maximum accuracy on custom providers, implement token-level timing in your inference backend.

Can I use community tok/s data without running benchmarks myself?

Yes. The measured_tps_for function automatically queries the embedded community cache when analyzing model fit. Installed binaries include all historical submissions, so no network access is required. The lookup matches against your detected SystemSpecs, returning only data from comparable hardware configurations.

What happens if my exact hardware-model combination has no community data?

measured_tps_for returns None, and the fit engine falls back to formula-based estimation using quantization parameters and theoretical throughput models. Running llmfit bench <model> --share populates the cache for your configuration, improving accuracy for future users with similar hardware.

How does llmfit prevent benchmark gaming or fraudulent submissions?

Submissions route through GitHub pull requests requiring maintainer approval before merging into data/community/. The build.rs embedding process only includes validated JSON files. Additionally, outlier detection during MeasuredTpsIndex construction flags statistically anomalous tok/s values for manual review.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}