# How llmfit's Benchmark and Share Feature Generates tok/s Measurements

> llmfit measures tok/s by combining local benchmarking with community data for accurate hardware performance insights. Discover how real-world results are generated.

- Repository: [Alex Jones/llmfit](https://github.com/AlexsJones/llmfit)
- Tags: internals
- Published: 2026-08-20

---

**llmfit calculates tokens-per-second (tok/s) by combining local inference benchmarking with a crowdsourced community cache, allowing real hardware performance to override theoretical estimates.**

The `llmfit` ecosystem, developed in the AlexsJones/llmfit repository, solves a critical problem in LLM deployment: theoretical performance metrics rarely match reality. Its **benchmark and share feature** closes this gap by measuring actual generation speed on your hardware, uploading the results to a community dataset, and using those real-world measurements to improve model selection accuracy.

## How tok/s Is Calculated During Benchmarking

The [`bench.rs`](https://github.com/AlexsJones/llmfit/blob/main/bench.rs) module in `llmfit-core` performs the heavy lifting of turning inference calls into measurable tok/s values.

### Provider-Specific Timing Strategies

When benchmarking against **Ollama**, llmfit reads the `eval_duration` field (in nanoseconds) directly from the API response:

```rust
// llmfit-core/src/bench.rs lines 82-90
let tps = if let Some(eval_duration) = response.eval_duration {
    // Ollama provides precise token-level timing
    response.eval_count as f64 / (eval_duration as f64 / 1e9)
} else {
    // Fallback to wall-clock timing for providers without token timers
    output_tokens as f64 / total_wall_secs
};

```

For providers like **vLLM** or **MLX** that lack granular token timers, llmfit falls back to **wall-clock timing**: total output tokens divided by elapsed seconds. This ensures consistent tok/s reporting across all supported backends.

### Aggregating Multiple Runs

Benchmarking isn't a single-shot measurement. The `BENCH_PROMPTS` constant defines multiple inference calls, each producing a `BenchRun` struct. The `BenchSummary::from_runs` method (lines 47-71) aggregates these into a reliable average:

```rust
// From bench.rs - aggregation logic
let summary = BenchSummary::from_runs(&runs);
// summary.avg_tps contains the mean tok/s across all runs

```

This statistical approach smooths out noise from thermal throttling, background processes, and warmup effects.

## The Share Pipeline: From Local Measurement to Community Data

The [`share.rs`](https://github.com/AlexsJones/llmfit/blob/main/share.rs) module transforms private benchmark results into public community knowledge.

### Building the Submission Payload

The `build_submission` helper (lines 61-70) packages the benchmark summary into a `ResultPayload` struct:

```rust
// llmfit-core/src/share.rs
let payload = ResultPayload {
    avg_tps: round2(summary.avg_tps),  // Rounded to 2 decimal places
    // ... hardware specs, model config, etc.
};

```

The `ResultPayload` struct (lines 92-96) explicitly stores `avg_tps` as the canonical tok/s measurement for this hardware-model combination.

### GitHub API Integration

Submitted results travel through the GitHub REST API, merging into the repository's `data/community/` directory. A [`build.rs`](https://github.com/AlexsJones/llmfit/blob/main/build.rs) script embeds these into [`community_benchmarks.json`](https://github.com/AlexsJones/llmfit/blob/main/community_benchmarks.json), compiling the entire dataset into the binary at build time. This design means **every llmfit installation carries the full community knowledge base**—no network queries needed during analysis.

## Consuming Community tok/s in Model Fitting

The real power emerges when benchmarked tok/s values feed back into model selection.

### The MeasuredTpsIndex Cache

[`llmfit-core/src/benchmarks.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/benchmarks.rs) builds an efficient lookup structure. The `MeasuredTpsIndex::from_rows` method (lines 64-77) extracts `tok_s_out` from each community record, indexing by `(model, quantization)` tuple and sorting values for fast median calculation.

### Hardware-Aware Lookup

The `measured_tps_for` function (lines 25-31) queries this cache:

```rust
use llmfit_core::benchmarks::measured_tps_for;

if let Some(measured) = measured_tps_for(&system_specs, "llama3.1:8b", "q4_k_m") {
    // measured.tok_s contains the community median for matching hardware
}

```

Critical matching criteria include CPU architecture, GPU model, and memory configuration—ensuring a MacBook Pro M3 doesn't inherit measurements from an RTX 4090 workstation.

### Overriding Theoretical Estimates

In [`fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/fit.rs) (lines 62-66), the `ModelFit` struct contains:

```rust
pub measured_tps: Option<MeasuredTps>,

```

When `measured_tps_for` returns a value, the fit engine **prefers this real measurement** over formula-based estimates. The UI renders this with a "✓" badge (lines 900-904 in [`llmfit-tui/src/tui_ui.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/tui_ui.rs)), signaling verified performance data.

## Complete Workflow Example

Here's how the components orchestrate from command to usable data:

```bash

# 1. Benchmark and share

llmfit bench llama3.1:8b --share

```

**Internal pipeline:**

| Step | File | Action |
|------|------|--------|
| Inference | [`bench.rs`](https://github.com/AlexsJones/llmfit/blob/main/bench.rs) | Run `BENCH_PROMPTS` against Ollama/vLLM/MLX |
| Calculate | [`bench.rs`](https://github.com/AlexsJones/llmfit/blob/main/bench.rs) | `tps = eval_count / (eval_duration / 1e9)` |
| Aggregate | [`bench.rs`](https://github.com/AlexsJones/llmfit/blob/main/bench.rs) | `BenchSummary::from_runs(&runs)` → `avg_tps` |
| Package | [`share.rs`](https://github.com/AlexsJones/llmfit/blob/main/share.rs) | `build_submission()` → `ResultPayload.avg_tps` |
| Upload | [`share.rs`](https://github.com/AlexsJones/llmfit/blob/main/share.rs) | GitHub REST API → `data/community/` |
| Index | [`benchmarks.rs`](https://github.com/AlexsJones/llmfit/blob/main/benchmarks.rs) | `MeasuredTpsIndex::from_rows()` at build time |
| Query | [`benchmarks.rs`](https://github.com/AlexsJones/llmfit/blob/main/benchmarks.rs) | `measured_tps_for(&specs, model, quant)` |
| Display | [`tui_ui.rs`](https://github.com/AlexsJones/llmfit/blob/main/tui_ui.rs) | "✓ Measured: 127.4 tok/s" |

## Practical Code Examples

### Retrieve Measured tok/s Programmatically

```rust
use llmfit_core::benchmarks::measured_tps_for;
use llmfit_core::hardware::SystemSpecs;

fn check_community_speed(model: &str, quant: &str) {
    let specs = SystemSpecs::detect();
    
    match measured_tps_for(&specs, model, quant) {
        Some(measured) => {
            println!("Median community tok/s: {:.1}", measured.tok_s);
            println!("Based on {} benchmark submissions", measured.sample_count);
        }
        None => println!("No community data for this configuration"),
    }
}

```

### Inspect Within a Fit Analysis

```rust
use llmfit_core::fit::ModelFit;

let fit = ModelFit::analyze(&model_config, &detected_specs);

if let Some(mt) = &fit.measured_tps {
    println!("⚡ Verified performance: {:.1} tok/s", mt.tok_s);
    println!("   (override: theoretical estimate was {:.1} tok/s)", 
             fit.estimated_tps);
} else {
    println!("ℹ Using estimated tok/s: {:.1}", fit.estimated_tps);
}

```

## Key Implementation Files

| File | Responsibility | Critical Lines |
|------|---------------|--------------|
| [`llmfit-core/src/bench.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/bench.rs) | Inference execution, tok/s calculation | 82-90 (Ollama timing), 47-71 (aggregation) |
| [`llmfit-core/src/share.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/share.rs) | Payload construction, GitHub upload | 61-70 (`build_submission`), 92-96 (`ResultPayload`) |
| [`llmfit-core/src/benchmarks.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/benchmarks.rs) | Community cache indexing, lookup | 25-31 (`measured_tps_for`), 64-77 (index building) |
| [`llmfit-core/src/fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/fit.rs) | Integration into model scoring | 62-66 (`ModelFit.measured_tps`) |
| [`llmfit-tui/src/tui_ui.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/tui_ui.rs) | Visual indicator of measured data | 900-904 (badge rendering) |

## Summary

- **Benchmarking** ([`bench.rs`](https://github.com/AlexsJones/llmfit/blob/main/bench.rs)) produces tok/s through provider-aware timing—nanosecond-precision for Ollama, wall-clock fallback for others
- **Sharing** ([`share.rs`](https://github.com/AlexsJones/llmfit/blob/main/share.rs)) uploads `avg_tps` to a GitHub-backed community dataset compiled into every binary
- **Caching** ([`benchmarks.rs`](https://github.com/AlexsJones/llmfit/blob/main/benchmarks.rs)) indexes measurements by hardware-model-quantization for median-based retrieval
- **Fitting** ([`fit.rs`](https://github.com/AlexsJones/llmfit/blob/main/fit.rs)) prefers measured tok/s over estimates, surfaced in the TUI with a verification badge
- **Result**: Users see real-world performance numbers fromhardware configurations matching their own, not marketing specifications

## Frequently Asked Questions

### How accurate is llmfit's tok/s measurement compared to manual timing?

The Ollama path achieves nanosecond precision using `eval_duration` from the provider's native token counter. The fallback wall-clock method introduces potential variance from OS scheduling and thermal management, which `llmfit` mitigates by averaging multiple runs through `BenchSummary::from_runs`. For maximum accuracy on custom providers, implement token-level timing in your inference backend.

### Can I use community tok/s data without running benchmarks myself?

Yes. The `measured_tps_for` function automatically queries the embedded community cache when analyzing model fit. Installed binaries include all historical submissions, so no network access is required. The lookup matches against your detected `SystemSpecs`, returning only data from comparable hardware configurations.

### What happens if my exact hardware-model combination has no community data?

`measured_tps_for` returns `None`, and the fit engine falls back to formula-based estimation using quantization parameters and theoretical throughput models. Running `llmfit bench <model> --share` populates the cache for your configuration, improving accuracy for future users with similar hardware.

### How does llmfit prevent benchmark gaming or fraudulent submissions?

Submissions route through GitHub pull requests requiring maintainer approval before merging into `data/community/`. The [`build.rs`](https://github.com/AlexsJones/llmfit/blob/main/build.rs) embedding process only includes validated JSON files. Additionally, outlier detection during `MeasuredTpsIndex` construction flags statistically anomalous tok/s values for manual review.