# How to Benchmark LLM Models with llmfit: A Complete Technical Guide

> Benchmark LLM models using llmfit. Learn the three-stage pipeline for measuring performance including tokens per second and time to first token. A complete technical guide.

- Repository: [Alex Jones/llmfit](https://github.com/AlexsJones/llmfit)
- Tags: how-to-guide
- Published: 2026-08-19

---

**Benchmarking LLMs in llmfit follows a three-stage pipeline—live provider detection via [`src/providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/src/providers.rs), benchmark execution through `bench::run_benchmark` in [`src/bench.rs`](https://github.com/AlexsJones/llmfit/blob/main/src/bench.rs), and optional community sharing via [`src/share.rs`](https://github.com/AlexsJones/llmfit/blob/main/src/share.rs)—measuring tokens-per-second and time-to-first-token across three real inference passes.**

llmfit is a Rust-based framework for evaluating Large Language Model performance across heterogeneous hardware. When you need to quantify real-world inference latency and throughput, **benchmark LLM models with llmfit** using its automated provider discovery and standardized measurement protocols that feed into a calibrated community leaderboard.

## How the llmfit Benchmarking Engine Works

### Live Provider Detection and Registration

The benchmarking process begins in [`llmfit-core/src/providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/providers.rs). This module implements a watcher that detects running backends—including **llama.cpp**, **Ollama**, **vLLM**, and **MLX**—and automatically registers their endpoints (e.g., `http://127.0.0.1:8080`). When a model server is active, the provider registers it as an available target for the benchmark runner, eliminating manual configuration.

### The Three-Pass Measurement Protocol

The core measurement logic resides in [`llmfit-core/src/bench.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/bench.rs). The **`bench::run_benchmark`** function executes three independent inference passes against the live endpoint to ensure statistical stability. During execution, llmfit captures two primary metrics:

- **Tokens-per-second**: Sustained generation throughput after the initial response
- **Time-to-first-token (TTFT)**: End-to-end latency from request dispatch to the first response byte

Results are immediately persisted to the local bench store at `~/.config/llmfit` via the `share` module ([`src/share.rs`](https://github.com/AlexsJones/llmfit/blob/main/src/share.rs)), ensuring data survives process termination even when community sharing is disabled.

### Community Leaderboard Integration

After benchmark completion, [`llmfit-core/src/share.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/share.rs) handles optional result submission. The module creates GitHub Pull Requests automatically using either the **GitHub device-flow** or a **`GITHUB_TOKEN`** environment variable, requiring no local `gh` CLI installation. Accepted results merge into the remote index managed by [`llmfit-core/src/benchmarks.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/benchmarks.rs), which calibrates performance estimates for other users with identical hardware configurations.

## Benchmarking via the Terminal UI

The TUI orchestrates the workflow through the **`App::open_benchmarks`** method in [`llmfit-tui/src/tui_app.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-tui/src/tui_app.rs).

Start your model server with GPU offload enabled:

```bash
llama-server -m ~/.cache/llmfit/models/Qwen3.5-0.8B-Q8_0.gguf \
             --port 8080 -ngl 99

```

Launch llmfit and refresh the provider list:

```bash
llmfit  # Opens TUI

r       # Refresh installed models (auto-detects the running server)

```

Execute the benchmark using TUI controls:

- Press **b** to open the **Benchmark this model** popup
- Press **Enter** to start the three-pass benchmark in a background thread
- Press **Space** to toggle "share results as a PR" (optional)
- Press **Esc** to return to browsing while the benchmark continues

View results by pressing **b** again to open the leaderboard; shared results appear marked as "you (shared)".

## Benchmarking via the Command Line

The CLI supports automation through the `llmfit bench` subcommand defined in [`docs/cli.md`](https://github.com/AlexsJones/llmfit/blob/main/docs/cli.md).

Benchmark a specific model with auto-detected provider:

```bash
llmfit bench -m qwen3.5-0.8b

```

Benchmark all installed models and contribute to the community leaderboard:

```bash
llmfit bench --all --share

```

Export machine-readable results for CI/CD integration:

```bash
llmfit bench -m qwen3.5-0.8b --json

```

**Available flags:**
- **`--share`**: Automatically create a PR to the community leaderboard
- **`--all`**: Run benchmarks against every installed model sequentially
- **`--dry-run`**: Simulate execution without performing inference
- **`--json`**: Output results in JSON format for parsing by external tools

## Summary

- **Provider Detection**: The [`src/providers.rs`](https://github.com/AlexsJones/llmfit/blob/main/src/providers.rs) module automatically discovers running LLM backends (llama.cpp, Ollama, vLLM, MLX) and registers their endpoints.
- **Measurement Protocol**: The `bench::run_benchmark` function in [`src/bench.rs`](https://github.com/AlexsJones/llmfit/blob/main/src/bench.rs) performs three real inference passes, capturing **tokens-per-second** and **time-to-first-token**.
- **Local Persistence**: Results are stored in `~/.config/llmfit` via [`src/share.rs`](https://github.com/AlexsJones/llmfit/blob/main/src/share.rs) before optional community submission.
- **Community Sharing**: The `share` module creates GitHub PRs using device-flow or `GITHUB_TOKEN`, populating the remote index in [`src/benchmarks.rs`](https://github.com/AlexsJones/llmfit/blob/main/src/benchmarks.rs) for hardware-specific calibration.
- **Dual Interfaces**: Both the TUI (`App::open_benchmarks` in [`src/tui_app.rs`](https://github.com/AlexsJones/llmfit/blob/main/src/tui_app.rs)) and CLI (`llmfit bench` with `--share`, `--json` flags) support complete benchmarking workflows.

## Frequently Asked Questions

### What metrics does llmfit capture during benchmarking?

llmfit measures **tokens-per-second** for sustained throughput and **time-to-first-token (TTFT)** for end-to-end latency. The `bench::run_benchmark` function in [`llmfit-core/src/bench.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/bench.rs) calculates these across three real inference passes to ensure statistical significance and account for caching effects.

### Do I need the GitHub CLI installed to share benchmark results?

No. The `share` module in [`llmfit-core/src/share.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/share.rs) implements the GitHub device-flow directly or accepts a `GITHUB_TOKEN` environment variable. It creates Pull Requests programmatically without requiring the `gh` CLI binary to be present on the system.

### Where are benchmark results stored before sharing?

Results are written to the local bench store at `~/.config/llmfit` via the `share` module. This local persistence ensures your data survives application restarts even if you choose not to submit results to the community leaderboard hosted in [`llmfit-core/src/benchmarks.rs`](https://github.com/AlexsJones/llmfit/blob/main/llmfit-core/src/benchmarks.rs).

### Can I benchmark multiple models simultaneously from the command line?

You can benchmark all installed models sequentially using the `llmfit bench --all` flag. This runs the three-pass protocol against every detected model in series. Combine with `--share` to batch-submit results to the community index, or use `--json` to capture structured output for each model.