How to Benchmark LLM Models with llmfit: A Complete Technical Guide

Benchmarking LLMs in llmfit follows a three-stage pipeline—live provider detection via src/providers.rs, benchmark execution through bench::run_benchmark in src/bench.rs, and optional community sharing via src/share.rs—measuring tokens-per-second and time-to-first-token across three real inference passes.

llmfit is a Rust-based framework for evaluating Large Language Model performance across heterogeneous hardware. When you need to quantify real-world inference latency and throughput, benchmark LLM models with llmfit using its automated provider discovery and standardized measurement protocols that feed into a calibrated community leaderboard.

How the llmfit Benchmarking Engine Works

Live Provider Detection and Registration

The benchmarking process begins in llmfit-core/src/providers.rs. This module implements a watcher that detects running backends—including llama.cpp, Ollama, vLLM, and MLX—and automatically registers their endpoints (e.g., http://127.0.0.1:8080). When a model server is active, the provider registers it as an available target for the benchmark runner, eliminating manual configuration.

The Three-Pass Measurement Protocol

The core measurement logic resides in llmfit-core/src/bench.rs. The bench::run_benchmark function executes three independent inference passes against the live endpoint to ensure statistical stability. During execution, llmfit captures two primary metrics:

• Tokens-per-second: Sustained generation throughput after the initial response
• Time-to-first-token (TTFT): End-to-end latency from request dispatch to the first response byte

Results are immediately persisted to the local bench store at ~/.config/llmfit via the share module (src/share.rs), ensuring data survives process termination even when community sharing is disabled.

Community Leaderboard Integration

After benchmark completion, llmfit-core/src/share.rs handles optional result submission. The module creates GitHub Pull Requests automatically using either the GitHub device-flow or a GITHUB_TOKEN environment variable, requiring no local gh CLI installation. Accepted results merge into the remote index managed by llmfit-core/src/benchmarks.rs, which calibrates performance estimates for other users with identical hardware configurations.

Benchmarking via the Terminal UI

The TUI orchestrates the workflow through the App::open_benchmarks method in llmfit-tui/src/tui_app.rs.

Start your model server with GPU offload enabled:

llama-server -m ~/.cache/llmfit/models/Qwen3.5-0.8B-Q8_0.gguf \
             --port 8080 -ngl 99

Launch llmfit and refresh the provider list:

llmfit  # Opens TUI

r       # Refresh installed models (auto-detects the running server)

Execute the benchmark using TUI controls:

• Press b to open the Benchmark this model popup
• Press Enter to start the three-pass benchmark in a background thread
• Press Space to toggle "share results as a PR" (optional)
• Press Esc to return to browsing while the benchmark continues

View results by pressing b again to open the leaderboard; shared results appear marked as "you (shared)".

Benchmarking via the Command Line

The CLI supports automation through the llmfit bench subcommand defined in docs/cli.md.

Benchmark a specific model with auto-detected provider:

llmfit bench -m qwen3.5-0.8b

Benchmark all installed models and contribute to the community leaderboard:

llmfit bench --all --share

Export machine-readable results for CI/CD integration:

llmfit bench -m qwen3.5-0.8b --json

Available flags:

• --share: Automatically create a PR to the community leaderboard
• --all: Run benchmarks against every installed model sequentially
• --dry-run: Simulate execution without performing inference
• --json: Output results in JSON format for parsing by external tools

Summary

• Provider Detection: The src/providers.rs module automatically discovers running LLM backends (llama.cpp, Ollama, vLLM, MLX) and registers their endpoints.
• Measurement Protocol: The bench::run_benchmark function in src/bench.rs performs three real inference passes, capturing tokens-per-second and time-to-first-token.
• Local Persistence: Results are stored in ~/.config/llmfit via src/share.rs before optional community submission.
• Community Sharing: The share module creates GitHub PRs using device-flow or GITHUB_TOKEN, populating the remote index in src/benchmarks.rs for hardware-specific calibration.
• Dual Interfaces: Both the TUI (App::open_benchmarks in src/tui_app.rs) and CLI (llmfit bench with --share, --json flags) support complete benchmarking workflows.

Frequently Asked Questions

What metrics does llmfit capture during benchmarking?

llmfit measures tokens-per-second for sustained throughput and time-to-first-token (TTFT) for end-to-end latency. The bench::run_benchmark function in llmfit-core/src/bench.rs calculates these across three real inference passes to ensure statistical significance and account for caching effects.

Do I need the GitHub CLI installed to share benchmark results?

No. The share module in llmfit-core/src/share.rs implements the GitHub device-flow directly or accepts a GITHUB_TOKEN environment variable. It creates Pull Requests programmatically without requiring the gh CLI binary to be present on the system.

Where are benchmark results stored before sharing?

Results are written to the local bench store at ~/.config/llmfit via the share module. This local persistence ensures your data survives application restarts even if you choose not to submit results to the community leaderboard hosted in llmfit-core/src/benchmarks.rs.

Can I benchmark multiple models simultaneously from the command line?

You can benchmark all installed models sequentially using the llmfit bench --all flag. This runs the three-pass protocol against every detected model in series. Combine with --share to batch-submit results to the community index, or use --json to capture structured output for each model.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}