How to Benchmark Inference Performance Using llama-bench in llama.cpp

llama-bench is a dedicated performance-testing utility shipped with llama.cpp that automates the measurement of prompt-processing speed, token-generation throughput, and combined latency for any GGUF model.

Benchmarking inference performance is essential when optimizing large language model deployments. The llama-bench tool, included in the ggml-org/llama.cpp repository, provides a standardized way to measure raw model evaluation throughput across different hardware configurations and runtime parameters.

What is llama-bench?

llama-bench is a command-line utility located in tools/llama-bench/llama-bench.cpp. Unlike the interactive llama-cli or server modes, this tool is designed specifically for automated performance testing. It executes standardized benchmark loops and reports statistics in multiple formats including Markdown, CSV, JSON, and SQL.

The tool deliberately excludes tokenization and sampling time from its measurements. According to the source documentation, this design choice ensures that reported tokens-per-second (t/s) values reflect only the raw model evaluation throughput—the metric most critical for performance tuning work.

Architecture and Key Components

Understanding the internal structure of llama-bench helps interpret its output and debug configuration issues.

Command-Line Parser

The argument processing logic resides in parse_cmd_params (lines 500-540) and get_cmd_params_instances in tools/llama-bench/llama-bench.cpp. This system handles repeatable values, ranges, and defaults, expanding every combination of supplied parameters into concrete test instances.

Model Loader

For each test instance, the tool loads a llama_model using to_llama_mparams and to_llama_cparams (lines 778-829). These functions translate CLI options into the llama_model_params and llama_context_params structs used by the core library, handling GPU offload layers (-ngl), tensor splits, and backend selection.

Benchmark Executor

The actual measurement occurs in loops around lines 2100-2250, which call run_prompt, run_generation, or both. The tool uses get_time_ns() for high-resolution wall-clock timing and computes statistics via helper functions avg and stdev (lines 96-113).

Installation and Prerequisites

llama-bench compiles alongside the main llama.cpp project. Ensure you have:

• A C++ compiler supporting C++11 or later
• CMake or the provided Makefile
• The appropriate backend libraries for your hardware (CUDA, Vulkan, Metal, etc.)

Build the specific target:

make llama-bench

Or using CMake:

cmake -B build
cmake --build build --target llama-bench

The binary will be available at ./llama-bench or ./build/bin/llama-bench depending on your build system.

Command-Line Options and Usage Patterns

The tool accepts numerous flags to control the benchmark environment. Key parameters include:

• -m, --model: Path to the GGUF model file (repeatable for multiple models)
• -p, --n-prompt: Number of tokens to process in prompt evaluation (0 to disable)
• -n, --n-gen: Number of tokens to generate (0 to disable)
• -b, --batch-size: Batch size for prompt processing (repeatable for sweeps)
• -t, --threads: Number of CPU threads to use
• -ngl, --n-gpu-layers: Number of layers to offload to GPU
• -r, --repetitions: Number of times to repeat each test for statistical significance
• -o, --output: Output format (md, csv, json, jsonl, sql)

Practical Benchmarking Examples

Compare Two Models on Fixed Prompt Length

Test how different model sizes handle the same prompt workload:

./llama-bench \
    -m models/7B/ggml-model-q4_0.gguf \
    -m models/13B/ggml-model-q4_0.gguf \
    -p 512 -n 0 \
    -o md

Sweep Batch Size for Prompt Processing

Identify optimal batch sizes for your hardware by testing multiple configurations:

./llama-bench \
    -m models/7B/ggml-model-q4_0.gguf \
    -p 1024 -n 0 \
    -b 128,256,512,1024 \
    -t 1,2,4,8 \
    -r 3 \
    -o csv

Benchmark GPU Offload Impact

Measure performance gains from layer offloading on CUDA-enabled systems:

./llama-bench \
    -m models/7B/ggml-model-q4_0.gguf \
    -p 0 -n 128 \
    -ngl 0,10,20,30 \
    -o json

Export Results to SQLite Database

Store benchmark results for longitudinal analysis:

./llama-bench \
    -m models/7B/ggml-model-q4_0.gguf \
    -p 512 -n 0 \
    -o sql | sqlite3 bench_results.db

Understanding the Output Formats

llama-bench supports multiple output formats via the -o flag:

• Markdown (md): Default human-readable table format suitable for documentation and GitHub issues
• CSV: Comma-separated values for spreadsheet analysis
• JSON: Single JSON object containing all test results
• JSONL: One JSON object per line for streaming processing
• SQL: INSERT statements for direct database import

The tool reports both nanoseconds and tokens-per-second (t/s) with average and standard deviation values when multiple repetitions are requested.

Summary

• llama-bench is the official benchmarking utility in tools/llama-bench/llama-bench.cpp for measuring llama.cpp inference performance
• It measures raw evaluation throughput excluding tokenization and sampling overhead
• The tool supports parameter sweeps via repeatable flags and ranges, testing every combination automatically
• Three benchmark modes: prompt processing (-p), token generation (-n), or combined (-pg)
• Multiple output formats (Markdown, CSV, JSON, SQL) facilitate integration with analysis pipelines
• Key implementation files include parse_cmd_params (lines 500-540), the benchmark executor (lines 2100-2250), and statistical helpers avg/stdev (lines 96-113)

Frequently Asked Questions

What hardware metrics does llama-bench actually measure?

llama-bench measures wall-clock time using high-resolution nanosecond timers (get_time_ns()) during the core model evaluation loops. It specifically tracks how long the forward pass takes for prompt tokens (batch processing) and generation tokens (autoregressive sampling). It does not measure memory bandwidth, GPU utilization percentages, or power consumption—only the end-to-end throughput in tokens per second.

Why are tokenization and sampling excluded from the benchmark results?

According to the source documentation in tools/llama-bench/README.md, tokenization and sampling are excluded because they are not part of the raw model evaluation that constitutes the computational bottleneck in LLM inference. By isolating the forward pass timing, developers can accurately compare backend optimizations (CUDA vs. Vulkan vs. CPU), quantization effects, or layer offloading strategies without noise from tokenizer implementation details or sampling algorithm choices.

How do I benchmark multiple configurations without running separate commands?

The parse_cmd_params function (lines 500-540) and get_cmd_params_instances support parameter expansion via comma-separated values. When you pass multiple values to flags like -b (batch size), -t (threads), or -ngl (GPU layers), the tool automatically generates the cartesian product of all combinations and runs them sequentially. For example, -b 128,256 -t 4,8 executes four distinct benchmarks: (128,4), (128,8), (256,4), and (256,8).

Can I integrate llama-bench into automated CI/CD pipelines?

Yes, the -o json and -o csv output formats are designed for machine parsing. The JSON output provides a structured object containing all test parameters and results including averages and standard deviations, while the CSV format produces flat tables suitable for spreadsheet ingestion. For database integration, the -o sql format emits standard INSERT statements that can be piped directly into SQLite or other SQL databases, making it straightforward to track performance regressions across commits or hardware configurations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}