How to Debug Slow Token Generation Performance Issues in llama.cpp

Enable params.no_perf = false and call common_perf_print() after generation to identify whether bottlenecks exist in prompt evaluation, token decoding, sampling, or KV-cache management.

Token generation performance in llama.cpp depends on a tight loop that repeatedly invokes llama_decode on batched prompts. When throughput drops below expected levels—whether measured in tokens per second (tok/s) or latency per token—the root cause typically lies in suboptimal batch sizing, disabled performance instrumentation, or hardware misconfiguration. This guide walks through the diagnostic workflow using the actual source implementation to isolate and resolve these bottlenecks.

Understanding the Token Generation Pipeline

The core generation loop resides in src/llama-context.cpp, where llama_decode processes tokens in batches sized by llama_n_batch(ctx)【src/llama-context.cpp#L3295-L3301】. Each iteration performs matrix multiplications through GGML kernels, updates the KV-cache, and runs the sampler chain. Performance degrades when any single stage dominates execution time, necessitating targeted instrumentation to pinpoint the culprit.

Critical Configuration Checks

Verify Performance Timing is Enabled

The most common oversight is disabling the lightweight timing infrastructure. In include/llama.h, the llama_context_params struct contains the no_perf boolean【include/llama.h#L62-L66】. When set to true, the library skips all timing collection, making it impossible to determine whether slowdowns occur during prompt evaluation, token decoding, or sampling.

struct llama_context_params cparams = llama_context_default_params();
cparams.no_perf = false;  // Essential for diagnostics
cparams.n_batch = 64;     // Adjust based on your hardware

Optimize Batch Size to Prevent KV-Cache Thrashing

The n_batch parameter controls how many tokens enter llama_decode simultaneously. A value too small (e.g., 1) forces excessive kernel launches; too large causes KV-cache overflow, triggering fallback logic in the server implementation that repeatedly halves the batch size until it reaches 1【src/tools/server/server-context.cpp#L799-L807】.

Monitor the effective batch size during runtime:

// Insert debug logging in server-context.cpp or your main loop
fprintf(stderr, "[perf] n_batch=%d, n_ctx=%d\n", 
        llama_n_batch(ctx), llama_n_ctx(ctx));

If you observe n_batch dropping from your configured value (e.g., 512) down to 1, the KV-cache cannot accommodate the request. Increase the context size (n_ctx) or reduce the batch size to match available memory.

Tune CPU Thread Parallelism

GGML kernels utilize multi-threading via params.n_threads and params.n_threads_batch. Setting these below your physical core count leaves compute idle; exceeding it causes oversubscription and cache thrashing. For most modern CPUs, set:

cparams.n_threads = std::thread::hardware_concurrency();
cparams.n_threads_batch = cparams.n_threads;  // Or separate pool for batch processing

Enable Backend Offloading for GPU Acceleration

Hardware acceleration dramatically reduces per-token latency, but requires explicit configuration. The offload_kqv flag moves the KV-cache to GPU memory, while op_offload transfers tensor operations【src/llama-kv-cache.cpp#L344-L351】.

cparams.offload_kqv = true;   // Move KV cache to GPU/Metal/Vulkan
cparams.op_offload = true;    // Offload host operations

Misconfiguration (e.g., requesting offloading without available VRAM) silently falls back to CPU execution, adding memory copy overhead without warning.

Monitor KV-Cache Fragmentation

Frequent insert and evict cycles fragment the KV-cache, forcing expensive data copies during defragmentation. The defrag_thold parameter in llama_context_params controls when the runtime compacts the cache. A value of 0.0 disables defragmentation entirely, while values between 0.1 and 0.5 (representing 10-50% fragmentation tolerance) generally optimize throughput.

Profile Sampler Chain Overhead

Complex sampling strategies—combining top-k, top-p, tail-free sampling, and grammar constraints—can become CPU-bound. The sampler chain implementation in src/llama-sampler.cpp supports instrumentation via time_meas tm(chain->t_sample_us, chain->params.no_perf)【src/llama-sampler.cpp#L33-L38】.

Ensure your sampler chain also has performance enabled:

struct llama_sampler_chain_params scparams = llama_sampler_chain_default_params();
scparams.no_perf = false;

struct llama_sampler * chain = llama_sampler_chain_init(scparams);
// Add samplers...

Step-by-Step Debugging Workflow

Follow this sequence to isolate the bottleneck:

1. Enable full performance reporting using common_perf_print() after your generation loop【common/sampling.cpp#L96-L140】:

   // After generation completes
   common_perf_print(ctx, smpl);

   This outputs:

   
   llama_context: prompt eval time =  12.34 ms /  20 tokens (0.62 ms per token, 1614.2 TPS)
   llama_context: eval time =  45.67 ms / 200 runs   (0.23 ms per token, 4335.1 TPS)
   llama_context: unaccounted time = 123.45 ms
   

2. Analyze the breakdown:

   • High prompt eval time: Check n_batch and GPU offloading
   • High eval time: Verify thread counts and backend utilization
   • High unaccounted time: Indicates KV-cache fragmentation or sampler overhead

3. Validate batch stability: If "unaccounted time" dominates, check if n_batch is shrinking due to KV-cache limits as implemented in src/tools/server/server-context.cpp.

4. Isolate sampler costs: If using grammar or complex filters, temporarily replace with simple greedy sampling. If performance improves significantly, optimize the sampler chain parameters.

Practical Code Examples

Enabling Diagnostics in the Simple Example

Modify examples/simple/simple.cpp to capture comprehensive timing data:

struct llama_context_params cparams = llama_context_default_params();
cparams.no_perf = false;            
cparams.n_batch = 64;               
cparams.n_threads = std::thread::hardware_concurrency();

llama_context * ctx = llama_init_from_model(path_model, cparams);

// ... generation logic ...

common_perf_print(ctx, g_smpl);  // Line 96 in common/sampling.cpp

Detecting Batch Size Regression

Add instrumentation to detect when the server reduces batch size automatically:

// In your inference loop or server-context.cpp
int32_t current_batch = llama_n_batch(ctx);
if (current_batch < target_batch_size) {
    fprintf(stderr, "[perf] Batch reduced to %d (target: %d)\n", 
            current_batch, target_batch_size);
}

Measuring Individual Sampler Latency

Build an instrumented sampler chain to measure specific filter overhead:

struct llama_sampler_chain_params scparams = llama_sampler_chain_default_params();
scparams.no_perf = false;

struct llama_sampler * chain = llama_sampler_chain_init(scparams);
llama_sampler_chain_add(chain, llama_sampler_top_k_init(50));
llama_sampler_chain_add(chain, llama_sampler_tail_free_init(0.5f));

g_smpl = common_sampler_init(chain, ctx, nullptr);
common_perf_print(ctx, g_smpl);  // Displays "samplers time = X ms"

Summary

• Always set no_perf = false in both llama_context_params and llama_sampler_chain_params before benchmarking to capture timing data from common_perf_print.
• Monitor n_batch stability; automatic reduction to 1 indicates KV-cache exhaustion requiring either increased context size or reduced batch size.
• Match thread counts to physical core counts using std::thread::hardware_concurrency() to prevent oversubscription.
• Enable GPU offloading via offload_kqv and op_offload when hardware acceleration is available, but verify fallback isn't occurring silently.
• Check "unaccounted time" in performance reports to identify KV-cache fragmentation or sampler bottlenecks that escape standard timing categories.

Frequently Asked Questions

Why is my llama.cpp generation slower than expected despite using a fast GPU?

Disable no_perf and run common_perf_print() to check if time is actually spent in GPU kernels or wasted in CPU fallback code. If the KV-cache cannot fit in VRAM, the runtime falls back to CPU execution in src/llama-kv-cache.cpp, causing silent performance degradation. Also verify that n_batch isn't shrinking to 1 due to context size constraints【src/tools/server/server-context.cpp#L799-L807】.

How do I know if my batch size is causing performance issues?

Insert fprintf(stderr, "n_batch=%d\n", llama_n_batch(ctx)); inside your generation loop. If the value drops below your configured setting, the KV-cache is overflowing and the server is automatically halving the batch until it succeeds. This repeated retry logic destroys throughput. Reduce n_batch to fit within n_ctx or increase the context size.

What does "unaccounted time" mean in the performance print output?

According to the implementation in common/sampling.cpp, unaccounted time represents the delta between total wall-clock time and the sum of measured prompt evaluation, token evaluation, and sampling time【common/sampling.cpp#L96-L140】. High unaccounted time usually indicates KV-cache defragmentation, memory copying between CPU and GPU, or inefficiencies in the sampler chain not captured by individual timers.

Should I disable performance monitoring (no_perf = true) for production deployments?

While no_perf = true eliminates the minimal overhead of timing collection, it strips you of diagnostic capability when performance regressions occur. The timing infrastructure in src/llama-context.cpp and src/llama-sampler.cpp uses lightweight high-resolution counters that add negligible latency【src/llama-sampler.cpp#L33-L38】. Keep performance monitoring enabled in production unless you have proven through profiling that these specific calls appear in your critical path.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}