How to Configure Batch Processing for Improved Inference Throughput in Llama.cpp

Configure batch processing in Llama.cpp by adjusting the logical batch size (-b), physical batch size (-ub), and continuous batching (-cb) flags to amortize KV-cache operations across multiple tokens per llama_decode call.

Llama.cpp groups tokens into batches to maximize hardware utilization during both prompt processing and token generation. By submitting multiple tokens to llama_decode in a single call, the library reduces kernel launch overhead and improves cache locality, significantly increasing throughput on CPU and GPU backends.

Understanding Batch Processing Architecture

Llama.cpp separates inference into two distinct stages: prompt processing (prefill) and token generation. The batch configuration determines how many tokens move through these stages simultaneously. According to the source code in common/common.h, three primary parameters control this behavior.

Logical vs Physical Batch Sizes

The logical batch size (n_batch) defines the maximum number of tokens the decoder may accept in one API call, defaulting to 2048. The physical batch size (n_ubatch) sets the hard ceiling for tokens packed into a single GPU or CPU kernel launch, defaulting to 512.

In common/common.h lines 60-63, these are declared as:

int32_t n_batch = 2048;    // logical batch
int32_t n_ubatch = 512;    // physical batch (u = micro)

When the logical batch exceeds the physical limit, the decoder automatically splits the work into multiple kernel launches. Increasing n_batch allows the server to accumulate more tokens from different requests before decoding, while n_ubatch controls the granularity of individual compute operations.

Continuous Batching

Continuous batching (enabled by default in common/common.h lines 71-72 via common_params::cont_batching) allows the server to add new prompt tokens to the current batch while generation is ongoing. Instead of waiting for a request to complete, the system dynamically fills available slots in the batch, keeping the GPU saturated.

The Batch Processing Pipeline

The core batching logic lives in tools/server/server-context.cpp within the update_slots() function (lines 2106-2113). The loop builds a llama_batch structure and submits it to llama_decode:

int32_t n_batch = llama_n_batch(ctx);               // logical size
common_batch_clear(batch);
for (auto &slot : ready_slots) {
    // add prompt tokens or sampled token to the batch
    common_batch_add(batch, token, pos, {slot.id}, true);
    if (batch.n_tokens >= n_batch) break;          // batch full → decode
}
if (batch.n_tokens) llama_decode(ctx, batch);      // single kernel call

This implementation updates the KV-cache for all sequences simultaneously, eliminating the overhead of per-sequence memory operations.

Configuration Parameters Reference

All batch-related settings are exposed through common_params in common/common.h and mapped to command-line flags:

Parameter	CLI Flag	Default	Location	Purpose
Logical batch size	-b, --batch-size	2048	common/common.h#L60-L63	Max tokens per llama_decode call
Physical batch size	-ub, --ubatch-size	512	common/common.h#L60-L63	Max tokens per kernel launch
Continuous batching	-cb, --cont-batching	Enabled	common/common.h#L71-L72	Dynamic slot filling during generation
Batch thread pool	-tb, --threads-batch	Same as -t	common/common.h#L93-L95	Dedicated threads for prefill (cpuparams_batch)

The public API in include/llama.h lines 329-331 exposes these via llama_context_params:

uint32_t n_batch;    // logical
uint32_t n_ubatch;   // physical

Practical Configuration Examples

Maximizing GPU Throughput

For server deployments with ample VRAM, increase both batch sizes and allocate dedicated threads for prompt processing:

./llama-server \
    -m models/7B/ggml-model-q4_0.gguf \
    -c 4096 \
    -b 4096 \
    -ub 1024 \
    -tb 12 \
    --cont-batching

Setting -b 4096 allows the server to accumulate large prompts, while -ub 1024 ensures each GPU kernel processes substantial token groups. The -tb 12 flag creates a separate thread pool (cpuparams_batch) for the compute-intensive prefill stage.

Optimizing for Limited VRAM

Reduce the physical batch size to lower memory pressure per kernel launch:

./llama-cli \
    -m models/7B/ggml-model-q4_0.gguf \
    -c 2048 \
    -b 1024 \
    -ub 256 \
    -fit on

The -ub 256 limitation prevents OOM errors during large context processing, while -fit on enables automatic parameter reduction if the configuration exceeds available resources.

Low-Latency Interactive Use

For chat interfaces requiring immediate response, minimize batch sizes and disable continuous batching:

./llama-server \
    -m models/7B/ggml-model-q4_0.gguf \
    -b 256 \
    -ub 128 \
    --no-cont-batching

Small batches complete decoding cycles faster, and disabling dynamic batching (--no-cont-batching) prevents latency spikes from waiting for batch slots to fill.

Combining with Speculative Decoding

Batch processing integrates with speculative decoding via draft batches. Configure a draft model to generate candidate tokens that merge with the main batch:

./llama-server \
    -m models/70B/ggml-model-q4_0.gguf \
    --draft models/7B/ggml-model-q4_0.gguf \
    --draft-max 32 \
    --draft-p-min 0.8 \
    -b 4096 \
    -ub 1024

The draft model generates 32-token batches that the main model verifies in parallel, effectively doubling throughput when KV-cache bandwidth is the bottleneck.

Summary

• Logical batch size (-b) controls how many tokens accumulate before calling llama_decode, with a default of 2048 defined in common/common.h.
• Physical batch size (-ub) limits tokens per kernel launch to 512 by default; increasing this improves GPU utilization but requires more VRAM.
• Continuous batching (-cb) keeps the GPU busy by dynamically adding new requests to active batches during generation.
• Batch thread pools (-tb) isolate prompt-processing threads from sampling threads, preventing generation stalls during long prefill operations.
• The batching implementation in tools/server/server-context.cpp uses common_batch_add to populate llama_batch structures before submitting to llama_decode.

Frequently Asked Questions

What is the difference between n_batch and n_ubatch?

n_batch (logical) is the maximum tokens passed to llama_decode in one API call, while n_ubatch (physical) is the maximum tokens processed in a single GPU or CPU kernel launch. If n_batch exceeds n_ubatch, the decoder automatically splits the work into multiple smaller kernel calls. You typically want n_batch large for throughput and n_ubatch sized to fit your hardware's parallel compute units.

How does continuous batching improve throughput?

Continuous batching, enabled by default via common_params::cont_batching in common/common.h, allows the server to insert new prompt tokens into the current batch while other sequences are generating. Without this feature, the system waits for all active sequences to complete before starting new requests, leaving GPU compute units idle. Dynamic slot filling maximizes hardware utilization across variable-length conversations.

When should I use a separate thread pool for batch processing?

Use -tb (--threads-batch) to create a distinct thread pool (cpuparams_batch in common/common.h lines 93-95) when running the server with large batch sizes or long prompt inputs. This isolates the computationally heavy prefill phase from the lightweight sampling phase, preventing generation hiccups when new requests arrive. It is especially effective on CPU-only deployments where thread contention significantly impacts latency.

Can batch processing work with speculative decoding?

Yes. Speculative decoding generates "draft batches" using a smaller model, which are then verified by the main model in a single batched operation. Configure this using --draft-max to set the draft batch size and ensure your -ub (physical batch) value accommodates both the draft tokens and active sequences. This effectively doubles or triples throughput by amortizing the cost of large model forward passes across multiple speculative tokens.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}