How to Implement Speculative Decoding for Faster Token Generation in llama.cpp

Speculative decoding accelerates LLM inference by drafting candidate tokens with a fast auxiliary model or self-speculative algorithm, then verifying them in parallel with the target model to reduce generation latency while maintaining output quality.

Speculative decoding is a performance optimization technique that reduces per-token latency in large language model inference. In llama.cpp, this capability is implemented through a modular architecture defined in common/speculative.h and common/speculative.cpp that supports both separate draft models and lightweight self-speculative n-gram methods. The system integrates seamlessly with llama-server and CLI tools via a unified state machine API.

Architecture Overview

The speculative decoding system in llama.cpp operates on a draft-then-verify paradigm. It generates candidate token sequences using a fast approximation method, then validates them against the target model in a single batch decode operation. The implementation supports multiple drafting strategies through a common interface defined by the common_speculative_state abstract base class.

Core Types and State Machine

The public API is declared in common/speculative.h and centers around the common_speculative_type enumeration:

enum common_speculative_type {
    COMMON_SPECULATIVE_TYPE_NONE,
    COMMON_SPECULATIVE_TYPE_DRAFT,
    COMMON_SPECULATIVE_TYPE_EAGLE3,
    COMMON_SPECULATIVE_TYPE_NGRAM_SIMPLE,
    COMMON_SPECULATIVE_TYPE_NGRAM_MAP_K,
    COMMON_SPECULATIVE_TYPE_NGRAM_MAP_K4V,
    COMMON_SPECULATIVE_TYPE_NGRAM_MOD,
    COMMON_SPECULATIVE_TYPE_NGRAM_CACHE,
    COMMON_SPECULATIVE_TYPE_COUNT
};

The state machine interface in common/speculative.cpp (lines 9–45) defines three virtual methods that every implementation must provide:

• begin – Initializes internal data structures when a new prompt starts.
• draft – Generates candidate tokens based on the current context.
• accept – Updates statistics after the target model verifies the draft.

Each implementation maintains counters for calls, generated drafts, accepted tokens, and timing statistics for performance analysis.

Draft Model Implementation

The draft model approach (source: common/speculative.cpp, lines 47–215) creates a secondary llama_context (ctx_dft) using a smaller, faster model. The draft method performs the following steps:

1. Prompt translation – Converts tokens between vocabularies if the draft and target models differ using replace_to_dft.
2. Context reuse – Attempts to reuse previously computed KV cache entries (reuse_i, reuse_n) to minimize redundant computation.
3. Batch decoding – Constructs a llama_batch and executes llama_decode on the draft model.
4. Token sampling – Uses a lightweight sampler with top-k = 10 and terminates when token probability falls below params.p_min.
5. Vocabulary mapping – Translates draft tokens back to the target model's vocabulary if necessary.

This implementation prioritizes speed over quality for the draft generation, accepting that many tokens may be rejected during verification.

Self-Speculative Algorithms

For scenarios where loading a second model is impractical, llama.cpp provides self-speculative methods that analyze the existing context to predict future tokens:

• ngram-simple (common_speculative_state_ngram_simple, lines 65–88): Searches the prompt history for the last matching n-gram of size config.size_ngram and reuses the following m-gram (config.size_mgram) as the draft sequence.

• ngram-map-k (common_speculative_state_ngram_map_k, lines 94–115): Maintains an in-memory hash map (common_ngram_map from common/ngram-map.h) for O(1) lookups. The begin method builds this map from the current prompt, and draft performs fast hash-based retrieval.

• ngram-mod (common_speculative_state_ngram_mod, lines 121–224): Implements a rolling hash table (common_ngram_mod) shared across all server slots. It predicts the next token based on historical patterns and stops early if the table returns EMPTY.

• ngram-cache (common_speculative_state_ngram_cache, lines 226–336): Loads pre-computed n-gram statistics from disk using common_ngram_cache_load and maintains a dynamic cache that updates during generation.

These methods require no additional model weights and operate entirely within the target model's context window.

Configuration and Lifecycle

The speculative system follows a strict lifecycle managed through the common_speculative_* API functions.

System Lifecycle Functions

The primary functions driving the speculative process (source: common/speculative.cpp, lines 337–495) are:

• common_speculative_init – Constructs the implementation list based on params.type and initializes the draft context if needed. Self-speculative methods are added first, with draft models last in priority order.

• common_speculative_begin – Invokes the begin method on all implementations to prepare data structures (e.g., building n-gram maps) before generation starts.

• common_speculative_draft – Iterates through implementations in priority order, returning the first non-empty draft and setting curr_impl to the successful implementation.

• common_speculative_accept – Notifies the current implementation how many tokens were verified, allowing it to update internal statistics or reset caches.

• common_speculative_print_stats – Emits per-implementation metrics including acceptance rates and timing data.

Command-Line Interface

Configure speculative decoding via llama-server or CLI tools using these flags (documented in docs/speculative.md):

Flag	Default	Description
--draft / --draft-max N	16	Maximum tokens to draft when using a draft model.
--draft-min N	0	Minimum draft tokens required.
--draft-p-min P	0.75	Minimum greedy probability for draft token acceptance.
--spec-type TYPE	none	Self-speculative algorithm (ngram-simple, ngram-map-k, ngram-mod, ngram-cache).
--spec-ngram-size-n N	12	Size of lookup n-gram.
--spec-ngram-size-m M	48	Size of draft m-gram.
--spec-ngram-min-hits H	1	Minimum hit count for ngram-map eligibility.

Example configuration combining draft model with n-gram fallback:

./llama-server \
  -m models/llama-7B/ggml-model-q4_0.gguf \
  -md models/llama-0.5B/ggml-model-q4_0.gguf \
  --draft-max 32 \
  --spec-type ngram-simple \
  --spec-ngram-size-n 16 \
  --spec-ngram-size-m 64

C++ API Integration

Implement speculative decoding programmatically using the common_params_speculative structure:

// Configure speculative parameters
common_params_speculative spec_params = {
    .type = COMMON_SPECULATIVE_TYPE_DRAFT,
    .model_dft = "models/llama-0.5B/ggml-model-q4_0.gguf",
    .draft_n_max = 32,
    .draft_p_min = 0.80f,
    .ngram_size_n = 16,
    .ngram_size_m = 64,
};

// Initialize with target context
common_speculative *spec = common_speculative_init(spec_params, ctx_target);

// Start new prompt
common_speculative_begin(spec, prompt_tokens);

// Generation loop
while (generating) {
    // Generate draft
    llama_tokens draft = common_speculative_draft(
        spec, spec_params, prompt_tokens, last_token_id);
    
    // Verify draft with target model...
    uint16_t n_accepted = verify_with_target_model(draft);
    
    // Update state
    common_speculative_accept(spec, n_accepted);
}

// Cleanup
common_speculative_print_stats(spec);
common_speculative_free(spec);

Example Programs

The repository provides two reference implementations demonstrating usage patterns:

• examples/speculative/speculative.cpp – Full-featured server-style demo combining draft models with self-speculative fallbacks, including proper batching and verification loops.

• examples/speculative-simple/speculative-simple.cpp – Minimal implementation using only the greedy n-gram simple method, ideal for understanding the basic API surface.

Both examples demonstrate proper parameter configuration, lifecycle management, and statistics reporting according to the patterns established in common/speculative.cpp.

Summary

• Speculative decoding in llama.cpp uses a pluggable architecture supporting both external draft models and self-speculative n-gram methods.
• The system is driven by the common_speculative_* API functions defined in common/speculative.h and common/speculative.cpp.
• Implementation priority places lightweight self-speculative methods first, falling back to draft models only when cheaper methods fail.
• Configuration occurs through common_params_speculative or command-line flags like --spec-type and --draft-max.
• The lifecycle requires calling common_speculative_begin, common_speculative_draft, and common_speculative_accept within the generation loop to maintain state.

Frequently Asked Questions

What is the difference between draft model and self-speculative decoding?

Draft model decoding uses a separate, smaller language model to generate candidate tokens quickly, while self-speculative decoding analyzes the target model's own context window using n-gram pattern matching. According to the llama.cpp source code, draft models typically achieve higher acceptance rates but require additional memory and load time, whereas self-speculative methods like ngram-simple have zero memory overhead but depend on repetitive patterns in the prompt.

How do I choose the right n-gram size for speculative decoding?

Set --spec-ngram-size-n (lookup size) based on your context's repetitiveness; the default of 12 works well for most code and text generation. For --spec-ngram-size-m (draft size), larger values like 48 or 64 increase potential speedups but risk higher rejection rates if the draft diverges from the target model's distribution. Monitor acceptance rates using common_speculative_print_stats to tune these parameters.

Can I combine multiple speculative decoding methods?

Yes. The common_speculative_init function (lines 54–95 in common/speculative.cpp) builds a vector of implementations in priority order. When you specify multiple types or provide both a draft model and n-gram configuration, the system tries cheaper methods first (n-gram lookups) before invoking the expensive draft model, automatically falling back through the priority chain until a valid draft is produced.

Does speculative decoding affect output quality?

No. The verification step in the target model ensures that only tokens matching its probability distribution are accepted. If the draft model or n-gram heuristic suggests a token that the target model would not have generated with its standard sampling parameters, that token and all subsequent draft tokens are discarded. This guarantee preserves the target model's output distribution exactly while reducing latency through parallel verification.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}