How to Set Up and Tune the DSpark Speculative Decoding Confidence Threshold in DS4

Set the --dspark-confidence flag to a value between 0.0 and 1.0 (default 0.6 for Metal, 0.7 for CUDA) to control how aggressively DSpark accepts draft tokens during speculative decoding.

DSpark is the speculative-decoding driver in the antirez/ds4 repository, enabled via the --dspark flag. When active, DSpark uses a tiny confidence head—a linear projection within the model—to predict whether each draft token is likely correct. The confidence threshold determines the cutoff for accepting these predictions, directly balancing inference speed against token accuracy.

Understanding the Confidence Threshold Mechanism

During a speculative cycle, DSpark generates a draft token sequence followed by a verification pass on the target model. The confidence head outputs a logit for each draft token, which is passed through a sigmoid function and compared against the configured threshold.

The Verification Process

In ds4.c, the function dspark_apply_markov_confidence_lazy_runtime (line 33472) performs the actual pruning test:

if (sigmoid_stable(confidence_logit) < confidence_threshold) {
    /* reject draft token */
} else {
    /* accept draft token */
}

If the sigmoid output falls below the threshold, the draft token is rejected and the verifier re-runs; if above, the token is accepted and the speculative state commits. The threshold value is stored in engine->dspark_confidence_threshold, defined at line 36854 in ds4.c.

Threshold Impact on Performance

The confidence threshold acts as a dial for speculation aggressiveness:

• Low values (e.g., 0.4-0.5): High acceptance rates, greater speed-up potential, but increased risk of incorrect drafts requiring re-decoding.
• Default values (0.6-0.7): Balanced trade-off between throughput and accuracy.
• High values (e.g., 0.8-0.9): Conservative acceptance, fewer speculative errors, but reduced speed-up due to frequent verifier passes.

Default Confidence Thresholds by Backend

DS4 applies backend-specific defaults when the --dspark-confidence flag is omitted:

Backend	Default Threshold	Location
Metal	0.6	ds4_help.c (line 1868)
CUDA / ROCm	0.7	ds4_help.c (line 1868)

These values are baked into the CLI help text and initialized in the engine struct at runtime.

Configuring the Confidence Threshold

You can override the default threshold using the --dspark-confidence flag across all DS4 binaries. The flag accepts a floating-point value and must be used in conjunction with --dspark.

Interactive Client (ds4)

# Use a conservative threshold on Metal

ds4 --model mymodel.gguf --dspark --dspark-confidence 0.85

HTTP Server (ds4-server)

# Run server with aggressive speculation

ds4-server --model mymodel.gguf --dspark --dspark-confidence 0.55 --port 8080

Coding Assistant (ds4-agent)

# Balanced setting for agent workflows

ds4-agent --model mymodel.gguf --dspark --dspark-confidence 0.65

The threshold can also be specified in configuration files (ds4.cfg) or via environment variables if your wrapper script forwards them, though the CLI flag provides the most direct control.

Practical Tuning Strategies

Follow these steps to optimize the DSpark speculative decoding confidence threshold for your specific hardware and model:

1. Start with the backend default—0.6 for Metal or 0.7 for CUDA/ROCm—to establish a baseline.
2. Enable verbose logging using --verbose to monitor per-token confidence values and acceptance decisions in real time.
3. If you observe frequent verification failures (indicated by "re-decode" messages in logs), raise the threshold to 0.8 or higher to reduce speculative errors.
4. For maximum throughput on high-end GPUs, lower the threshold to 0.5 or 0.55, accepting the occasional re-decoding step in exchange for higher acceptance rates.
5. Validate model support—DS4 prints "confidence: yes" during model load if the confidence head is present. If the model lacks this head, the threshold parameter is ignored and DSpark falls back to default MTP behavior.

Implementation Details

The threshold check occurs within the speculative decoding loop. Here is the relevant excerpt from ds4.c showing how the engine applies the threshold:

float confidence_threshold = s->engine->dspark_confidence_threshold;
if (sigmoid_stable(confidence_logit) < confidence_threshold) {
    /* Draft token rejected – fall back to verifier */
    reject_token();
} else {
    /* Accept draft token */
    commit_token();
}

Unit tests in tests/ds4_test.c (lines 6298-6299) verify this behavior by setting a custom threshold of 0.9 and asserting correct rejection logic.

Summary

• The DSpark confidence threshold controls draft token acceptance in DS4's speculative decoding pipeline.
• Default values are 0.6 for Metal and 0.7 for CUDA/ROCm, stored in engine->dspark_confidence_threshold.
• Set the threshold via --dspark-confidence <float> in ds4, ds4-server, or ds4-agent.
• Lower values increase speed but risk more verification failures; higher values improve accuracy at the cost of throughput.
• The model must contain a confidence head (indicated by "confidence: yes" at load time) for this parameter to take effect.

Frequently Asked Questions

What is the default DSpark confidence threshold for Metal versus CUDA?

The Metal backend defaults to 0.6, while CUDA and ROCm backends default to 0.7. These values are defined in the CLI help text within ds4_help.c and applied to engine->dspark_confidence_threshold during engine initialization.

How do I know if my model supports confidence-based speculative decoding?

During model loading, DS4 prints a diagnostic message indicating "confidence: yes" if the model includes a confidence head, or "confidence: no" if it does not. If the head is absent, the --dspark-confidence flag is ignored and DSpark operates in fallback mode without confidence-based pruning.

What happens if I set the confidence threshold too low?

Setting the threshold too low (e.g., 0.3 or 0.4) causes the engine to accept draft tokens that are likely incorrect. This increases the acceptance rate and speculative speed-up initially, but results in frequent verification failures and re-decoding steps, potentially reducing overall throughput and increasing latency.

Can I adjust the confidence threshold without restarting the server?

No, the confidence threshold is parsed at startup and stored in the engine struct (engine->dspark_confidence_threshold in ds4.c). Changes require restarting the ds4, ds4-server, or ds4-agent process with the new --dspark-confidence value.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}