How to Switch to Target-Only AR Decoding for a Specific Request in MTPLX

Set the generation_mode parameter to "ar" in your request payload to force MTPLX to use target-only autoregressive decoding for that specific call without restarting the server.

MTPLX supports both speculative Multitoken Prediction (MTP) and standard autoregressive (AR) generation. While MTP uses a draft head to accelerate token generation, you can switch to target-only AR decoding for a specific request—useful for baselines, debugging, or when your model checkpoint lacks an MTP head. This toggle happens at the request level and does not require reloading the runtime.

Understanding MTPLX Generation Modes

MTPLX operates in two distinct generation modes:

• MTP (default) – Uses the draft head for speculative decoding, validating multiple tokens per forward pass.
• AR-only (target-only) – Returns only logits from the target model, bypassing the draft head entirely. This mode is triggered when the runtime’s mtp_enabled flag is set to False.

The mode is determined per-request via the generation_mode field in the JSON payload or the --no-mtp CLI flag.

How to Enable Target-Only AR Decoding Per Request

Using the HTTP API (cURL)

Add "generation_mode": "ar" to your JSON body. In mtplx/server/openai.py (line 26729), the server extracts this field and stores it in the request metadata.

curl -X POST https://your-mtplx-host/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "gpt-4o-mini",
        "messages": [{"role":"user","content":"Explain quantum tunneling"}],
        "generation_mode": "ar"
      }'

The server routes this request through the target-only AR path, returning only logits for each token.

Using the Python Client

If you use an OpenAI-compatible Python client, pass generation_mode="ar" as an extra argument:

import openai

client = openai.OpenAI(base_url="https://your-mtplx-host/v1")
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Explain quantum tunneling"}],
    generation_mode="ar",          # Forces target-only AR for this call

)

print(resp.choices[0].message.content)

Using the CLI

For one-off requests, use the --no-mtp flag or the /mtp REPL command. In mtplx/commands/public.py (lines 720‑728), the CLI normalizes --no-mtp to generation_mode="ar".

# Launch REPL in AR-only mode

mtplx start cli --no-mtp

# Or toggle dynamically inside the REPL

> /mtp off
> ask "Explain quantum tunneling"

The /mtp off command sets the internal no_mtp flag to True, which translates to generation_mode="ar" for the next request (see mtplx/cli.py, lines 304‑313).

Internal Implementation Details

Request Parsing in openai.py

When a request arrives, mtplx/server/openai.py extracts the generation_mode value from the incoming JSON and injects it into the request metadata. This metadata travels with the request through the entire pipeline, ensuring the flag persists for the duration of generation.

Runtime Dispatch in generation.py

The actual mode switch happens in mtplx/generation.py (line 776) inside the forward_ar function. When runtime.mtp_enabled is False, the function returns (logits, hidden) with hidden=None, signaling the target-only AR loop to proceed without speculative drafts.

CLI Normalization in public.py

In mtplx/commands/public.py (lines 720‑728), the CLI layer reconciles the --no-mtp flag with any explicit generation_mode parameter passed via the API. If either is present, the system sets the internal representation to "ar" before dispatching to the runtime.

Mixing AR-Only and MTP Requests

Because the mode is attached to request metadata rather than global state, you can alternate between modes without restarting the server. After processing an AR-only request, re-enable MTP for the next call:

# AR-only request

> /mtp off
> ask "Explain quantum tunneling"

# Back to MTP speculative decoding

> /mtp on
> ask "Summarize the previous answer"

The /mtp on command resets the internal flag to False, restoring the default MTP pipeline for subsequent requests.

Summary

• Target-only AR decoding is enabled by setting generation_mode: "ar" in the request payload.
• The request metadata carries this flag through mtplx/server/openai.py, allowing per-request toggling.
• When mtp_enabled is False, mtplx/generation.py executes the forward_ar path without draft heads.
• You can mix modes on the same server instance; the runtime adjusts dynamically without model reloading.
• The CLI provides --no-mtp and /mtp off shortcuts that normalize to the same internal flag.

Frequently Asked Questions

Can I mix AR-only and MTP requests on the same server instance?

Yes. The generation mode is stored in per-request metadata parsed by mtplx/server/openai.py. The server keeps the same loaded runtime and simply toggles the mtp_enabled flag for the duration of each request, allowing you to alternate between speculative and target-only decoding without restarting the process.

Does switching to AR-only mode require reloading the model?

No. The runtime remains resident in memory. When generation_mode="ar" is detected, the system temporarily sets runtime.mtp_enabled = False for that specific request, causing forward_ar in mtplx/generation.py to skip the draft head and return logits only.

What happens if I use both the --no-mtp CLI flag and the generation_mode="ar" API parameter?

The CLI handling in mtplx/commands/public.py (lines 720‑728) normalizes both inputs to the same internal representation. If you specify --no-mtp globally but override with generation_mode="mtp" in a specific API call, the explicit API parameter typically takes precedence for that request.

Is target-only AR faster or slower than MTP?

Target-only AR decoding eliminates the overhead of the draft head and validation logic, but it loses the latency benefits of speculative generation. MTP is generally faster for throughput, while AR-only provides a deterministic baseline useful for benchmarking or when the checkpoint lacks an MTP head.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}