How the POST /v1/chat/completions Endpoint Works in MTPLX: Complete Architecture Guide

The POST /v1/chat/completions endpoint processes chat requests through a ten-stage pipeline that validates input, authenticates via API keys, selects between MTP batch or AR generation modes, monitors stop sequences, and returns OpenAI-compatible JSON enriched with detailed telemetry metrics.

The MTPLX repository implements an OpenAI-compatible API server using FastAPI, with POST /v1/chat/completions serving as the primary interface for chat-based inference. This endpoint orchestrates request validation, scheduler selection, token generation, and response streaming while supporting both high-throughput Multi-Token Prediction (MTP) batching and traditional autoregressive generation.

Request Reception and Validation

When a client sends a POST request to /v1/chat/completions, FastAPI receives and parses the JSON payload into a ChatCompletionRequest Pydantic model. This validation step normalizes critical fields including stop, max_tokens, and response_format to ensure consistent downstream processing.

According to the source code in mtplx/server/openai.py at line 1020, the request model definition enforces type safety and default values before the request enters the processing pipeline. Invalid payloads return HTTP 422 errors immediately, preventing unnecessary compute allocation.

Authentication and Runtime Configuration

After parsing, the server authenticates the request using the resolve_api_key function located at line 1260 in mtplx/server/openai.py. Missing or invalid API keys trigger an HTTP 401 Unauthorized response, terminating the request before model loading begins.

Once authenticated, the system builds runtime environment overrides via _server_runtime_env_overrides (line 3390 in mtplx/server/openai.py). This function configures kernels and verification guards based on the request's generation_mode, verify_strategy, and model family (such as Qwen4-exp), ensuring optimal execution parameters for the specific hardware and model combination.

Scheduling Policy and Mode Selection

The resolve_request_policy function in mtplx/server/request_policy.py (line 45) determines the execution strategy for each request. This policy engine evaluates whether the request qualifies for the fast-path MTP batch lane or requires fallback to autoregressive (AR) mode.

MTP Batch Mode leverages parallel token prediction for high-throughput scenarios, while AR Mode provides traditional sequential generation for unsupported configurations or specific verification strategies. The scheduler selects between serial, MTP batch, and hyper scheduling modes based on request characteristics and current server load.

Job Dispatch and Generation Execution

With the policy determined, the _run_generation_dispatched function (line 3800 in mtplx/server/openai.py) creates the appropriate job type—either an MTPBatchJob for MTP mode or a standard AR job. This dispatcher submits the job to either MTPBatchGenerationService or the AR service, attaching a cancellation event that monitors for client disconnections, stop-sequence triggers, or internal errors.

The cancellation mechanism ensures that aborted requests release GPU resources immediately, preventing compute waste from disconnected clients or early termination conditions.

Streaming Response and Content Processing

The endpoint handles both streaming and non-streaming response paths based on the stream parameter in the request.

Streaming Implementation

When stream=True, the server returns a StreamingResponse that yields Server-Sent Events (SSE) chunks. The implementation at line 4150 in mtplx/server/openai.py manages the _StreamCancelled exception, which propagates cancellation signals cleanly and converts partial outputs into appropriate finish reasons without corrupting the SSE stream.

Stop-Sequence Monitoring

During generation, the _StopSequenceStreamMonitor class (line 1090 in mtplx/server/openai.py) inspects generated text on-the-fly. Upon detecting a stop token, it truncates the output and raises _StopSequenceHit to abort generation early, ensuring responses terminate exactly at the specified boundary conditions.

Tool Call and Thinking Extraction

For requests including function calling capabilities, the OMLX bridge in mtplx/server/omlx_bridge.py processes the raw model output. The extract_tool_calls function (line 210) parses tool invocations and extracts embedded "thinking" markup, transforming the output into OpenAI-compatible tool-call schema before inclusion in the final response.

Telemetry and Final Response Assembly

After generation completes, the _metrics_envelope function (line 13500 in mtplx/server/openai.py) aggregates runtime statistics including prefill latency, verification counts, and cache hit rates. This telemetry data populates the /v1/mtplx/health endpoint and feeds operational dashboards.

The build_generation_result function in mtplx/server/response_envelope.py (line 30) constructs the final OpenAI-compatible JSON payload, inserting generated text, finish reasons, token usage counters, and the metrics envelope. Both streaming chunks and non-streaming responses follow this standardized schema.

Practical Code Examples

Non-Streaming cURL Request

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $MTPLX_API_KEY" \
  -d '{
        "model": "mtplx-test",
        "messages": [{ "role": "user", "content": "Explain MTP batching in one sentence." }],
        "max_tokens": 50,
        "temperature": 0.7
      }'

Streaming Request with Python

import requests
import json

url = "http://127.0.0.1:8000/v1/chat/completions"
headers = {
    "Authorization": f"Bearer {open('api-key.txt').read().strip()}",
    "Content-Type": "application/json",
}
payload = {
    "model": "mtplx-test",
    "messages": [{"role": "user", "content": "Write a short poem about AI."}],
    "max_tokens": 100,
    "stream": True,
}

resp = requests.post(url, headers=headers, json=payload, stream=True)
for line in resp.iter_lines():
    if line:
        chunk = json.loads(line.decode())
        print(chunk["choices"][0]["delta"]["content"], end="")

OpenAI-Compatible Client

import openai

openai.api_key = "YOUR_MTPLX_API_KEY"
openai.base_url = "http://127.0.0.1:8000/v1"

resp = openai.ChatCompletion.create(
    model="mtplx-test",
    messages=[{"role": "user", "content": "Give a TL;DR of the MTPLX repo."}],
    max_tokens=80,
    temperature=0.5,
)

print(resp.choices[0].message.content)

Summary

• The POST /v1/chat/completions endpoint validates requests using Pydantic models (ChatCompletionRequest) in mtplx/server/openai.py before processing.
• Authentication occurs via resolve_api_key, with runtime configuration handled by _server_runtime_env_overrides to optimize kernel selection.
• The policy engine (resolve_request_policy) selects between MTP batch mode for throughput and AR mode for compatibility.
• Generation dispatch (_run_generation_dispatched) manages job lifecycle with cancellation support for resource efficiency.
• Streaming responses use SSE chunks with _StopSequenceStreamMonitor for real-time truncation and omlx_bridge.py for tool-call extraction.
• Telemetry aggregation via _metrics_envelope provides operational visibility through the health endpoint and response footers.

Frequently Asked Questions

What is the difference between MTP batch mode and AR mode in MTPLX?

MTP batch mode utilizes Multi-Token Prediction to generate multiple tokens simultaneously, significantly increasing throughput for supported models and configurations. AR (autoregressive) mode generates tokens sequentially one at a time, serving as a fallback when verification strategies or model architectures do not support parallel prediction. The resolve_request_policy function in mtplx/server/request_policy.py automatically selects the appropriate mode based on request parameters and model capabilities.

How does MTPLX handle client disconnections during streaming?

The endpoint attaches a cancellation event during job dispatch (_run_generation_dispatched in mtplx/server/openai.py) that monitors connection state. If a client disconnects, the server raises _StreamCancelled to halt generation immediately, converting partial outputs into a valid finish reason. This prevents wasted compute on abandoned requests and ensures clean resource deallocation.

Where does MTPLX validate the API key for chat completion requests?

API key validation occurs in the resolve_api_key function at line 1260 of mtplx/server/openai.py. The function checks the Authorization header against configured valid keys, returning HTTP 401 for missing or invalid credentials before any model loading or generation begins.

How are tool calls extracted in the MTPLX chat completions endpoint?

Tool calls are processed by the OMLX bridge located in mtplx/server/omlx_bridge.py. The extract_tool_calls function (line 210) parses the model's raw output to identify function invocations and "thinking" markup, then transforms these into the OpenAI-compatible tool-call schema required by the API specification.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}