How to Run DeepSeek-R1 Models Locally Using SGLang: A Complete Setup Guide

Use SGLang's asynchronous inference server to serve DeepSeek-R1 checkpoints with tensor parallelism, exposing a local HTTP API for high-throughput generation.

DeepSeek-R1 (including the Zero variant) are MoE-based large language models available as standard HuggingFace checkpoints. Because the Transformers library does not yet provide native support for these specific checkpoints, the recommended way to run DeepSeek-R1 models locally is to use SGLang, an inference engine capable of handling contexts up to 128 K tokens with efficient tensor-parallel execution.

Prerequisites and Installation

System Requirements

Running DeepSeek-R1 locally requires a Linux environment with Python ≥3.9, PyTorch, and sufficient GPU memory for the model size you select. The distilled variants (e.g., DeepSeek-R1-Distill-Qwen-32B) require less VRAM than the full 671B parameter model, but all variants benefit from multi-GPU setups via tensor parallelism.

Installing SGLang

Install the SGLang package from PyPI. This provides the sglang.launch_server module used to start the inference backend.

pip install sglang

Downloading the Model Checkpoint

SGLang can load models directly from HuggingFace Hub if you have internet connectivity, or you can pre-download the weights for offline use. For the 32B distilled variant used in the official examples:

git lfs install
git clone https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B

Alternatively, verify accessibility without cloning:

python -c "from huggingface_hub import hf_hub_download; \
print(hf_hub_download('deepseek-ai/DeepSeek-R1-Distill-Qwen-32B', 'config.json'))"

Launching the SGLang Server

The DeepSeek-R1 repository's README documents the minimal command to start an SGLang server for these models. According to the source code at README.md (line 183), you must pass --trust-remote-code to handle the MoE architecture and --tp to specify tensor parallelism across GPUs.

python3 -m sglang.launch_server \
    --model deepseek-ai/DeepSeek-R1-Distill-Qwen-32B \
    --trust-remote-code \
    --tp 2

Adjust the --tp value to match your available GPU count. For example, use --tp 4 on a 4-GPU machine to distribute the model layers evenly and increase throughput.

Sending Inference Requests

Once the server is running on the default port (8000), send prompts via HTTP using the SGLang client library or raw REST calls. The following Python example demonstrates connecting to the local server and configuring generation parameters optimized for DeepSeek-R1's reasoning capabilities:

import sglang as sgl

# Create a client pointing to the locally-running server

client = sgl.Client("http://localhost:8000")

prompt = """Please solve the following math problem step-by-step and give the final answer inside \\boxed{}:

What is the integral of sin(x)^2 from 0 to π?"""

# Generation settings: temperature between 0.5-0.7 yields optimal reasoning

response = client.generate(
    prompt,
    temperature=0.6,
    top_p=0.95,
    max_new_tokens=512,
    stop=None,
)

print("Model response:", response.text)

Optimizing Performance for DeepSeek-R1

Temperature and Sampling

According to the repository's recommendations, DeepSeek-R1 performs best when the temperature is set between 0.5 and 0.7. Values below 0.5 may produce repetitive reasoning traces, while values above 0.7 can degrade mathematical accuracy.

Context Length Configuration

SGLang supports extremely long contexts up to 128 K tokens for DeepSeek-R1. Ensure your GPU memory is sufficient for the cache size when processing long documents, or reduce --max-model-len if you encounter out-of-memory errors during extended conversations.

Summary

• SGLang is the recommended backend for running DeepSeek-R1 models locally because standard Transformers does not yet natively support these MoE checkpoints.
• Use --trust-remote-code when launching the server to properly initialize the model architecture.
• Tensor parallelism via --tp distributes the model across multiple GPUs, enabling larger batch sizes and faster inference.
• Optimal generation parameters include a temperature of 0.6 and top_p of 0.95 for mathematical reasoning tasks.

Frequently Asked Questions

What hardware is required to run DeepSeek-R1 locally?

You need a multi-GPU setup for the larger variants. The 32B distilled model runs comfortably on two modern GPUs with tensor parallelism (--tp 2), while the full 671B parameter model requires significantly more VRAM and GPUs. All variants require Python ≥3.9 and CUDA-capable hardware.

Why does SGLang require the --trust-remote-code flag?

DeepSeek-R1 uses a Mixture-of-Experts (MoE) architecture with custom layer implementations not yet merged into the standard Transformers library. The --trust-remote-code parameter allows SGLang to execute the model-specific Python files included in the HuggingFace repository, ensuring correct weight loading and forward passes.

How do I adjust the server for different model sizes?

Change the --model path to point to any DeepSeek-R1 checkpoint (e.g., deepseek-ai/DeepSeek-R1-Distill-Llama-70B) and modify --tp to match your GPU count. The launch command structure remains identical regardless of model size, though you must ensure you have sufficient aggregate GPU memory for the chosen checkpoint.

Can I use the server without the Python client library?

Yes. The SGLang server exposes a standard OpenAI-compatible HTTP API at http://localhost:8000/v1/chat/completions. You can send POST requests with JSON payloads containing the prompt, temperature, and max_tokens fields using curl, JavaScript, or any HTTP client.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}