DeepSeek-R1-Zero Training Without Supervised Fine-Tuning: A Pure RL Approach

DeepSeek-R1-Zero is built by applying large-scale reinforcement learning directly to the base model, completely skipping the conventional supervised fine-tuning stage that initializes most modern LLMs.

The deepseek-ai/DeepSeek-R1 repository introduces a radical departure from standard training pipelines. Unlike traditional instruction-tuned models that require curated human demonstrations before reinforcement learning begins, DeepSeek-R1-Zero demonstrates that sophisticated reasoning capabilities can emerge purely through RL optimization. This methodology validates that supervised fine-tuning is not a prerequisite for complex chain-of-thought reasoning.

The RL-Only Training Pipeline

DeepSeek-R1-Zero’s architecture eliminates the supervised fine-tuning (SFT) phase entirely. According to the repository documentation in README.md (lines 34-35), the model undergoes large-scale reinforcement learning directly on the base model without any prior instruction tuning.

This pure-RL phase exposes the model to massive exploration incentives that encourage chain-of-thought (CoT) solving strategies. During training, the model autonomously develops advanced reasoning behaviors including self-verification, reflection, and the generation of extended CoT sequences. These capabilities emerge organically through RL reward signals rather than imitation learning from human-labeled examples.

Why Skip Supervised Fine-Tuning?

Traditional LLM pipelines rely on SFT as a foundational "seed" stage where models learn basic language understanding and task formats from curated datasets before RL fine-tuning. As documented in README.md (line 53), DeepSeek-R1-Zero specifically tests the hypothesis that reasoning capabilities can be incentivized purely through RL rewards.

By removing the SFT bottleneck, the training process avoids biasing the model toward human-demonstrated reasoning patterns. Instead, the RL objective allows the model to discover potentially superior problem-solving strategies that human annotators might not exhibit. This approach proves that base models possess latent reasoning capacities unlockable through pure reinforcement signals.

Emergent Behaviors and Training Challenges

Self-Developed Reasoning Patterns

Without SFT constraints, DeepSeek-R1-Zero naturally discovers powerful reasoning heuristics. The model learns to verify its own outputs, backtrack when detecting errors, and construct elaborate multi-step reasoning chains. These behaviors emerge strictly from the RL objective function optimizing for correct answers rather than mimicking human thought processes.

RL-Only Training Artifacts

The absence of supervised fine-tuning introduces specific challenges documented in README.md (lines 37-38). The model exhibits issues typical of pure RL training:

• Endless repetition where the model loops through similar reasoning steps indefinitely
• Reduced readability including poor formatting and unclear logical structure
• Language mixing where the model switches between languages mid-reasoning

These limitations motivated the development of the full DeepSeek-R1 model, which adds a small "cold-start" supervised dataset before the second RL stage to stabilize output quality while retaining reasoning depth.

Running DeepSeek-R1-Zero Inference

While the training code remains proprietary, the repository provides guidance for serving the RL-only trained model using popular inference engines.

Serve with vLLM:

vllm serve deepseek-ai/DeepSeek-R1-Zero \
      --tensor-parallel-size 2 \
      --max-model-len 32768 \
      --enforce-eager

Serve with SGLang:

python -m sglang.launch_server \
      --model deepseek-ai/DeepSeek-R1-Zero \
      --trust-remote-code \
      --tp 2

Basic inference via OpenAI-compatible API:

import openai

client = openai.OpenAI(
    base_url="https://api.deepseek.com/v1",
    api_key="YOUR_API_KEY"
)

resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-Zero",
    messages=[{"role": "user", "content": "Explain the chain-of-thought reasoning process."}],
    temperature=0.6
)

print(resp.choices[0].message.content)

Summary

• DeepSeek-R1-Zero uses pure RL: The model receives no supervised fine-tuning, with reinforcement learning applied directly to the base architecture according to README.md lines 34-35.
• SFT is not required for reasoning: As stated in line 53 of the repository documentation, sophisticated reasoning emerges solely from RL incentives without curated instruction datasets.
• Trade-offs exist: The RL-only approach generates powerful reasoning but suffers from readability issues, endless repetition, and language mixing (lines 37-38).
• DeepSeek-R1 addresses limitations: The full model adds cold-start supervised data to fix Zero’s output quality issues while maintaining reasoning capabilities.

Frequently Asked Questions

What is the main difference between DeepSeek-R1-Zero and standard LLM training?

Standard pipelines use supervised fine-tuning to teach models basic task formats and reasoning patterns from human demonstrations before applying RL. DeepSeek-R1-Zero eliminates this initial SFT phase entirely, applying reinforcement learning directly to the raw base model to let reasoning strategies emerge organically.

Does DeepSeek-R1-Zero use any supervised data at all?

No. According to the deepseek-ai/DeepSeek-R1 documentation, DeepSeek-R1-Zero is trained with zero supervised fine-tuning. The full DeepSeek-R1 model (non-Zero) subsequently added a small supervised "cold-start" phase to address readability and coherence issues observed in the Zero version.

What problems arise from training without supervised fine-tuning?

The RL-only training approach documented in README.md lines 37-38 produces models prone to endless repetition loops, inconsistent formatting that reduces readability, and spontaneous language mixing during reasoning chains. These artifacts occur because the model lacks exposure to human-preferred output structures during its initial training phase.

How can I run DeepSeek-R1-Zero locally?

You can serve the model using vLLM or SGLang with tensor parallelism for multi-GPU setups, or access it via the DeepSeek API using OpenAI-compatible endpoints. The model requires significant VRAM and benefits from tensor-parallel-size configurations of 2 or more for efficient inference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}