# DeepSeek-R1-Zero Training Without Supervised Fine-Tuning: A Pure RL Approach

> Discover DeepSeek-R1-Zero training, a pure RL approach that bypasses supervised fine-tuning. Learn how this method directly applies large-scale reinforcement learning to the base model for optimal LLM initialization.

- Repository: [DeepSeek/DeepSeek-R1](https://github.com/deepseek-ai/DeepSeek-R1)
- Tags: deep-dive
- Published: 2026-02-27

---

**DeepSeek-R1-Zero is built by applying large-scale reinforcement learning directly to the base model, completely skipping the conventional supervised fine-tuning stage that initializes most modern LLMs.**

The `deepseek-ai/DeepSeek-R1` repository introduces a radical departure from standard training pipelines. Unlike traditional instruction-tuned models that require curated human demonstrations before reinforcement learning begins, DeepSeek-R1-Zero demonstrates that sophisticated reasoning capabilities can emerge purely through RL optimization. This methodology validates that supervised fine-tuning is not a prerequisite for complex chain-of-thought reasoning.

## The RL-Only Training Pipeline

DeepSeek-R1-Zero’s architecture eliminates the supervised fine-tuning (SFT) phase entirely. According to the repository documentation in [`README.md`](https://github.com/deepseek-ai/DeepSeek-R1/blob/main/README.md) (lines 34-35), the model undergoes **large-scale reinforcement learning directly on the base model** without any prior instruction tuning.

This pure-RL phase exposes the model to massive exploration incentives that encourage chain-of-thought (CoT) solving strategies. During training, the model autonomously develops advanced reasoning behaviors including **self-verification**, **reflection**, and the generation of extended CoT sequences. These capabilities emerge organically through RL reward signals rather than imitation learning from human-labeled examples.

## Why Skip Supervised Fine-Tuning?

Traditional LLM pipelines rely on SFT as a foundational "seed" stage where models learn basic language understanding and task formats from curated datasets before RL fine-tuning. As documented in [`README.md`](https://github.com/deepseek-ai/DeepSeek-R1/blob/main/README.md) (line 53), DeepSeek-R1-Zero specifically tests the hypothesis that **reasoning capabilities can be incentivized purely through RL rewards**.

By removing the SFT bottleneck, the training process avoids biasing the model toward human-demonstrated reasoning patterns. Instead, the RL objective allows the model to discover potentially superior problem-solving strategies that human annotators might not exhibit. This approach proves that base models possess latent reasoning capacities unlockable through pure reinforcement signals.

## Emergent Behaviors and Training Challenges

### Self-Developed Reasoning Patterns

Without SFT constraints, DeepSeek-R1-Zero naturally discovers powerful reasoning heuristics. The model learns to verify its own outputs, backtrack when detecting errors, and construct elaborate multi-step reasoning chains. These behaviors emerge strictly from the RL objective function optimizing for correct answers rather than mimicking human thought processes.

### RL-Only Training Artifacts

The absence of supervised fine-tuning introduces specific challenges documented in [`README.md`](https://github.com/deepseek-ai/DeepSeek-R1/blob/main/README.md) (lines 37-38). The model exhibits issues typical of pure RL training:

- **Endless repetition** where the model loops through similar reasoning steps indefinitely
- **Reduced readability** including poor formatting and unclear logical structure
- **Language mixing** where the model switches between languages mid-reasoning

These limitations motivated the development of the full DeepSeek-R1 model, which adds a small "cold-start" supervised dataset before the second RL stage to stabilize output quality while retaining reasoning depth.

## Running DeepSeek-R1-Zero Inference

While the training code remains proprietary, the repository provides guidance for serving the RL-only trained model using popular inference engines.

**Serve with vLLM:**

```bash
vllm serve deepseek-ai/DeepSeek-R1-Zero \
      --tensor-parallel-size 2 \
      --max-model-len 32768 \
      --enforce-eager

```

**Serve with SGLang:**

```bash
python -m sglang.launch_server \
      --model deepseek-ai/DeepSeek-R1-Zero \
      --trust-remote-code \
      --tp 2

```

**Basic inference via OpenAI-compatible API:**

```python
import openai

client = openai.OpenAI(
    base_url="https://api.deepseek.com/v1",
    api_key="YOUR_API_KEY"
)

resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-R1-Zero",
    messages=[{"role": "user", "content": "Explain the chain-of-thought reasoning process."}],
    temperature=0.6
)

print(resp.choices[0].message.content)

```

## Summary

- **DeepSeek-R1-Zero uses pure RL**: The model receives no supervised fine-tuning, with reinforcement learning applied directly to the base architecture according to [`README.md`](https://github.com/deepseek-ai/DeepSeek-R1/blob/main/README.md) lines 34-35.
- **SFT is not required for reasoning**: As stated in line 53 of the repository documentation, sophisticated reasoning emerges solely from RL incentives without curated instruction datasets.
- **Trade-offs exist**: The RL-only approach generates powerful reasoning but suffers from readability issues, endless repetition, and language mixing (lines 37-38).
- **DeepSeek-R1 addresses limitations**: The full model adds cold-start supervised data to fix Zero’s output quality issues while maintaining reasoning capabilities.

## Frequently Asked Questions

### What is the main difference between DeepSeek-R1-Zero and standard LLM training?

Standard pipelines use supervised fine-tuning to teach models basic task formats and reasoning patterns from human demonstrations before applying RL. DeepSeek-R1-Zero eliminates this initial SFT phase entirely, applying reinforcement learning directly to the raw base model to let reasoning strategies emerge organically.

### Does DeepSeek-R1-Zero use any supervised data at all?

No. According to the `deepseek-ai/DeepSeek-R1` documentation, DeepSeek-R1-Zero is trained with **zero supervised fine-tuning**. The full DeepSeek-R1 model (non-Zero) subsequently added a small supervised "cold-start" phase to address readability and coherence issues observed in the Zero version.

### What problems arise from training without supervised fine-tuning?

The RL-only training approach documented in [`README.md`](https://github.com/deepseek-ai/DeepSeek-R1/blob/main/README.md) lines 37-38 produces models prone to endless repetition loops, inconsistent formatting that reduces readability, and spontaneous language mixing during reasoning chains. These artifacts occur because the model lacks exposure to human-preferred output structures during its initial training phase.

### How can I run DeepSeek-R1-Zero locally?

You can serve the model using vLLM or SGLang with tensor parallelism for multi-GPU setups, or access it via the DeepSeek API using OpenAI-compatible endpoints. The model requires significant VRAM and benefits from `tensor-parallel-size` configurations of 2 or more for efficient inference.