# LLM Post-Training: SFT vs RL Tradeoffs and Implementation Strategies

> Explore LLM post-training: understand Supervised Fine-Tuning SFT vs Reinforcement Learning RL tradeoffs and implementation strategies. Optimize your AI agent's performance.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: deep-dive
- Published: 2026-08-01

---

**Supervised Fine-Tuning (SFT) excels at sample-efficient memorization of fixed distributions, while Reinforcement Learning (RL) enables generalization beyond training data through reward optimization, with production pipelines typically combining both stages.**

After pre-training, large language models require post-training to align with specific tasks and user preferences. This article examines the **LLM post-training SFT vs RL tradeoffs** through the lens of the `bojieli/ai-agent-book` repository, analyzing actual implementation files and training pipelines used in multilingual reasoning and tool-use scenarios.

## Core Paradigm Differences

### Supervised Fine-Tuning: Fixed Distribution Learning

**SFT** aligns models to human-written **ground-truth** examples through direct supervision. According to the repository's visualization in [`book-hu/localize_figures.py`](https://github.com/bojieli/ai-agent-book/blob/main/book-hu/localize_figures.py) (lines 1122-1126), SFT **memorises the distribution** present in the training data. This approach requires only a few thousand labeled examples to achieve strong performance because the loss function directly compares predictions against fixed targets.

### Reinforcement Learning: Reward Optimization

**RL** optimizes models to maximize a **reward signal** reflecting task performance, such as tool-call success or human feedback. Unlike SFT, RL **generalises the learned strategy** beyond exact training trajectories, enabling the model to discover novel solutions to achieve rewards. However, this comes at the cost of **sample efficiency**—RL requires many rollout episodes to estimate gradients reliably.

## Behavioral Tradeoffs in Practice

### Memorization vs Generalization

The dichotomy is stark: SFT tends to reproduce patterns it has seen, leading to overfitting on the supervised set, while RL encourages adaptation to unseen situations. As implemented in `bojieli/ai-agent-book`, SFT provides deterministic behavior ideal for well-defined tasks, whereas RL refines the decision-making process for complex, non-differentiable objectives.

### Tool-Call Internalization

SFT learns tool usage as **token-level patterns**—the model emits correct API syntax without necessarily understanding underlying semantics. In contrast, RL can convert tool-calling into a **native capability**: when the model receives reward feedback upon tool success, it learns to treat the tool as part of its action space. This distinction is evident in [`chapter7/orpheus/orpheus_sft_unsloth.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter7/orpheus/orpheus_sft_unsloth.py), where initial SFT establishes voice-cloning patterns before RL stages improve tool-use integration.

## Implementation Examples from ai-agent-book

The repository provides concrete implementations illustrating both paradigms and their combination.

### SFT Training with trl.SFTTrainer

The script [`chapter7/MultilingualReasoning/gpt_oss_20b_sft.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter7/MultilingualReasoning/gpt_oss_20b_sft.py) demonstrates efficient supervised fine-tuning:

```python

# File: chapter7/MultilingualReasoning/gpt_oss_20b_sft.py

from trl import SFTTrainer, SFTConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "gpt2"                     # replace with a 20‑B checkpoint in practice

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# 𝑆𝑓𝑡 𝑫𝑎𝑡𝑎𝑠𝑒𝑡 – a JSONL file where each entry has {"prompt":…, "completion":…}

train_dataset = ...  # see utils in this file for loading

training_args = SFTConfig(
    output_dir="sft_output",
    per_device_train_batch_size=4,
    num_train_epochs=2,
    learning_rate=5e-5,
    logging_steps=10,
)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    args=training_args,
    train_dataset=train_dataset,
)

trainer.train()

```

This configuration achieves fast convergence with minimal compute, making it suitable for deterministic instruction-following tasks.

### RL Training with Trajectory Verification

For RL implementations, [`chapter8/trajectory-verifier/run_experiment_8_1.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/trajectory-verifier/run_experiment_8_1.py) shows how to optimize against non-differentiable rewards:

```python

# File: chapter8/trajectory-verifier/run_experiment_8_1.py

from agentbook.providers.openrouter import OpenRouterProvider
from chapter8.trajectory_verifier.llm_judge import LLMJudge

# Load the SFT‑tuned checkpoint

model_name = "sft_output/checkpoint"
provider = OpenRouterProvider(model=model_name, api_key="YOUR_KEY")

# Define a simple reward: +1 if the verifier says the action succeeded

judge = LLMJudge(provider=provider, endpoint="https://verifier.example.com")

def reward_fn(state, action, next_state):
    verdict = judge.evaluate(state, action, next_state)
    return 1.0 if verdict["success"] else 0.0

# RL loop (pseudo‑code, actual RL implementation lives in the same directory)

for episode in range(100):
    state = env.reset()
    done = False
    while not done:
        action = provider.generate(state)          # model proposes an action

        next_state, done = env.step(action)        # environment transition

        r = reward_fn(state, action, next_state)   # compute reward

        # ... update policy with REINFORCE / PPO ...

        state = next_state

```

The `LLMJudge` class in [`chapter8/trajectory-verifier/llm_judge.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/trajectory-verifier/llm_judge.py) converts verification services into scalar reward signals, enabling the policy to learn from outcome-based feedback rather than fixed examples.

### Combined Pipeline Execution

Production workflows in the repository follow a **pre-train → SFT → RL** sequence:

```bash

# 1️⃣ Pre‑train (e.g., Korean Wikipedia) – see chapter7/continued-pretraining/continued-pretrain.py

python chapter7/continued-pretraining/continued-pretrain.py --data ko_wiki

# 2️⃣ SFT on Korean Alpaca data – see chapter7/continued-pretraining/continued-pretrain.py

python chapter7/continued-pretraining/continued-pretrain.py --sft-data ko_alpaca

# 3️⃣ RL fine‑tuning (e.g., tool‑use RL) – see chapter8/trajectory-verifier/run_experiment_8_1.py

python chapter8/trajectory-verifier/run_experiment_8_1.py

```

## Why Combine SFT and RL?

Integrating both stages leverages complementary strengths. **Stabilization** occurs because SFT provides a strong initial policy that reduces variance for subsequent RL updates. **Sample-efficiency** improves since RL built on an SFT-trained model converges faster than RL from scratch—the policy already understands language and basic tool syntax. Additionally, **safety** considerations favor SFT for enforcing basic constraints (e.g., avoiding toxic outputs) before exposing the model to reward loops that might otherwise incentivize unsafe behavior.

## Summary

- **SFT** offers high sample efficiency and fast convergence by memorizing fixed distributions from human-written examples, making it ideal for deterministic tasks with clear ground truth.
- **RL** enables generalization beyond training data by optimizing for reward signals, effectively internalizing tools as native capabilities rather than token patterns.
- **Implementation** in `bojieli/ai-agent-book` demonstrates that production pipelines typically sequence SFT before RL to stabilize training and improve sample efficiency.
- **Tool integration** differs fundamentally: SFT learns surface syntax while RL acquires semantic understanding through outcome-based rewards.
- **Compute tradeoffs** favor SFT for resource-constrained scenarios, while RL justifies higher costs for complex, non-differentiable objectives.

## Frequently Asked Questions

### When should I choose SFT over RL for LLM post-training?

Choose **SFT** when you have high-quality labeled datasets and need deterministic, predictable outputs for well-defined tasks. It requires significantly less compute and converges faster than RL. According to the `ai-agent-book` source code, SFT is preferred for multilingual instruction following and voice-cloning scenarios where the target distribution is clearly specified.

### Why does RL require more compute than SFT?

RL necessitates generating many rollout episodes to estimate policy gradients and compute rewards, whereas SFT computes gradients directly against fixed targets in a single forward-backward pass. As shown in [`chapter8/trajectory-verifier/run_experiment_8_1.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/trajectory-verifier/run_experiment_8_1.py), each RL episode requires environment interaction and reward evaluation, multiplying the computational overhead compared to the supervised approach in [`chapter7/MultilingualReasoning/gpt_oss_20b_sft.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter7/MultilingualReasoning/gpt_oss_20b_sft.py).

### Can I skip SFT and train RL from scratch?

While theoretically possible, the repository demonstrates that **SFT warm-up** is critical for stabilization. Starting RL from a pre-trained base without SFT introduces high variance and slow convergence because the model lacks basic task structure. The [`chapter7/orpheus/orpheus_sft_unsloth.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter7/orpheus/orpheus_sft_unsloth.py) implementation specifically uses SFT to establish initial voice and context patterns before RL refines tool-use capabilities.

### How do tool capabilities differ between SFT and RL training?

SFT learns tool usage as **token-level patterns**—it reproduces API syntax seen in training data without guaranteed semantic understanding. In contrast, RL treats tool success as a reward signal, enabling the model to discover robust strategies and handle novel edge cases. The [`book-hu/localize_figures.py`](https://github.com/bojieli/ai-agent-book/blob/main/book-hu/localize_figures.py) visualization captures this as "SFT memorises distribution → RL generalises strategy", reflecting how RL internalizes tools as first-class actions rather than surface patterns.