LLM Post-Training: SFT vs RL Tradeoffs and Implementation Strategies

Supervised Fine-Tuning (SFT) excels at sample-efficient memorization of fixed distributions, while Reinforcement Learning (RL) enables generalization beyond training data through reward optimization, with production pipelines typically combining both stages.

After pre-training, large language models require post-training to align with specific tasks and user preferences. This article examines the LLM post-training SFT vs RL tradeoffs through the lens of the bojieli/ai-agent-book repository, analyzing actual implementation files and training pipelines used in multilingual reasoning and tool-use scenarios.

Core Paradigm Differences

Supervised Fine-Tuning: Fixed Distribution Learning

SFT aligns models to human-written ground-truth examples through direct supervision. According to the repository's visualization in book-hu/localize_figures.py (lines 1122-1126), SFT memorises the distribution present in the training data. This approach requires only a few thousand labeled examples to achieve strong performance because the loss function directly compares predictions against fixed targets.

Reinforcement Learning: Reward Optimization

RL optimizes models to maximize a reward signal reflecting task performance, such as tool-call success or human feedback. Unlike SFT, RL generalises the learned strategy beyond exact training trajectories, enabling the model to discover novel solutions to achieve rewards. However, this comes at the cost of sample efficiency—RL requires many rollout episodes to estimate gradients reliably.

Behavioral Tradeoffs in Practice

Memorization vs Generalization

The dichotomy is stark: SFT tends to reproduce patterns it has seen, leading to overfitting on the supervised set, while RL encourages adaptation to unseen situations. As implemented in bojieli/ai-agent-book, SFT provides deterministic behavior ideal for well-defined tasks, whereas RL refines the decision-making process for complex, non-differentiable objectives.

Tool-Call Internalization

SFT learns tool usage as token-level patterns—the model emits correct API syntax without necessarily understanding underlying semantics. In contrast, RL can convert tool-calling into a native capability: when the model receives reward feedback upon tool success, it learns to treat the tool as part of its action space. This distinction is evident in chapter7/orpheus/orpheus_sft_unsloth.py, where initial SFT establishes voice-cloning patterns before RL stages improve tool-use integration.

Implementation Examples from ai-agent-book

The repository provides concrete implementations illustrating both paradigms and their combination.

SFT Training with trl.SFTTrainer

The script chapter7/MultilingualReasoning/gpt_oss_20b_sft.py demonstrates efficient supervised fine-tuning:

# File: chapter7/MultilingualReasoning/gpt_oss_20b_sft.py

from trl import SFTTrainer, SFTConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "gpt2"                     # replace with a 20‑B checkpoint in practice

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# 𝑆𝑓𝑡 𝑫𝑎𝑡𝑎𝑠𝑒𝑡 – a JSONL file where each entry has {"prompt":…, "completion":…}

train_dataset = ...  # see utils in this file for loading

training_args = SFTConfig(
    output_dir="sft_output",
    per_device_train_batch_size=4,
    num_train_epochs=2,
    learning_rate=5e-5,
    logging_steps=10,
)

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    args=training_args,
    train_dataset=train_dataset,
)

trainer.train()

This configuration achieves fast convergence with minimal compute, making it suitable for deterministic instruction-following tasks.

RL Training with Trajectory Verification

For RL implementations, chapter8/trajectory-verifier/run_experiment_8_1.py shows how to optimize against non-differentiable rewards:

# File: chapter8/trajectory-verifier/run_experiment_8_1.py

from agentbook.providers.openrouter import OpenRouterProvider
from chapter8.trajectory_verifier.llm_judge import LLMJudge

# Load the SFT‑tuned checkpoint

model_name = "sft_output/checkpoint"
provider = OpenRouterProvider(model=model_name, api_key="YOUR_KEY")

# Define a simple reward: +1 if the verifier says the action succeeded

judge = LLMJudge(provider=provider, endpoint="https://verifier.example.com")

def reward_fn(state, action, next_state):
    verdict = judge.evaluate(state, action, next_state)
    return 1.0 if verdict["success"] else 0.0

# RL loop (pseudo‑code, actual RL implementation lives in the same directory)

for episode in range(100):
    state = env.reset()
    done = False
    while not done:
        action = provider.generate(state)          # model proposes an action

        next_state, done = env.step(action)        # environment transition

        r = reward_fn(state, action, next_state)   # compute reward

        # ... update policy with REINFORCE / PPO ...

        state = next_state

The LLMJudge class in chapter8/trajectory-verifier/llm_judge.py converts verification services into scalar reward signals, enabling the policy to learn from outcome-based feedback rather than fixed examples.

Combined Pipeline Execution

Production workflows in the repository follow a pre-train → SFT → RL sequence:

# 1️⃣ Pre‑train (e.g., Korean Wikipedia) – see chapter7/continued-pretraining/continued-pretrain.py

python chapter7/continued-pretraining/continued-pretrain.py --data ko_wiki

# 2️⃣ SFT on Korean Alpaca data – see chapter7/continued-pretraining/continued-pretrain.py

python chapter7/continued-pretraining/continued-pretrain.py --sft-data ko_alpaca

# 3️⃣ RL fine‑tuning (e.g., tool‑use RL) – see chapter8/trajectory-verifier/run_experiment_8_1.py

python chapter8/trajectory-verifier/run_experiment_8_1.py

Why Combine SFT and RL?

Integrating both stages leverages complementary strengths. Stabilization occurs because SFT provides a strong initial policy that reduces variance for subsequent RL updates. Sample-efficiency improves since RL built on an SFT-trained model converges faster than RL from scratch—the policy already understands language and basic tool syntax. Additionally, safety considerations favor SFT for enforcing basic constraints (e.g., avoiding toxic outputs) before exposing the model to reward loops that might otherwise incentivize unsafe behavior.

Summary

• SFT offers high sample efficiency and fast convergence by memorizing fixed distributions from human-written examples, making it ideal for deterministic tasks with clear ground truth.
• RL enables generalization beyond training data by optimizing for reward signals, effectively internalizing tools as native capabilities rather than token patterns.
• Implementation in bojieli/ai-agent-book demonstrates that production pipelines typically sequence SFT before RL to stabilize training and improve sample efficiency.
• Tool integration differs fundamentally: SFT learns surface syntax while RL acquires semantic understanding through outcome-based rewards.
• Compute tradeoffs favor SFT for resource-constrained scenarios, while RL justifies higher costs for complex, non-differentiable objectives.

Frequently Asked Questions

When should I choose SFT over RL for LLM post-training?

Choose SFT when you have high-quality labeled datasets and need deterministic, predictable outputs for well-defined tasks. It requires significantly less compute and converges faster than RL. According to the ai-agent-book source code, SFT is preferred for multilingual instruction following and voice-cloning scenarios where the target distribution is clearly specified.

Why does RL require more compute than SFT?

RL necessitates generating many rollout episodes to estimate policy gradients and compute rewards, whereas SFT computes gradients directly against fixed targets in a single forward-backward pass. As shown in chapter8/trajectory-verifier/run_experiment_8_1.py, each RL episode requires environment interaction and reward evaluation, multiplying the computational overhead compared to the supervised approach in chapter7/MultilingualReasoning/gpt_oss_20b_sft.py.

Can I skip SFT and train RL from scratch?

While theoretically possible, the repository demonstrates that SFT warm-up is critical for stabilization. Starting RL from a pre-trained base without SFT introduces high variance and slow convergence because the model lacks basic task structure. The chapter7/orpheus/orpheus_sft_unsloth.py implementation specifically uses SFT to establish initial voice and context patterns before RL refines tool-use capabilities.

How do tool capabilities differ between SFT and RL training?

SFT learns tool usage as token-level patterns—it reproduces API syntax seen in training data without guaranteed semantic understanding. In contrast, RL treats tool success as a reward signal, enabling the model to discover robust strategies and handle novel edge cases. The book-hu/localize_figures.py visualization captures this as "SFT memorises distribution → RL generalises strategy", reflecting how RL internalizes tools as first-class actions rather than surface patterns.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}