SFT vs RL for AI Agent Post-Training: Core Differences and Implementation Guide

Supervised Fine-Tuning (SFT) trains AI agents to memorize fixed input-output mappings through maximum likelihood estimation, while Reinforcement Learning (RL) optimizes for expected reward, enabling generalization to novel strategies beyond the training demonstrations.

Training large language model (LLM) agents typically follows a three-stage pipeline—pre-training, supervised fine-tuning (SFT), and reinforcement learning (RL)—each employing distinct optimization objectives. According to the bojieli/ai-agent-book repository, understanding the fundamental differences between SFT and RL for AI Agent post-training is critical for building robust agents that balance instruction fidelity with adaptive decision-making.

What SFT Does in AI Agent Post-Training

Optimization Objective and Mechanism

Supervised Fine-Tuning optimizes for maximum likelihood estimation on labeled demonstration pairs. As implemented in book-en/chapter7.md, the model is presented with input-output pairs and learns to predict the next token exclusively on the response portion, effectively maximizing the probability of the labeled answer given the prompt.

This approach treats training as a supervised learning problem where the model memorizes the exact mapping from specific prompts to demonstrated outputs. The training signal consists of a single standard answer per token position, derived from human-written or curated demonstrations.

Practical Effects

SFT yields memorization of the demonstrated behavior, producing agents with excellent instruction-following capabilities and strict output format consistency. According to the source analysis, this method offers high sample efficiency—often requiring only thousands of examples—and converges quickly with high training stability. The computational cost remains relatively low, typically requiring hours to days of training time.

Typical Use Cases

SFT excels at solidifying the form of an agent before any trial-and-error learning begins. Common applications include enforcing JSON schema compliance, standardizing tool-call syntax, and establishing consistent output styles. In book-en/chapter7.md (lines 9-12), the authors emphasize that SFT is best suited for stable environments where demonstration data accurately represents the desired behavior.

What RL Does in AI Agent Post-Training

Reward-Based Optimization

Reinforcement Learning shifts the optimization target from likelihood maximization to expected reward maximization. Instead of mimicking fixed answers, the model generates its own responses and receives scalar reward signals (or penalties) from a task-specific reward function. As detailed in book-en/chapter7.md (lines 60-66), the policy updates based on the cumulative reward of self-generated trajectories rather than comparison against a reference answer.

Generalization Capabilities

Unlike SFT's memorization approach, RL enables generalization by learning transferable decision-making strategies. The model can discover behaviors that never appeared in any training demonstration, optimizing for outcomes rather than output similarity. This proves particularly valuable when exhaustive demonstration writing would be prohibitive, but cheap verification signals exist—such as test-case passing for code generation or numeric correctness for mathematical reasoning.

Training Characteristics

RL exhibits lower sample efficiency, often requiring tens to hundreds of times more data than SFT, and demonstrates lower training stability with a propensity for oscillation. However, under distribution shift, RL proves more robust: while SFT applies old answers and suffers performance drops, RL agents re-solve problems using the learned strategy.

Core Differences Between SFT and RL

The fundamental distinctions between these post-training methods span multiple dimensions:

• Optimization Objective: SFT maximizes the probability of labeled answers (Maximum Likelihood), whereas RL maximizes expected cumulative reward.
• Training Signal: SFT relies on single standard answers per token; RL utilizes multiple self-generated responses evaluated by reward functions.
• Data Requirements: SFT requires "Input → Output" demonstration pairs; RL requires only task definitions paired with reward functions (no standard answers).
• Learned Behavior: SFT encodes fixed input-to-output mappings (memorization); RL learns transferable decision-making strategies (generalization).
• Sample Efficiency: SFT achieves high efficiency with thousands of examples; RL demands significantly larger datasets.
• Training Stability: SFT converges quickly and stably; RL requires careful tuning and exhibits oscillation risks.

These comparisons are systematically documented in Table 7-2 of book-en/chapter7.md (lines 78-88).

Why SFT Must Precede RL in Agent Training

The industry standard pipeline—SFT first, then RL—is not arbitrary. RL requires parsable, well-formed outputs (such as valid JSON tool calls) to compute meaningful rewards. Without prior SFT, a model's raw generations are often malformed or structurally inconsistent, making reward estimation impossible or noisy.

As stated in book-en/chapter7.md (lines 64-67), SFT establishes the form while RL refines the spirit. SFT provides the stable interface and output structure necessary for RL to optimize content and strategy effectively. Attempting RL without this foundation typically results in training instability and poor reward signal quality.

Implementation Examples

SFT with LoRA: Practical Implementation

The following example from chapter7/sesame/sesame_csm_sft_unsloth.py demonstrates a minimal SFT workflow using Low-Rank Adaptation (LoRA):

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
import torch

model_name = "Qwen2.5-32B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

# LoRA configuration (rank=64, learning-rate ≈ 10× full-fine-tune)

lora_cfg = LoraConfig(r=64, lora_alpha=32, target_modules=["q_proj","v_proj"])
model = get_peft_model(model, lora_cfg)

# Dummy supervised dataset: list of {"input":…, "output":…}

train_data = [{"input":"Explain the capital of France.", "output":"Paris"}]

# Basic training loop (illustrative)

optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
for epoch in range(2):
    for sample in train_data:
        ids = tokenizer(sample["input"], return_tensors="pt").input_ids
        labels = tokenizer(sample["output"], return_tensors="pt").input_ids
        # Mask the prompt part

        loss_mask = torch.ones_like(labels)
        loss = model(input_ids=ids, labels=labels).loss
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

This implementation illustrates the SFT workflow described in book-en/chapter7.md (lines 58-59), utilizing loss masking to compute gradients only on the response tokens.

RL Training with the verl Framework

For the RL stage, the verl framework provides the training infrastructure. The following skeleton from chapter8/gaia-experience/AWorld/train/adapter/verl/aworld_agent_loop.py demonstrates an RLHF loop:

from verl import Trainer, PPOConfig, RewardModel

# Load a base LLM (already SFT-ed)

base_model = ...  # e.g., Qwen2.5-32B-Instruct fine-tuned on SFT data

reward = RewardModel.load("path/to/reward_model")  # computes scalar reward

cfg = PPOConfig(
    epochs=3,
    lr=1e-5,
    clip_range=0.2,
    target_kl=0.01,
)

trainer = Trainer(model=base_model, reward_model=reward, config=cfg)

# Simulated environment: each step returns a prompt and the model's response

def env_step(prompt):
    response = base_model.generate(prompt, max_new_tokens=256)
    reward_score = reward.compute(prompt, response)
    return response, reward_score

# RL training loop

for iteration in range(1000):
    prompt = "Solve 24 = ? using given numbers."
    resp, r = env_step(prompt)
    trainer.step(prompt, resp, r)

This approach uses Proximal Policy Optimization (PPO) to refine the SFT-trained base model, as referenced in chapter7/README.en.md (lines 14-16).

End-to-End Workflow

The complete two-stage pipeline can be executed sequentially:

# 1️⃣ SFT (LoRA)

python -m scripts.sft \
    --model Qwen2.5-32B-Instruct \
    --lora-rank 64 \
    --data ./sft_data.jsonl \
    --output ./sft_adapter

# 2️⃣ RLHF (PPO via verl)

python -m verl.train_ppo \
    --base-model ./sft_adapter \
    --reward-model ./reward_checkpoint \
    --env ./my_env.py \
    --output ./rl_adapter

This workflow reflects the "SFT first, then RL" paradigm emphasized throughout book-en/chapter7.md (lines 64-67).

Summary

• SFT maximizes the likelihood of labeled answers through maximum likelihood estimation, producing memorized input-output mappings with high sample efficiency and training stability.
• RL maximizes expected reward from scalar feedback signals, enabling generalization to novel strategies but requiring significantly more data and careful hyperparameter tuning.
• Data requirements differ fundamentally: SFT needs input-output demonstration pairs, while RL requires only task definitions and reward functions.
• Pipeline ordering is critical: SFT must precede RL to establish parsable output formats (the "form"), allowing RL to optimize decision-making strategies (the "spirit").
• Use SFT for formatting, protocol adherence, and stable environments; use RL for outcome-driven tasks with cheap verification signals and unseen scenarios.

Frequently Asked Questions

What is the fundamental technical difference between SFT and RL for AI agents?

Supervised Fine-Tuning optimizes for maximum likelihood estimation on fixed input-output pairs, training the model to reproduce demonstrated answers token-by-token. Reinforcement Learning instead optimizes for expected cumulative reward, updating the policy based on scalar feedback from self-generated responses rather than comparison to reference answers.

Should I use SFT or RL for my AI agent project?

Choose SFT when you have high-quality demonstration data and need consistent output formatting or protocol adherence. Choose RL when you have access to reliable reward signals (such as unit test passing or numeric correctness) but lack comprehensive demonstrations, or when you need the agent to discover novel strategies beyond the training data.

Why can't I train an AI agent with RL alone without SFT?

RL requires well-formed, parsable outputs to compute meaningful rewards—such as valid JSON tool calls or structured command syntax. Without prior SFT, model generations are often malformed or inconsistent, making reward estimation unreliable or impossible. SFT establishes the necessary output structure before RL optimizes the content.

How do SFT and RL handle distribution shift differently?

Under distribution shift, SFT-trained agents attempt to apply memorized answers from training, causing performance degradation when inputs diverge from demonstrations. RL-trained agents employ learned decision-making strategies to re-solve problems, making them more robust to novel situations and input variations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}