SFT vs RL for Training AI Agents: Key Differences Explained

Supervised Fine-Tuning (SFT) trains AI agents to memorize specific input-output mappings using labeled demonstrations, while Reinforcement Learning (RL) optimizes for expected reward through trial-and-error, enabling generalization to novel strategies beyond the training data.

Training large language model (LLM) based agents typically follows a three-stage pipeline—pre-training, supervised fine-tuning (SFT), and reinforcement learning (RL)—each serving distinct optimization objectives according to the bojieli/ai-agent-book source code. Understanding the precise difference between SFT and RL is critical for practitioners designing agent training workflows, as these methods differ fundamentally in their training signals, sample efficiency, and generalization capabilities.

What is Supervised Fine-Tuning (SFT)?

Optimization Objective and Training Signal

SFT maximizes the likelihood of a labeled answer using maximum-likelihood estimation. In book-en/chapter7.md, the training process is described as presenting the model with input-output demonstration pairs, where the model learns to "predict the next token" only on the response part while masking the prompt portion (lines 47-53).

The training signal consists of single standard answers per token, requiring the model to reproduce exact mappings from given prompts to demonstrated outputs. This approach yields high sample efficiency—often requiring only thousands of examples—and converges quickly with stable training dynamics.

What SFT Learns: Memorization

SFT effectively memorizes fixed input-to-output mappings. As implemented in the repository, this creates excellent instruction-following behavior and output format consistency, making it ideal for solidifying the "form" of an agent—including output style, JSON schema, and tool-call syntax—before any trial-and-error learning begins (chapter7.md, lines 9-12).

The chapter7/sesame/sesame_csm_sft_unsloth.py file demonstrates this practically using LoRA (Low-Rank Adaptation):

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model

model_name = "Qwen2.5-32B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

# LoRA configuration for efficient SFT

lora_cfg = LoraConfig(r=64, lora_alpha=32, target_modules=["q_proj","v_proj"])
model = get_peft_model(model, lora_cfg)

# Training on input-output pairs

train_data = [{"input":"Explain the capital of France.", "output":"Paris"}]

for epoch in range(2):
    for sample in train_data:
        ids = tokenizer(sample["input"], return_tensors="pt").input_ids
        labels = tokenizer(sample["output"], return_tensors="pt").input_ids
        # Loss computation on response tokens only

        loss = model(input_ids=ids, labels=labels).loss
        loss.backward()
        optimizer.step()

What is Reinforcement Learning (RL)?

Optimization Objective and Reward Function

RL maximizes expected reward from a task-specific reward function rather than mimicking labeled outputs. According to chapter7.md (lines 60-66), the model generates its own responses, receives a scalar reward (or penalty), and updates its policy to favor high-reward trajectories.

Unlike SFT's single correct answer approach, RL operates on multiple self-generated responses evaluated against a reward signal. This makes RL suitable for tasks where writing exhaustive demonstrations would be prohibitive, but where cheap verification signals exist—such as test-case passing or numeric correctness.

What RL Learns: Generalization

RL enables the model to generalize by learning transferable decision-making strategies rather than reproducing specific answers. As noted in the source analysis (chapter7.md, lines 70-76), RL can discover behaviors that never appeared in any demonstration, making it robust to distribution shifts where applying old answers would fail.

The chapter8/gaia-experience/AWorld/train/adapter/verl/aworld_agent_loop.py file illustrates this with the verl framework:

from verl import Trainer, PPOConfig, RewardModel

base_model = ...  # SFT-ed checkpoint

reward = RewardModel.load("path/to/reward_model")

cfg = PPOConfig(epochs=3, lr=1e-5, clip_range=0.2, target_kl=0.01)
trainer = Trainer(model=base_model, reward_model=reward, config=cfg)

def env_step(prompt):
    response = base_model.generate(prompt, max_new_tokens=256)
    reward_score = reward.compute(prompt, response)
    return response, reward_score

# RL training loop exploring possible responses

for iteration in range(1000):
    prompt = "Solve 24 = ? using given numbers."
    resp, r = env_step(prompt)
    trainer.step(prompt, resp, r)

Core Differences Between SFT and RL

The book-en/chapter7.md file contains Table 7-2 (lines 78-88) detailing the architectural distinctions:

Dimension	SFT	RL
Optimization Objective	Maximize probability of labeled answer (Maximum Likelihood)	Maximize expected reward
Training Signal	Single standard answer per token	Multiple self-generated responses + reward
Data Form	"Input → Output" demonstration pairs	Task + Reward function (no standard answer)
What Is Learned	Fixed input-to-output mapping (memorization)	Transferable decision-making strategy (generalization)
Under Distribution Shift	Applies old answer → performance drops	Re-solves using same strategy → more stable
Sample Efficiency	High (thousands of examples)	Low (often tens-to-hundreds× SFT data)
Training Stability	High, converges quickly	Low, prone to oscillation
Best Suited For	Formatting, protocol, stable environments	Unseen scenarios, outcome-driven tasks

Why SFT Must Precede RL

RL requires parsable outputs to compute rewards—such as valid JSON tool calls or structured responses. Without SFT, raw model generations are often malformed, making reward estimation impossible.

According to chapter7.md (lines 64-67), the industry standard "SFT first, then RL" pipeline exists because SFT establishes the form (output structure and syntax), while RL refines the spirit (decision-making quality). Attempting RL without SFT typically results in training instability due to invalid action spaces.

End-to-End Workflow Example

The complete pipeline from the repository demonstrates this sequencing:

# Stage 1: SFT with LoRA

python -m scripts.sft \
    --model Qwen2.5-32B-Instruct \
    --lora-rank 64 \
    --data ./sft_data.jsonl \
    --output ./sft_adapter

# Stage 2: RLHF with PPO via verl

python -m verl.train_ppo \
    --base-model ./sft_adapter \
    --reward-model ./reward_checkpoint \
    --env ./my_env.py \
    --output ./rl_adapter

Summary

• SFT optimizes for maximum likelihood on labeled demonstrations, creating memorized input-output mappings with high sample efficiency and stable convergence.
• RL optimizes for expected reward, enabling generalization to novel strategies through trial-and-error exploration, though requiring significantly more data and careful tuning.
• Sequential application is mandatory: SFT establishes parsable output formats required for RL reward computation, following the "form first, spirit second" paradigm documented in book-en/chapter7.md.
• Implementation specifics include LoRA-based SFT in sesame_csm_sft_unsloth.py and PPO-based RL in aworld_agent_loop.py using the verl framework.

Frequently Asked Questions

Can you train an AI agent with RL alone without SFT?

Training with RL alone is generally impractical for LLM-based agents. According to the bojieli/ai-agent-book source code, RL requires parsable outputs (like valid JSON tool calls) to compute rewards. Without SFT, models generate malformed outputs that prevent reward estimation, leading to training failure. SFT first establishes the syntax and structure that makes RL training possible.

Why does RL require more training data than SFT?

RL requires tens to hundreds of times more data than SFT because it must explore multiple response trajectories to estimate which actions lead to higher rewards. While SFT learns directly from single correct demonstrations (maximum likelihood), RL generates numerous candidate responses per prompt and learns from scalar reward signals, necessitating extensive sampling to reduce variance in policy gradients.

What happens if you apply RL before distribution shift versus SFT?

Under distribution shift, SFT-trained agents typically drop in performance because they apply memorized mappings from the training distribution. RL-trained agents demonstrate greater stability because they learn transferable decision-making strategies rather than specific answers. When encountering novel scenarios, RL agents can re-solve problems using their learned strategy, whereas SFT agents attempt to reproduce inappropriate historical outputs.

Which tasks are best suited for SFT versus RL?

SFT excels at formatting, protocol adherence, and stable environments where correct answers are easily demonsterable—such as outputting specific JSON schemas or following conversation templates. RL is superior for outcome-driven tasks with cheap verification signals but expensive demonstrations—such as mathematical reasoning, code generation with test cases, or multi-step tool use where optimal paths are difficult to annotate but final results are easily scored.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}