SFT vs RL for Tool Calling in LLMs: A Complete Technical Guide

Supervised Fine-Tuning (SFT) teaches Large Language Models the precise syntax of tool calls, while Reinforcement Learning (RL) optimizes the strategic decision of when to invoke those tools based on environmental feedback, with the combined SFT→RL pipeline delivering superior task accuracy and lower API costs compared to either approach in isolation.

The bojieli/ai-agent-book repository outlines a three-phase post-training progression—pre-training, Supervised Fine-Tuning (SFT), and Reinforcement Learning (RL)—designed to transform base models into agents capable of reliable tool invocation. Understanding the distinct roles of SFT vs RL for tool calling in LLMs is essential for building systems that not only generate valid API requests but also optimize their decision-making strategies to minimize unnecessary calls.

Supervised Fine-Tuning (SFT) for Tool Calling

SFT aligns the model to follow human instructions and produce deterministic, correct API payloads. It operates on curated datasets of prompt → tool-call pairs, allowing the model to learn exact syntax, required arguments, and error-handling patterns.

Syntax Acquisition and Training Parameters

The SFT phase guarantees the model can generate syntactically valid tool requests (e.g., search(query=…) or retrieve(url=…)). According to the source code, this phase typically runs for 1–3 epochs on a few-gigabyte dataset, completing in minutes to hours on modern GPUs. This provides a strong baseline where the model already knows how to call a tool, significantly reducing the exploration space required for subsequent RL training.

Implementation in train_sft.py

The repository implements SFT training in chapter7/SFTvsRL/train_sft.py using the SFTTrainer class. The trainer consumes JSONL-formatted datasets where each line contains a prompt and completion pair:

# 1️⃣ SFT – fine‑tune a base model to emit valid tool calls

from agentbook.providers.models import load_model
from agentbook.trainers.sft import SFTTrainer

model = load_model("meta-llama/Meta-Llama-3-8B-Instruct")
trainer = SFTTrainer(
    model=model,
    dataset="data/sft_tool_calls.jsonl",   # {"prompt": "...", "completion": "..."}

    max_steps=2000,
    learning_rate=2e-5,
)
trainer.run()

Reinforcement Learning (RL) for Tool Calling

RL further improves sample efficiency by adapting the model to use tools as part of a decision-making loop. Unlike SFT's next-token prediction objective, RL training receives a scalar reward reflecting the outcome of the tool call (e.g., retrieval success, computation correctness).

Reward Design and Strategic Optimization

The RL phase encourages the model to decide when to call a tool versus answering directly. Reward functions are defined per-experiment in the codebase, incorporating metrics such as retrieval-accuracy, cost-penalty, and step-efficiency. This optimization targets long-term objectives like reducing total API calls and achieving higher task accuracy while obeying safety constraints.

The RL Training Loop

Implemented in chapter7/SFTvsRL/train_rl.py, the RL trainer wraps the model in a loop that executes generated tool calls, captures responses, and feeds scalar rewards back to the policy optimizer. Training uses algorithms like PPO or DPO and typically requires several days because each episode triggers real tool execution or simulation:

# 2️⃣ RL – reinforce the SFT‑pre‑trained model to decide *when* to call tools

from agentbook.trainers.rl import RLTrainer
from agentbook.envs.tool_env import ToolEnv   # wraps tool execution & reward logic

env = ToolEnv(
    toolset=["search", "calculator"],
    reward_fn=lambda obs, act, resp: 1.0 if resp["success"] else -0.5,
)
rl_trainer = RLTrainer(
    model=model,          # the SFT‑fine‑tuned checkpoint

    env=env,
    algorithm="ppo",      # PPO or DPO (see chapter7 experiments)

    total_timesteps=500_000,
)
rl_trainer.run()

Why You Need Both: The SFT → RL Pipeline

SFT provides the syntax and basic competence for reliable tool calling, while RL enables strategic decision-making based on feedback. Without SFT, RL wastes massive sample budgets exploring invalid request formats. Without RL, the model calls tools in every situation, hurting efficiency and over-using costly APIs.

Experimental Results from Chapter 7

The book's Chapter 7 (documented in chapter7/README.en.md) provides concrete experiments contrasting three approaches:

• SFT-only: Agents produce correct tool calls but cannot choose when to call a tool, leading to unnecessary API usage and inflated token counts.
• RL-only: Agents eventually learn valid tool calls, but the early training phase struggles with malformed requests, causing high initial failure rates.
• SFT→RL: Agents inherit the best of both worlds—starting with low error rates from SFT, then quickly specializing to call tools only when beneficial, yielding higher overall task accuracy and lower average token usage.

Key Architectural Components

The MCP Protocol

The repository defines a universal MCP (Model-Client-Protocol) in chapter4/perception-tools/src/base.py that standardizes JSON-based tool requests across perception, execution, and collaboration tools. This protocol ensures consistent interface definitions between the SFT training data and the RL environment.

Dataset Generation Pipeline

For teams building their own SFT datasets, chapter7/cot-distillation/README.en.md demonstrates how to generate instruction-to-tool pairs via chain-of-thought distillation. This provides a practical method to bootstrap the curated datasets required for the initial SFT phase.

Summary

• SFT teaches LLMs the syntactic structure and argument schemas required for valid tool calls through supervised learning on curated prompt-completion pairs.
• RL optimizes the strategic policy of when to invoke tools, using scalar rewards from environmental feedback to minimize unnecessary API calls.
• The SFT→RL pipeline combines both approaches, eliminating the high failure rates of RL-only training and the inefficiency of SFT-only agents.
• Implementation files chapter7/SFTvsRL/train_sft.py and chapter7/SFTvsRL/train_rl.py provide the complete training loops using SFTTrainer and RLTrainer classes respectively.
• The MCP protocol in chapter4/perception-tools/src/base.py standardizes tool interfaces across both training phases.

Frequently Asked Questions

Can RL alone teach tool calling without SFT?

While possible, RL-only training is highly inefficient. According to the bojieli/ai-agent-book experiments, RL-only agents struggle with malformed calls during early training phases, wasting significant sample budget exploring invalid request formats that SFT eliminates within the first few epochs.

How long does each training phase take in the SFT vs RL pipeline?

SFT typically completes in minutes to hours (1–3 epochs on modern GPUs), while RL training requires several days because each episode involves real or simulated tool execution. The SFT phase provides a warm start that significantly reduces the total RL training time needed to reach convergence.

What reward functions work best for RL-based tool calling?

Effective reward functions combine task success metrics (such as retrieval accuracy or computation correctness) with efficiency penalties (such as API call cost or step count). The ToolEnv class in the repository allows custom reward functions to be injected via the reward_fn parameter to optimize for specific use cases.

Where can I find the complete experimental benchmarks comparing these methods?

Detailed experimental results comparing SFT-only, RL-only, and combined SFT→RL approaches are documented in chapter7/README.en.md, including specific metrics for task accuracy, token usage, and API call efficiency across different benchmark scenarios.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}