# SFT vs RL for Tool Calling in LLMs: A Complete Technical Guide

> Compare SFT vs RL for LLM tool calling. Discover how SFT teaches syntax and RL optimizes strategy, with the combined SFT->RL pipeline achieving higher accuracy and lower costs. Master effective tool use.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: deep-dive
- Published: 2026-07-31

---

**Supervised Fine-Tuning (SFT) teaches Large Language Models the precise syntax of tool calls, while Reinforcement Learning (RL) optimizes the strategic decision of when to invoke those tools based on environmental feedback, with the combined SFT→RL pipeline delivering superior task accuracy and lower API costs compared to either approach in isolation.**

The `bojieli/ai-agent-book` repository outlines a three-phase post-training progression—pre-training, **Supervised Fine-Tuning (SFT)**, and **Reinforcement Learning (RL)**—designed to transform base models into agents capable of reliable tool invocation. Understanding the distinct roles of **SFT vs RL for tool calling in LLMs** is essential for building systems that not only generate valid API requests but also optimize their decision-making strategies to minimize unnecessary calls.

## Supervised Fine-Tuning (SFT) for Tool Calling

SFT aligns the model to follow human instructions and produce deterministic, correct API payloads. It operates on curated datasets of *prompt → tool-call* pairs, allowing the model to learn exact syntax, required arguments, and error-handling patterns.

### Syntax Acquisition and Training Parameters

The SFT phase guarantees the model can generate syntactically valid tool requests (e.g., `search(query=…)` or `retrieve(url=…)`). According to the source code, this phase typically runs for **1–3 epochs** on a few-gigabyte dataset, completing in **minutes to hours** on modern GPUs. This provides a strong baseline where the model already knows *how* to call a tool, significantly reducing the exploration space required for subsequent RL training.

### Implementation in train_sft.py

The repository implements SFT training in [`chapter7/SFTvsRL/train_sft.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter7/SFTvsRL/train_sft.py) using the `SFTTrainer` class. The trainer consumes JSONL-formatted datasets where each line contains a prompt and completion pair:

```python

# 1️⃣ SFT – fine‑tune a base model to emit valid tool calls

from agentbook.providers.models import load_model
from agentbook.trainers.sft import SFTTrainer

model = load_model("meta-llama/Meta-Llama-3-8B-Instruct")
trainer = SFTTrainer(
    model=model,
    dataset="data/sft_tool_calls.jsonl",   # {"prompt": "...", "completion": "..."}

    max_steps=2000,
    learning_rate=2e-5,
)
trainer.run()

```

## Reinforcement Learning (RL) for Tool Calling

RL further improves sample efficiency by adapting the model to *use* tools as part of a decision-making loop. Unlike SFT's next-token prediction objective, RL training receives a scalar reward reflecting the *outcome* of the tool call (e.g., retrieval success, computation correctness).

### Reward Design and Strategic Optimization

The RL phase encourages the model to *decide* when to call a tool versus answering directly. Reward functions are defined per-experiment in the codebase, incorporating metrics such as **retrieval-accuracy**, **cost-penalty**, and **step-efficiency**. This optimization targets long-term objectives like reducing total API calls and achieving higher task accuracy while obeying safety constraints.

### The RL Training Loop

Implemented in [`chapter7/SFTvsRL/train_rl.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter7/SFTvsRL/train_rl.py), the RL trainer wraps the model in a loop that executes generated tool calls, captures responses, and feeds scalar rewards back to the policy optimizer. Training uses algorithms like **PPO** or **DPO** and typically requires **several days** because each episode triggers real tool execution or simulation:

```python

# 2️⃣ RL – reinforce the SFT‑pre‑trained model to decide *when* to call tools

from agentbook.trainers.rl import RLTrainer
from agentbook.envs.tool_env import ToolEnv   # wraps tool execution & reward logic

env = ToolEnv(
    toolset=["search", "calculator"],
    reward_fn=lambda obs, act, resp: 1.0 if resp["success"] else -0.5,
)
rl_trainer = RLTrainer(
    model=model,          # the SFT‑fine‑tuned checkpoint

    env=env,
    algorithm="ppo",      # PPO or DPO (see chapter7 experiments)

    total_timesteps=500_000,
)
rl_trainer.run()

```

## Why You Need Both: The SFT → RL Pipeline

SFT provides the *syntax* and *basic competence* for reliable tool calling, while RL enables *strategic* decision-making based on feedback. Without SFT, RL wastes massive sample budgets exploring invalid request formats. Without RL, the model calls tools in every situation, hurting efficiency and over-using costly APIs.

### Experimental Results from Chapter 7

The book's **Chapter 7** (documented in [`chapter7/README.en.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter7/README.en.md)) provides concrete experiments contrasting three approaches:

- **SFT-only**: Agents produce correct tool calls but cannot *choose* when to call a tool, leading to unnecessary API usage and inflated token counts.
- **RL-only**: Agents eventually learn valid tool calls, but the early training phase struggles with malformed requests, causing high initial failure rates.
- **SFT→RL**: Agents inherit the best of both worlds—starting with low error rates from SFT, then quickly specializing to call tools only when beneficial, yielding **higher overall task accuracy** and **lower average token usage**.

## Key Architectural Components

### The MCP Protocol

The repository defines a universal **MCP** (Model-Client-Protocol) in [`chapter4/perception-tools/src/base.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter4/perception-tools/src/base.py) that standardizes JSON-based tool requests across perception, execution, and collaboration tools. This protocol ensures consistent interface definitions between the SFT training data and the RL environment.

### Dataset Generation Pipeline

For teams building their own SFT datasets, [`chapter7/cot-distillation/README.en.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter7/cot-distillation/README.en.md) demonstrates how to generate *instruction-to-tool* pairs via chain-of-thought distillation. This provides a practical method to bootstrap the curated datasets required for the initial SFT phase.

## Summary

- **SFT** teaches LLMs the syntactic structure and argument schemas required for valid tool calls through supervised learning on curated prompt-completion pairs.
- **RL** optimizes the strategic policy of *when* to invoke tools, using scalar rewards from environmental feedback to minimize unnecessary API calls.
- The **SFT→RL pipeline** combines both approaches, eliminating the high failure rates of RL-only training and the inefficiency of SFT-only agents.
- Implementation files [`chapter7/SFTvsRL/train_sft.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter7/SFTvsRL/train_sft.py) and [`chapter7/SFTvsRL/train_rl.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter7/SFTvsRL/train_rl.py) provide the complete training loops using `SFTTrainer` and `RLTrainer` classes respectively.
- The **MCP protocol** in [`chapter4/perception-tools/src/base.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter4/perception-tools/src/base.py) standardizes tool interfaces across both training phases.

## Frequently Asked Questions

### Can RL alone teach tool calling without SFT?

While possible, RL-only training is highly inefficient. According to the `bojieli/ai-agent-book` experiments, RL-only agents struggle with malformed calls during early training phases, wasting significant sample budget exploring invalid request formats that SFT eliminates within the first few epochs.

### How long does each training phase take in the SFT vs RL pipeline?

SFT typically completes in **minutes to hours** (1–3 epochs on modern GPUs), while RL training requires **several days** because each episode involves real or simulated tool execution. The SFT phase provides a warm start that significantly reduces the total RL training time needed to reach convergence.

### What reward functions work best for RL-based tool calling?

Effective reward functions combine **task success metrics** (such as retrieval accuracy or computation correctness) with **efficiency penalties** (such as API call cost or step count). The `ToolEnv` class in the repository allows custom reward functions to be injected via the `reward_fn` parameter to optimize for specific use cases.

### Where can I find the complete experimental benchmarks comparing these methods?

Detailed experimental results comparing SFT-only, RL-only, and combined SFT→RL approaches are documented in [`chapter7/README.en.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter7/README.en.md), including specific metrics for task accuracy, token usage, and API call efficiency across different benchmark scenarios.