Fine-Tuning Methodologies in AI Engineering From Scratch: From SFT to Production Pipelines

The later phases of AI Engineering From Scratch cover eight distinct fine-tuning methodologies ranging from supervised instruction-tuning (SFT) to parameter-efficient adapters (LoRA/QLoRA) and preference-based optimization (DPO/GRPO), culminating in production-grade pipelines that chain these techniques for deploying aligned language models.

The curriculum in rohitg00/ai-engineering-from-scratch progressively deepens its treatment of model alignment across phases 10 through 19. These fine-tuning methodologies transform base language models into instruction-following assistants and specialized classifiers while mitigating catastrophic forgetting through token masking, low-rank adaptation, and quantized training.

Supervised Fine-Tuning (SFT) and Instruction Alignment

The foundation of the curriculum rests on Supervised Fine-Tuning (SFT), which converts a base language model into an instruction-following assistant by training on paired instruction → response examples.

In phases/10-llms-from-scratch/06-instruction-tuning-sft/docs/en.md, the implementation builds an Alpaca-style training loop that masks instruction tokens with ignore_index=-100 to preserve pre-trained knowledge while teaching the model to generate responses. The collate function specifically removes the instruction portion from the loss calculation, ensuring gradients only update the model's ability to complete prompts rather than memorize instructions.

Phase 19 reinforces this methodology in phases/19-capstone-projects/39-instruction-tuning-sft/docs/en.md with a more extensive implementation using approximately 200 examples and richer evaluation analytics. This lesson demonstrates how to scale SFT without catastrophic forgetting by maintaining low learning rates and gradient clipping.

Parameter-Efficient Fine-Tuning with LoRA and QLoRA

The curriculum transitions to Parameter-Efficient Fine-Tuning (PEFT) in Phase 11, specifically focusing on Low-Rank Adaptation (LoRA).

As documented in phases/11-llm-engineering/08-fine-tuning-lora/docs/en.md, LoRA freezes base model weights and injects trainable low-rank adapter matrices (A and B) into each linear layer, transforming the architecture via W → W + BA. This approach achieves approximately 99% of full fine-tuning quality while using less than 6GB VRAM for a 7B parameter model.

The instruction extends to QLoRA in phases/11-llm-engineering/08-fine-tuning-lora/outputs/skill-fine-tuning-guide.md, which loads the base model in 4-bit NF4 precision, trains adapters in fp16, and enables fine-tuning 70B parameter models on a single 24GB GPU. The quantization layer processes weights before the LoRA adapters, with loss computed on de-quantized logits to preserve quality while drastically cutting memory usage.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
import torch

model_name = "meta-llama/Meta-Llama-3-8B"
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Load base model in fp16 for memory efficiency

base = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto",
)

# Configure rank-16 LoRA adapter

lora_cfg = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(base, lora_cfg)

# Training loop

from torch.utils.data import DataLoader
train_loader = DataLoader(train_dataset, batch_size=4, shuffle=True)

optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
model.train()

for epoch in range(3):
    for batch in train_loader:
        inputs = tokenizer(batch["prompt"], return_tensors="pt", padding=True).to("cuda")
        labels = tokenizer(batch["response"], return_tensors="pt", padding=True).to("cuda")["input_ids"]
        
        outputs = model(**inputs, labels=labels)
        loss = outputs.loss
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

Preference-Based Optimization: DPO and GRPO

Moving beyond supervised learning, the curriculum introduces Direct Preference Optimization (DPO) in phases/10-llms-from-scratch/08-dpo/docs/en.md. DPO eliminates the need for a separate reward model by directly optimizing the policy against pairwise preference datasets using a contrastive loss.

The implementation calculates loss as logsigmoid(p_ref - p_alt), where p_ref represents the log-probability of the preferred completion and p_alt represents the rejected completion. This architectural approach treats the policy as a standard transformer updated via contrastive learning from human-ranked pairs.

Phase 19 extends this with Generalized Reward-Based Optimization (GRPO) in the end-to-end pipeline, creating a hybrid approach that combines SFT, DPO, and optional GRPO stages for production-grade alignment.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B", 
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")

def dpo_loss(logits_ref, logits_alt):
    # Contrastive loss on log-probabilities

    return -torch.nn.functional.logsigmoid(logits_ref - logits_alt).mean()

optimizer = torch.optim.AdamW(model.parameters(), lr=5e-5)

for epoch in range(2):
    for pref in pref_dataset:  # entries contain (prompt, chosen, rejected)

        batch = tokenizer(
            [pref["prompt"] + pref["chosen"],
             pref["prompt"] + pref["rejected"]],
            return_tensors="pt", 
            padding=True
        ).to("cuda")
        
        outputs = model(**batch, labels=batch["input_ids"])
        
        # Calculate log-prob of last token (simplified)

        logp_ref = outputs.logits[:, -1, batch["input_ids"][0, -1]]
        logp_alt = outputs.logits[:, -1, batch["input_ids"][1, -1]]
        
        loss = dpo_loss(logp_ref, logp_alt)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

RLHF-Style and Classifier Fine-Tuning

The curriculum addresses RLHF-style fine-tuning in phases/09-reinforcement-learning/11-sim-to-real-transfer/docs/en.md, using Proximal Policy Optimization (PPO) to fine-tune pretrained policies on real-world datasets after simulation. This approach deliberately limits fine-tuning to a few minutes of real-world data to mitigate the sim-to-real gap without overfitting.

For classification tasks, phases/19-capstone-projects/38-classifier-finetuning/docs/en.md compares head-only versus full-model fine-tuning. The lesson rewires a pretrained LLM's head to a two-class linear layer, then evaluates three architectural strategies:

• Head-only: Freezes the transformer body, training only the classification head
• Partial: Unfreezes only the last transformer block
• Full-model: Updates all parameters for maximum accuracy

The same training loop supports all three modes via a configuration flag that toggles which parameters receive gradients, highlighting the trade-off between training speed/memory (head-only) and final accuracy (full fine-tuning).

from transformers import AutoModelForCausalLM, AutoTokenizer, Trainer, TrainingArguments

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B", 
    torch_dtype=torch.float16
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B")

def preprocess(example):
    # Format: <|im_start|>instruction<|im_sep|>response

    return tokenizer(
        f"<|im_start|>{example['instruction']}<|im_sep|>{example['response']}",
        truncation=True, 
        max_length=512
    )

train_dataset = raw_dataset.map(preprocess, remove_columns=raw_dataset.column_names)

training_args = TrainingArguments(
    output_dir="sft-checkpoint",
    per_device_train_batch_size=2,
    num_train_epochs=1,
    learning_rate=5e-5,
    fp16=True,
    logging_steps=10,
)

trainer = Trainer(
    model=model, 
    args=training_args, 
    train_dataset=train_dataset
)
trainer.train()

Production-Grade End-to-End Pipelines

The capstone phase consolidates these methodologies into a unified workflow in phases/19-capstone-projects/07-end-to-end-fine-tuning-pipeline/docs/en.md. This production pipeline chains SFT → DPO (or GRPO) → Quantization, using industry-standard tools:

• Axolotl for multi-GPU SFT
• TRL for DPO/GRPO implementation
• vLLM for optimized serving

The pipeline is driven by a declarative YAML manifest that configures each stage, mirroring real-world RLHF stacks. This architecture enables researchers to move from raw data to quantized, LoRA-adapted deployment without manual intervention between stages.

Summary

• Instruction-Tuning (SFT): Foundational methodology using Alpaca-style loops with instruction token masking (ignore_index=-100) to prevent catastrophic forgetting
• LoRA/QLoRA: Parameter-efficient approaches that freeze base weights and train low-rank adapters, with QLoRA enabling 70B model training on 24GB GPUs through 4-bit NF4 quantization
• DPO/GRPO: Preference-based optimization methods that eliminate separate reward models through contrastive losses
• Classifier Fine-Tuning: Modular architecture supporting head-only, partial, and full-model updates via configuration flags
• Production Pipelines: YAML-driven orchestration chaining SFT, preference optimization, and quantization using Axolotl, TRL, and vLLM

Frequently Asked Questions

What is the difference between LoRA and QLoRA in the curriculum?

LoRA freezes base model weights and injects trainable low-rank matrices into linear layers, while QLoRA adds 4-bit NF4 quantization to the base model weights, enabling the training of massive models (up to 70B parameters) on consumer GPUs with only 24GB VRAM. According to the implementation in phases/11-llm-engineering/08-fine-tuning-lora, both methods achieve approximately 99% of full fine-tuning quality with minimal memory overhead.

How does the curriculum prevent catastrophic forgetting during fine-tuning?

The methodologies employ several protective mechanisms: instruction token masking with ignore_index=-100 (removing instruction portions from the loss calculation), low learning rates with gradient clipping, and parameter-efficient adapters that leave backbone weights frozen. The SFT lessons in both Phase 10 and Phase 19 emphasize keeping fine-tuning data disjoint from evaluation data to prevent memorization.

What is the purpose of Direct Preference Optimization (DPO) in the training pipeline?

DPO optimizes language models directly against human preference pairs without requiring a separate reward model, simplifying the RLHF pipeline. As implemented in phases/10-llms-from-scratch/08-dpo, it uses a contrastive loss calculated as logsigmoid(p_ref - p_alt), updating the policy based on the relative probabilities of preferred versus rejected completions.

Can the end-to-end pipeline handle full-model fine-tuning or only LoRA?

The Phase 19 capstone pipeline supports both approaches. While the production examples emphasize LoRA and QLoRA for resource efficiency, the architecture in phases/19-capstone-projects/07-end-to-end-fine-tuning-pipeline uses Axolotl configurations that can toggle between full-model fine-tuning and parameter-efficient methods via the YAML manifest, allowing researchers to choose based on available compute and accuracy requirements.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}