Continual Pretraining vs Domain Adaptation for Agent Fine-Tuning: When to Use Each Strategy

Continual pretraining expands a model's language capabilities using broad corpora like Wikipedia, while domain adaptation injects specialized knowledge from curated datasets, with both strategies occurring during the pre-training phase before final instruction fine-tuning.

The bojieli/ai-agent-book repository provides concrete implementations for enhancing large language models before the supervised fine-tuning (SFT) stage. Understanding the distinction between these pre-training strategies determines whether your agent develops new linguistic foundations or specialized domain expertise.

Understanding the Pre-Training Phase

Both continual pretraining and domain-adaptive pretraining belong to the pre-training phase rather than the post-training instruction fine-tuning stage. As noted in the repository's conceptual documentation at cursor-chats/20251010_113500_continue_pretraining,_belong_to_pretrain_or_posttrain.md (line 49), these techniques augment the model's foundational knowledge before you teach it to follow instructions.

Continual Pretraining for Language Expansion

Goals and Data Requirements

Continual pretraining preserves the base model's general world knowledge while injecting new language-specific data. This approach targets scenarios where the model must learn new syntax, terminology, and writing styles—such as adapting an English-centric model to process Korean Wikipedia dumps or other broad linguistic corpora.

Implementation in the Repository

The script chapter7/continued-pretraining/continued-pretrain.py implements continual pretraining by configuring LoRA adapters to include critical embedding layers. Lines 45-55 specifically target embed_tokens and lm_head, allowing the model to learn new vocabulary representations while keeping most transformer parameters frozen.

# LoRA configuration from continued-pretrain.py (lines 45-55)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                   "gate_proj", "up_proj", "down_proj",
                   "embed_tokens", "lm_head"],  # Critical for new languages

    lora_alpha=16,
    use_gradient_checkpointing="unsloth",
    random_state=3407,
)

The script then runs a short pretraining stage on Korean Wikipedia before proceeding to instruction fine-tuning on a Korean Alpaca dataset.

Verification and Testing

The repository includes baseline and post-pretraining test sections (lines 71-84) that demonstrate marked improvements in Korean generation capabilities. These tests confirm that the model successfully acquired language-specific instruction-following abilities before the SFT stage refined alignment.

Domain-Adaptive Pretraining for Specialized Knowledge

Strategic Focus

Domain-adaptive pretraining shifts focus from language expansion to knowledge enrichment within specific verticals such as medical, legal, or scientific fields. Rather than learning new linguistic patterns, the model internalizes specialized terminology and factual knowledge from curated corpora like PubMed abstracts or legal case law.

Adapting the Repository Pipeline

While the repository emphasizes continual pretraining, you can repurpose the same continued-pretrain.py pipeline for domain adaptation by substituting the dataset and adjusting training hyperparameters. Replace the Korean Wikipedia corpus with domain-specific data and potentially reduce training epochs for embedding layers.

# Example: Adapting for medical domain adaptation

from datasets import load_dataset

# Load domain-specific corpus instead of Wikipedia

domain_dataset = load_dataset("allenai/scirex", split="train")

# Use the same formatting_prompts_func logic and UnslothTrainer

# configuration as in the original script, but with specialized text

Execution Example

To run the domain-adapted pipeline, modify the dataset parameters while keeping the same LoRA architecture:

python chapter7/continued-pretraining/continued-pretrain.py \
    --base_model unsloth/mistral-7b-v0.3 \
    --wiki_dataset your-domain-corpus/dataset-name \
    --pretrain_epochs 1 \
    --sft_epochs 3

Key Architectural and Data Differences

LoRA Target Modules

Continual pretraining requires adapting embedding layers (embed_tokens) and the language model head (lm_head) to handle new vocabulary tokens, as implemented in lines 45-55 of the repository script. Domain adaptation can often achieve sufficient knowledge transfer by training transformer attention layers if the domain shares the base language's vocabulary.

Training Duration and Learning Rates

Domain adaptation typically uses shorter training runs with lower learning rates for embeddings, as the model already understands the language structure. Continual pretraining may require longer epochs to establish new linguistic foundations and vocabulary mappings.

When to Use Each Strategy for Your Agent

Choose continual pretraining when:

• Deploying agents in new languages or significantly different linguistic variants
• Working with limited in-language data while preserving the base model's general knowledge
• Preparing models for language-specific instruction datasets where syntax and vocabulary differ substantially from the base model

Choose domain adaptation when:

• The target use case resides in well-defined verticals (medical, legal, scientific)
• You possess high-quality domain corpora requiring specialized terminology and factual accuracy
• You want the subsequent SFT stage to focus on instruction alignment rather than learning domain facts from scratch

Summary

• Both continual pretraining and domain adaptation occur before the final instruction-fine-tuning stage, belonging to the pre-training phase rather than post-training
• Continual pretraining focuses on language expansion using broad corpora, requiring LoRA adapters on embedding layers (embed_tokens, lm_head) as shown in chapter7/continued-pretraining/continued-pretrain.py lines 45-55
• Domain adaptation targets specialized knowledge injection through curated domain datasets, using the same pipeline with dataset substitution and adjusted hyperparameters
• The repository's implementation demonstrates measurable improvements in language-specific generation (lines 71-84) after continual pretraining on Korean Wikipedia
• Select the strategy based on whether your agent needs new linguistic foundations or specialized domain expertise within an existing language

Frequently Asked Questions

Can I combine continual pretraining and domain adaptation in sequence?

Yes, you can first perform continual pretraining to establish language capabilities, then conduct domain-adaptive pretraining to inject specialized knowledge, followed by instruction fine-tuning. The bojieli/ai-agent-book pipeline supports this sequential approach by saving intermediate checkpoints after each stage, allowing you to verify improvements before proceeding.

Why does continual pretraining require training the embedding layers while domain adaptation might not?

Continual pretraining often introduces new vocabulary tokens or significantly different linguistic structures that the original embeddings cannot represent. In continued-pretrain.py, including embed_tokens and lm_head in the LoRA targets (lines 45-55) allows the model to learn these new representations. Domain adaptation typically works within the existing vocabulary, focusing on recontextualizing known tokens for specialized domains rather than learning new token representations.

How do I verify that domain adaptation has improved my model before proceeding to SFT?

Following the repository's testing approach shown in lines 71-84, evaluate the model on domain-specific prompts that require factual knowledge or terminology unique to your target field. Compare outputs against the base model to confirm improved domain grounding and factual accuracy before investing computational resources in the full instruction fine-tuning stage.

Does domain adaptation work for low-resource domains?

Domain adaptation can work with smaller corpora than continual pretraining because it builds upon existing language understanding rather than learning new linguistic patterns from scratch. However, you may need to adjust the learning rate and training duration more conservatively to prevent catastrophic forgetting of general capabilities, potentially freezing embedding layers entirely if the domain uses standard vocabulary.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}