# How to Fine-Tune Eagle Models with LoRA for Efficient Training

> Fine-tune Eagle models efficiently with LoRA. This guide shows how to enable LoRA adapters and configure ranks to train faster and save GPU memory without altering original weights.

- Repository: [NVIDIA Research Projects/Eagle](https://github.com/NVlabs/Eagle)
- Tags: tutorial
- Published: 2026-06-28

---

**You can fine-tune Eagle vision-language models efficiently by setting `training_args.lora_enable=True` and configuring adapter ranks via `use_llm_lora` and `use_backbone_lora`, which wraps the base model with PEFT LoRA adapters while keeping original weights frozen to minimize GPU memory usage.**

Eagle is a family of vision-language models developed by NVIDIA that supports parameter-efficient fine-tuning through **LoRA (Low-Rank Adaptation)**. The implementation integrates directly with the Hugging Face PEFT library and is exposed through high-level training arguments. This guide covers the exact source code paths, configuration parameters, and runnable examples for fine-tuning Eagle models with LoRA.

## LoRA Integration in the Training Pipeline

The LoRA implementation lives in the core training loop and is activated through specific configuration flags.

### Training Entry Point ([`Eagle/train.py`](https://github.com/NVlabs/Eagle/blob/main/Eagle/train.py))

The primary training script detects LoRA mode via `training_args.lora_enable`. When enabled, it constructs a `peft.LoraConfig` and wraps the base model using `peft.get_peft_model` according to lines 1060–1078 in [`Eagle/train.py`](https://github.com/NVlabs/Eagle/blob/main/Eagle/train.py). This process injects trainable low-rank matrices into targeted linear layers while freezing the original pre-trained weights.

The configuration automatically identifies target modules through the `find_all_linear_names` helper (lines 1089–1100), which selects linear layers while skipping vision-specific modules that should remain frozen during language-model fine-tuning.

### Model Arguments for Adapter Configuration

The `ModelArguments` dataclass in [`Embodied/eaglevl/train/arguments.py`](https://github.com/NVlabs/Eagle/blob/main/Embodied/eaglevl/train/arguments.py) exposes two critical fields for controlling adapter placement:

- **`use_backbone_lora`** (default: `0`) – Sets the rank for the vision backbone adapter (lines 53–57).
- **`use_llm_lora`** (default: `0`) – Sets the rank for the language model head adapter (lines 57–60).

When either value is non-zero, the trainer passes these ranks to the LoRA configuration builder, allowing independent control over which modality receives trainable adapters.

### LoRA Hyperparameters

The `LoraConfig` receives the following parameters during initialization:

- **`r`** – Adapter rank (default: `64`), determining the dimensionality of the low-rank decomposition.
- **`lora_alpha`** – Scaling factor (default: `16`) applied to the adapter outputs.
- **`lora_dropout`** – Dropout probability (default: `0.05`) applied before the low-rank projection.
- **`target_modules`** – List of linear layer names selected by the automatic finder.
- **`bias`** – Bias training strategy (`none`, `all`, or `lora_only`).

## End-to-End Fine-Tuning Workflow

The training process follows these sequential steps:

1. **Environment Setup** – Set `USE_LLM_LORA` and/or `USE_BACKBONE_LORA` environment variables to specify ranks (e.g., `64` for a 64-dimensional adapter), and ensure `lora_enable=True` in the training arguments.

2. **Argument Parsing** – The training script [`locany_finetune_magi_stream.py`](https://github.com/NVlabs/Eagle/blob/main/locany_finetune_magi_stream.py) parses CLI arguments into `ModelArguments`, forwarding the LoRA ranks to the core trainer.

3. **Adapter Injection** – [`Eagle/train.py`](https://github.com/NVlabs/Eagle/blob/main/Eagle/train.py) detects `lora_enable`, builds the `LoraConfig` with the supplied ranks, and calls `get_peft_model` to insert adapters.

4. **Frozen Base Weights** – The PEFT runtime freezes the original model parameters, leaving only the low-rank matrices trainable.

5. **Training Loop** – The Hugging Face Trainer updates only LoRA parameters, keeping the base model in FP16/BF16 without gradients, significantly reducing VRAM requirements.

## Practical Code Examples

### Fine-Tuning with the Provided Shell Script

The repository includes a ready-to-run script for visual-prompt fine-tuning:

```bash
export HF_TOKEN=YOUR_HF_TOKEN
export META_PATH=path/to/your/meta.json
bash Embodied/shell/locate-anything-lora-visual-prompt.sh \
  1 \
  work_dirs/locany_lora_prompt

```

This script configures LoRA through environment variables (lines 49–80):

- `USE_LLM_LORA=64` – Attaches a 64-dim adapter to the language model.
- `USE_BACKBONE_LORA=0` – Keeps the vision backbone frozen (set to a positive integer to enable vision fine-tuning).
- `FREEZE_LLM=True` and `FREEZE_BACKBONE=True` – Ensures original weights remain frozen while only adapters update.

### Manual Python Invocation

For programmatic control, instantiate the arguments directly:

```python
from transformers import HfArgumentParser
from eaglevl.train.arguments import ModelArguments, DataTrainingArguments
from Eagle.train import main as train_main

parser = HfArgumentParser((ModelArguments, DataTrainingArguments))
model_args, data_args = parser.parse_args_into_dataclasses()

# Configure LoRA ranks

model_args.use_llm_lora = 64      # LLM adapter rank

model_args.use_backbone_lora = 32 # Vision backbone adapter rank

model_args.freeze_llm = True
model_args.freeze_backbone = True

# Training configuration

training_args = {
    "lora_enable": True,
    "lora_r": model_args.use_llm_lora,
    "lora_alpha": 16,
    "lora_dropout": 0.05,
    "lora_bias": "none",
    "bits": 16,
    "bf16": True,
    # ... additional Trainer arguments

}

train_main(model_args, data_args, training_args)

```

The `train_main` function handles the PEFT integration, creating the `LoraConfig` and applying it via `get_peft_model` before starting the training loop.

### Inspecting and Exporting LoRA Weights

After training, extract only the adapter weights for efficient storage or distribution:

```python
from peft import PeftModel
import torch

model = PeftModel.from_pretrained("work_dirs/locany_lora_prompt/checkpoint-500")
lora_state = model.state_dict()

# Save only LoRA parameters

torch.save(
    {k: v for k, v in lora_state.items() if "lora_" in k},
    "locany_lora_adapters.pt"
)

```

To create a single-file deployment model, merge the adapters back into the base checkpoint using `peft.merge_and_save`.

## Key Source Files and Implementation Details

Understanding these files helps with custom modifications:

- **[`Eagle/train.py`](https://github.com/NVlabs/Eagle/blob/main/Eagle/train.py)** – Core training loop containing the LoRA detection logic (lines 1060–1078) and `LoraConfig` construction.
- **[`Embodied/eaglevl/train/arguments.py`](https://github.com/NVlabs/Eagle/blob/main/Embodied/eaglevl/train/arguments.py)** – Defines `use_backbone_lora` and `use_llm_lora` fields in the `ModelArguments` dataclass.
- **[`Embodied/shell/locate-anything-lora-visual-prompt.sh`](https://github.com/NVlabs/Eagle/blob/main/Embodied/shell/locate-anything-lora-visual-prompt.sh)** – Reference implementation setting LoRA environment variables and launching distributed training.
- **[`Embodied/eaglevl/train/locany_finetune_magi_stream.py`](https://github.com/NVlabs/Eagle/blob/main/Embodied/eaglevl/train/locany_finetune_magi_stream.py)** – Entry point that parses LoRA-specific CLI arguments and forwards them to the trainer.
- **[`Embodied/eaglevl/model/locany/modeling_locateanything.py`](https://github.com/NVlabs/Eagle/blob/main/Embodied/eaglevl/model/locany/modeling_locateanything.py)** – Model architecture compatible with PEFT-injected modules.

## Summary

- Enable LoRA by setting `training_args.lora_enable=True` in the training configuration.
- Control adapter placement separately for the vision backbone (`use_backbone_lora`) and language model (`use_llm_lora`) through `ModelArguments`.
- Default hyperparameters use rank `64`, alpha `16`, and dropout `0.05`, targeting only linear layers while skipping vision-specific modules.
- The PEFT integration freezes base weights automatically, reducing memory consumption to adapter parameters only (approximately `2 × r × hidden_dim` per layer).
- Use the provided shell scripts for quick experiments or the Python API for custom training pipelines.

## Frequently Asked Questions

### What are the default LoRA hyperparameters in Eagle?

According to the source code in [`Eagle/train.py`](https://github.com/NVlabs/Eagle/blob/main/Eagle/train.py), the default configuration uses a rank (`r`) of `64`, a scaling factor (`lora_alpha`) of `16`, and a dropout rate of `0.05`. The `bias` parameter defaults to `"none"`, meaning bias terms are not trained unless explicitly configured otherwise.

### Can I fine-tune only the vision backbone or only the LLM?

Yes. The `ModelArguments` dataclass provides independent control through `use_backbone_lora` and `use_llm_lora`. Set the respective field to a positive integer (the desired rank) to enable adapters for that component, or leave it at `0` to keep that modality frozen. This flexibility allows visual prompt tuning without modifying the language model, or vice versa.

### How much memory does LoRA save compared to full fine-tuning?

LoRA adapters add only approximately `2 × r × hidden_dim` parameters per target layer. For a typical configuration with rank `64`, this amounts to a few megabytes per layer rather than gigabytes for full weight matrices. Since the base model weights remain frozen in FP16 or BF16 without gradients, VRAM usage drops significantly, often allowing fine-tuning on consumer GPUs that could only run inference with full fine-tuning.

### Where are the LoRA adapters saved after training?

Adapter weights are saved within the standard Hugging Face checkpoint directories (e.g., `work_dirs/locany_lora_prompt/checkpoint-500`). To extract only the LoRA parameters, filter the state dictionary for keys containing `"lora_"` as shown in the Python example above. You can reload these adapters using `PeftModel.from_pretrained()` or merge them permanently into the base model using PEFT's merge utilities.