# How Eagle Handles Multi-Image Input in a Single Forward Pass: Architecture and Implementation

> Discover how Eagle handles multi-image input in a single forward pass. Learn its architecture and implementation for efficient image processing and feature integration.

- Repository: [NVIDIA Research Projects/Eagle](https://github.com/NVlabs/Eagle)
- Tags: architecture
- Published: 2026-06-28

---

**Eagle processes multiple images by detecting list or 5-D tensor inputs, encoding all images in a single vision tower pass, and splicing the resulting features into the text embedding stream at `<image>` token positions.**

The NVlabs/Eagle multimodal language model extends the standard Llama causal language model with a custom routine that ingests any number of images alongside text. This capability is implemented in `EagleMetaForCausalLM.prepare_inputs_labels_for_multimodal` within [`eagle/model/eagle_arch.py`](https://github.com/NVlabs/Eagle/blob/main/eagle/model/eagle_arch.py), enabling efficient batch processing of visual inputs without repeated vision tower invocations.

## The Entry Point: Forward Method Signature

Multi-image handling begins in `EagleLlamaForCausalLM.forward` (lines 72-84 of [`eagle/model/language_model/eagle_llama.py`](https://github.com/NVlabs/Eagle/blob/main/eagle/model/language_model/eagle_llama.py)). The method accepts an `images` argument that can be a list of tensors or a 5-D tensor, along with optional `image_sizes` metadata. These arguments are immediately forwarded to `prepare_inputs_labels_for_multimodal`, which orchestrates the multimodal preprocessing.

## Detecting and Batching Multiple Images

The system detects multi-image batches at lines 71-78 of [`eagle/model/eagle_arch.py`](https://github.com/NVlabs/Eagle/blob/main/eagle/model/eagle_arch.py). If `images` is a Python list or a 5-D tensor with shape `(batch, N, C, H, W)`, Eagle treats it as a multi-image input.

When a list is provided:
- Each image is reshaped to `(1, C, H, W)` if necessary
- All images are concatenated along the batch dimension using `torch.cat`
- The original per-sample counts are stored in `split_sizes` for later reconstruction

This concatenation allows the vision tower to process all images in a single efficient forward pass rather than looping through samples individually.

## Encoding and Feature Extraction

The batched tensor flows through `self.encode_images` (lines 58-62 of [`eagle_arch.py`](https://github.com/NVlabs/Eagle/blob/main/eagle_arch.py)), which executes:
1. `self.get_model().get_vision_tower()(images)` – extracts visual features
2. `self.get_model().mm_projector(image_features)` – projects features into the language model's hidden dimension

The output `image_features` has shape `(total_images, D)`, combining all images from the batch into one feature tensor.

## Splitting and Merging with Text Embeddings

After encoding, `torch.split` uses the stored `split_sizes` (lines 75-78) to divide the joint feature tensor back into per-sample chunks. Each chunk corresponds to the images from a specific training sample.

The merging process (lines 81-89) scans each token sequence for `IMAGE_TOKEN_INDEX` (defined in [`constants.py`](https://github.com/NVlabs/Eagle/blob/main/constants.py)). When an image token is encountered:
- The corresponding image feature chunk is concatenated to the token embedding stream
- If `mm_patch_merge_type` is `"flat"` (default), features are flattened with `x.flatten(0, 1)` before insertion
- For `"spatial"` mode (lines 84-106), more complex reshaping preserves spatial structure

This splicing inserts image embeddings **between** textual tokens at the exact positions marked by `<image>` tokens.

## Padding and Attention Mask Updates

Post-insertion sequences may exceed the original length. The implementation truncates to `tokenizer_model_max_length` and pads the batch to the longest sequence (lines 92-124). The attention mask is updated to ensure the transformer attends only to valid tokens, including the newly added image embeddings.

Finally, the method calls `super().forward()` (lines 107-118 of [`eagle_llama.py`](https://github.com/NVlabs/Eagle/blob/main/eagle_llama.py)) with the combined `inputs_embeds` containing both text and image representations.

## Complete Code Example

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from Eagle.eagle.model.language_model.eagle_llama import EagleLlamaForCausalLM, EagleConfig

# Load model & tokenizer

tokenizer = AutoTokenizer.from_pretrained("nv-eagle/Eagle-LLaVA")
model = EagleLlamaForCausalLM.from_pretrained("nv-eagle/Eagle-LLaVA", config=EagleConfig())

# Text prompt with placeholders for two images

prompt = "<image> What are the objects in the scene? <image> Describe their relationship."
input_ids = tokenizer(prompt, return_tensors="pt").input_ids

# Create two dummy images (batch, 3, 224, 224)

img1 = torch.randn(1, 3, 224, 224)
img2 = torch.randn(1, 3, 224, 224)

# Forward pass – note the list of images

outputs = model(
    input_ids=input_ids,
    images=[img1, img2],          # <-- list triggers multi-image logic

    image_sizes=[[224, 224], [224, 224]]
)

logits = outputs.logits  # shape: (batch, seq_len, vocab_size)

```

Passing a **list** to the `images` parameter triggers the multi-image routing described above, encoding both images once and inserting them at their respective token positions.

## Summary

- **Single-pass encoding**: Eagle concatenates all images from a batch into one tensor, processing them through the vision tower in a single call rather than iterating per-image.
- **Dynamic detection**: The system automatically recognizes multi-image inputs via list or 5-D tensor checks at lines 71-78 of [`eagle_arch.py`](https://github.com/NVlabs/Eagle/blob/main/eagle_arch.py).
- **Feature splicing**: Image embeddings are inserted into the text stream at `IMAGE_TOKEN_INDEX` positions, supporting both flat and spatial merging modes.
- **Efficient batching**: Per-sample image counts are preserved using `split_sizes` and `torch.split`, enabling variable numbers of images per sample in the same batch.

## Frequently Asked Questions

### How does Eagle handle different numbers of images per sample in the same batch?

Eagle tracks the original per-sample image counts in `split_sizes` before concatenating images for encoding. After the vision tower processes the combined batch, `torch.split` reconstructs the individual feature chunks using these stored counts. This allows samples with one image and samples with three images to coexist in the same batch.

### What is the difference between "flat" and "spatial" mm_patch_merge_type?

The **"flat"** mode (default) flattens image features using `x.flatten(0, 1)`, creating a 1-D sequence of visual tokens that are concatenated directly into the text embedding stream. The **"spatial"** mode preserves the 2-D spatial structure of vision features through additional reshaping operations (lines 84-106 of [`eagle_arch.py`](https://github.com/NVlabs/Eagle/blob/main/eagle_arch.py)), which may benefit tasks requiring spatial reasoning.

### Can Eagle process a single image using the same code path?

Yes. Single images are handled by the same `prepare_inputs_labels_for_multimodal` function. When `images` is a 4-D tensor (batch × C × H × W) or a list with one element, the logic simplifies but follows the identical pipeline: encode, split (trivially), and merge at the image token position.

### Why is encoding all images in one pass more efficient than processing them individually?

Processing images individually would require separate forward passes through the vision tower for each image, resulting in redundant memory transfers and computation. By concatenating images into a single batch (`torch.cat`) and invoking the vision tower once, Eagle maximizes GPU utilization and reduces overhead, particularly beneficial when handling high-resolution images or large batch sizes.