How Eagle Handles Multi-Image Input in a Single Forward Pass: Architecture and Implementation

Eagle processes multiple images by detecting list or 5-D tensor inputs, encoding all images in a single vision tower pass, and splicing the resulting features into the text embedding stream at <image> token positions.

The NVlabs/Eagle multimodal language model extends the standard Llama causal language model with a custom routine that ingests any number of images alongside text. This capability is implemented in EagleMetaForCausalLM.prepare_inputs_labels_for_multimodal within eagle/model/eagle_arch.py, enabling efficient batch processing of visual inputs without repeated vision tower invocations.

The Entry Point: Forward Method Signature

Multi-image handling begins in EagleLlamaForCausalLM.forward (lines 72-84 of eagle/model/language_model/eagle_llama.py). The method accepts an images argument that can be a list of tensors or a 5-D tensor, along with optional image_sizes metadata. These arguments are immediately forwarded to prepare_inputs_labels_for_multimodal, which orchestrates the multimodal preprocessing.

Detecting and Batching Multiple Images

The system detects multi-image batches at lines 71-78 of eagle/model/eagle_arch.py. If images is a Python list or a 5-D tensor with shape (batch, N, C, H, W), Eagle treats it as a multi-image input.

When a list is provided:

• Each image is reshaped to (1, C, H, W) if necessary
• All images are concatenated along the batch dimension using torch.cat
• The original per-sample counts are stored in split_sizes for later reconstruction

This concatenation allows the vision tower to process all images in a single efficient forward pass rather than looping through samples individually.

Encoding and Feature Extraction

The batched tensor flows through self.encode_images (lines 58-62 of eagle_arch.py), which executes:

1. self.get_model().get_vision_tower()(images) – extracts visual features
2. self.get_model().mm_projector(image_features) – projects features into the language model's hidden dimension

The output image_features has shape (total_images, D), combining all images from the batch into one feature tensor.

Splitting and Merging with Text Embeddings

After encoding, torch.split uses the stored split_sizes (lines 75-78) to divide the joint feature tensor back into per-sample chunks. Each chunk corresponds to the images from a specific training sample.

The merging process (lines 81-89) scans each token sequence for IMAGE_TOKEN_INDEX (defined in constants.py). When an image token is encountered:

• The corresponding image feature chunk is concatenated to the token embedding stream
• If mm_patch_merge_type is "flat" (default), features are flattened with x.flatten(0, 1) before insertion
• For "spatial" mode (lines 84-106), more complex reshaping preserves spatial structure

This splicing inserts image embeddings between textual tokens at the exact positions marked by <image> tokens.

Padding and Attention Mask Updates

Post-insertion sequences may exceed the original length. The implementation truncates to tokenizer_model_max_length and pads the batch to the longest sequence (lines 92-124). The attention mask is updated to ensure the transformer attends only to valid tokens, including the newly added image embeddings.

Finally, the method calls super().forward() (lines 107-118 of eagle_llama.py) with the combined inputs_embeds containing both text and image representations.

Complete Code Example

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from Eagle.eagle.model.language_model.eagle_llama import EagleLlamaForCausalLM, EagleConfig

# Load model & tokenizer

tokenizer = AutoTokenizer.from_pretrained("nv-eagle/Eagle-LLaVA")
model = EagleLlamaForCausalLM.from_pretrained("nv-eagle/Eagle-LLaVA", config=EagleConfig())

# Text prompt with placeholders for two images

prompt = "<image> What are the objects in the scene? <image> Describe their relationship."
input_ids = tokenizer(prompt, return_tensors="pt").input_ids

# Create two dummy images (batch, 3, 224, 224)

img1 = torch.randn(1, 3, 224, 224)
img2 = torch.randn(1, 3, 224, 224)

# Forward pass – note the list of images

outputs = model(
    input_ids=input_ids,
    images=[img1, img2],          # <-- list triggers multi-image logic

    image_sizes=[[224, 224], [224, 224]]
)

logits = outputs.logits  # shape: (batch, seq_len, vocab_size)

Passing a list to the images parameter triggers the multi-image routing described above, encoding both images once and inserting them at their respective token positions.

Summary

• Single-pass encoding: Eagle concatenates all images from a batch into one tensor, processing them through the vision tower in a single call rather than iterating per-image.
• Dynamic detection: The system automatically recognizes multi-image inputs via list or 5-D tensor checks at lines 71-78 of eagle_arch.py.
• Feature splicing: Image embeddings are inserted into the text stream at IMAGE_TOKEN_INDEX positions, supporting both flat and spatial merging modes.
• Efficient batching: Per-sample image counts are preserved using split_sizes and torch.split, enabling variable numbers of images per sample in the same batch.

Frequently Asked Questions

How does Eagle handle different numbers of images per sample in the same batch?

Eagle tracks the original per-sample image counts in split_sizes before concatenating images for encoding. After the vision tower processes the combined batch, torch.split reconstructs the individual feature chunks using these stored counts. This allows samples with one image and samples with three images to coexist in the same batch.

What is the difference between "flat" and "spatial" mm_patch_merge_type?

The "flat" mode (default) flattens image features using x.flatten(0, 1), creating a 1-D sequence of visual tokens that are concatenated directly into the text embedding stream. The "spatial" mode preserves the 2-D spatial structure of vision features through additional reshaping operations (lines 84-106 of eagle_arch.py), which may benefit tasks requiring spatial reasoning.

Can Eagle process a single image using the same code path?

Yes. Single images are handled by the same prepare_inputs_labels_for_multimodal function. When images is a 4-D tensor (batch × C × H × W) or a list with one element, the logic simplifies but follows the identical pipeline: encode, split (trivially), and merge at the image token position.

Why is encoding all images in one pass more efficient than processing them individually?

Processing images individually would require separate forward passes through the vision tower for each image, resulting in redundant memory transfers and computation. By concatenating images into a single batch (torch.cat) and invoking the vision tower once, Eagle maximizes GPU utilization and reduces overhead, particularly beneficial when handling high-resolution images or large batch sizes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}