# How ViMax's Reference Image Selector Generates Initial Frames for Video Shots

> Discover how ViMax's reference image selector generates initial frames. This two-stage LLM process selects relevant images and creates structured prompts for coherent video generation.

- Repository: [✨Data Intelligence Lab@HKU✨/ViMax](https://github.com/HKUDS/ViMax)
- Tags: how-to-guide
- Published: 2026-05-20

---

**ViMax's reference image selector performs a two-stage LLM filtering process—first text-only, then multimodal—to select up to eight relevant reference images and generate a structured text prompt that guides the creation of coherent first frames in video generation.**

The HKUDS/ViMax repository implements a sophisticated agent-based pipeline for script-to-video generation. At the heart of this system lies the **ViMax reference image selector**, which ensures visual consistency by intelligently choosing reference materials before generating the initial frame of each shot. This component bridges script descriptions and visual output by curating character portraits, scene images, and camera references into a deterministic generation prompt.

## Pipeline Integration in `Script2VideoPipeline`

The selector activates during the first-frame generation phase within the `generate_frames_for_single_camera` method of [`pipelines/script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/script2video_pipeline.py). When processing a shot that lacks parent camera information or requires new visual context, the pipeline assembles candidate references—including character portraits and provisional camera images—into `available_image_path_and_text_pairs`.

The pipeline then invokes the selector asynchronously:

```python
ff_selector_output = await self.reference_image_selector.select_reference_images_and_generate_prompt(
    available_image_path_and_text_pairs=available_image_path_and_text_pairs,
    frame_description=shot_descriptions[first_shot_idx].ff_desc
)

```

As implemented in `HKUDS/ViMax`, the selector returns `reference_image_path_and_text_pairs` and a `text_prompt`, which the pipeline immediately passes to `self.image_generator.generate_single_image` to synthesize the initial frame.

## Two-Stage Selection Logic in `ReferenceImageSelector`

The core logic resides in [`agents/reference_image_selector.py`](https://github.com/HKUDS/ViMax/blob/main/agents/reference_image_selector.py), specifically within the `select_reference_images_and_generate_prompt` method. This implementation employs a hierarchical filtering strategy to balance cost and accuracy.

### Stage 1: Text-Only Filtering for Large Candidate Sets

When the candidate pool contains **eight or more images**, the selector first engages a text-only LLM using the `system_prompt_template_select_reference_images_only_text` template. The model receives the list of candidate descriptions and the target `frame_description`, then outputs a `RefImageIndicesAndTextPrompt` object specifying which indices to retain (capped at eight).

This text-only pass reduces computational overhead by eliminating irrelevant candidates before expensive multimodal processing.

### Stage 2: Multimodal Filtering with Visual Context

The surviving candidates undergo multimodal analysis via a vision-capable LLM (e.g., GPT-4V). For each candidate, the system encodes the image using `image_path_to_b64` from [`utils/image.py`](https://github.com/HKUDS/ViMax/blob/main/utils/image.py), creating base-64 data URLs paired with textual descriptions. The LLM processes these through the `system_prompt_template_select_reference_images_multimodal` system prompt alongside the frame description.

The multimodal stage returns final indices and a **text prompt** that explicitly references selected images (e.g., "Image 0 should provide the character’s face"), ensuring the subsequent generator receives precise visual instructions.

## Prompt Design and Consistency Constraints

Both LLM stages enforce strict consistency constraints through their system prompts:

- **Character consistency** – Select portraits matching gender, ethnicity, and pose requirements.
- **Environmental consistency** – Prioritize recent scene images with matching camera angles and lighting conditions.
- **Style consistency** – Maintain coherent visual styling across all reference materials.

The LLM must output valid JSON conforming to the `RefImageIndicesAndTextPrompt` structure:

```json
{
  "ref_image_indices": [0, 3, 5],
  "text_prompt": "Create an image ... Image 0 provides Alice's face ..."
}

```

This structured output guarantees that the generated first frame aligns with narrative context and established visual continuity.

## Composing the Final Generation Prompt

After selection, the pipeline constructs the complete generation prompt by prefixing the chosen reference descriptions to the LLM-generated instructions:

```python
prefix_prompt = ""
for i, (image_path, text) in enumerate(reference_image_path_and_text_pairs):
    prefix_prompt += f"Image {i}: {text}\n"
prompt = f"{prefix_prompt}\n{prompt}"

```

The resulting composite prompt (reference descriptions plus specific instructions) feeds directly into the image generation model, producing an initial frame that respects character identities, environmental context, and directorial intent.

## Practical Implementation Example

To invoke the selector directly for debugging or custom workflows, initialize the class with any LangChain-compatible model and execute the selection method:

```python
from agents.reference_image_selector import ReferenceImageSelector
from utils.image import image_path_to_b64
from langchain.chat_models import init_chat_model
import asyncio

# 1. Initialise a chat model

chat_model = init_chat_model(model_name="gpt-4o-mini")

# 2. Create the selector

selector = ReferenceImageSelector(chat_model=chat_model)

# 3. Build candidate list (path, description)

candidates = [
    ("/path/to/alice_front.png", "A front-view portrait of Alice."),
    ("/path/to/bob_side.png", "A side-view portrait of Bob."),
    ("/path/to/scene1.png", "Medium shot of the supermarket aisle."),
]

# 4. Describe the target frame

frame_desc = "[Camera 1] Shot from Alice's over-the-shoulder perspective..."

# 5. Run selector (async)

output = asyncio.run(
    selector.select_reference_images_and_generate_prompt(
        available_image_path_and_text_pairs=candidates,
        frame_description=frame_desc,
    )
)

print("Chosen references:", output["reference_image_path_and_text_pairs"])
print("Generated prompt:", output["text_prompt"])

```

When integrated into the standard ViMax pipeline, these steps execute automatically for every first-frame generation within `generate_frames_for_single_camera`.

## Summary

- **ViMax's reference image selector** operates within `Script2VideoPipeline` to prepare initial frame generation by curating visual references.
- The selector implements **two-stage filtering**: an optional text-only pass for large candidate sets followed by multimodal processing with base-64 encoded images.
- **Maximum eight references** are selected to balance context richness with model token limits.
- Output follows the **`RefImageIndicesAndTextPrompt`** JSON schema, producing deterministic prompts that reference specific image indices.
- Core implementation resides in [`agents/reference_image_selector.py`](https://github.com/HKUDS/ViMax/blob/main/agents/reference_image_selector.py), with utility functions in [`utils/image.py`](https://github.com/HKUDS/ViMax/blob/main/utils/image.py) and pipeline integration in [`pipelines/script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/script2video_pipeline.py).

## Frequently Asked Questions

### Why does ViMax limit reference images to eight?

The eight-image cap optimizes both cost and model performance. As implemented in [`agents/reference_image_selector.py`](https://github.com/HKUDS/ViMax/blob/main/agents/reference_image_selector.py), exceeding this threshold triggers the text-only pre-filtering stage, while the final multimodal LLM call receives at most eight base-64 encoded images. This constraint prevents context window overflow and reduces API costs while maintaining sufficient visual context for coherent generation.

### What is the purpose of the text-only filtering stage?

The text-only stage serves as a lightweight pre-processor when candidate pools contain eight or more images. By evaluating descriptions against the `frame_description` using a cheaper text-only LLM, ViMax eliminates irrelevant candidates before incurring the computational expense of encoding and processing images through the multimodal model. This two-tier approach significantly reduces generation latency and API costs.

### How does the selector ensure character consistency across frames?

The selector enforces consistency through specialized system prompts (`system_prompt_template_select_reference_images_only_text` and `system_prompt_template_select_reference_images_multimodal`) that instruct the LLM to match gender, ethnicity, pose, and lighting conditions. By explicitly selecting character portraits and scene references that align with the target description, and by generating text prompts that reference specific images (e.g., "Image 0 provides the character's face"), the system ensures the generated frame maintains visual continuity with established character representations.

### Which files contain the core selection logic?

The primary implementation resides in [`agents/reference_image_selector.py`](https://github.com/HKUDS/ViMax/blob/main/agents/reference_image_selector.py), containing the `ReferenceImageSelector` class and its `select_reference_images_and_generate_prompt` method. Pipeline integration occurs in [`pipelines/script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/script2video_pipeline.py) within the `generate_frames_for_single_camera` method. Image encoding utilities are located in [`utils/image.py`](https://github.com/HKUDS/ViMax/blob/main/utils/image.py), specifically the `image_path_to_b64` function used for multimodal LLM preparation.