How ViMax's Reference Image Selector Generates Initial Frames for Video Shots

ViMax's reference image selector performs a two-stage LLM filtering process—first text-only, then multimodal—to select up to eight relevant reference images and generate a structured text prompt that guides the creation of coherent first frames in video generation.

The HKUDS/ViMax repository implements a sophisticated agent-based pipeline for script-to-video generation. At the heart of this system lies the ViMax reference image selector, which ensures visual consistency by intelligently choosing reference materials before generating the initial frame of each shot. This component bridges script descriptions and visual output by curating character portraits, scene images, and camera references into a deterministic generation prompt.

Pipeline Integration in Script2VideoPipeline

The selector activates during the first-frame generation phase within the generate_frames_for_single_camera method of pipelines/script2video_pipeline.py. When processing a shot that lacks parent camera information or requires new visual context, the pipeline assembles candidate references—including character portraits and provisional camera images—into available_image_path_and_text_pairs.

The pipeline then invokes the selector asynchronously:

ff_selector_output = await self.reference_image_selector.select_reference_images_and_generate_prompt(
    available_image_path_and_text_pairs=available_image_path_and_text_pairs,
    frame_description=shot_descriptions[first_shot_idx].ff_desc
)

As implemented in HKUDS/ViMax, the selector returns reference_image_path_and_text_pairs and a text_prompt, which the pipeline immediately passes to self.image_generator.generate_single_image to synthesize the initial frame.

Two-Stage Selection Logic in ReferenceImageSelector

The core logic resides in agents/reference_image_selector.py, specifically within the select_reference_images_and_generate_prompt method. This implementation employs a hierarchical filtering strategy to balance cost and accuracy.

Stage 1: Text-Only Filtering for Large Candidate Sets

When the candidate pool contains eight or more images, the selector first engages a text-only LLM using the system_prompt_template_select_reference_images_only_text template. The model receives the list of candidate descriptions and the target frame_description, then outputs a RefImageIndicesAndTextPrompt object specifying which indices to retain (capped at eight).

This text-only pass reduces computational overhead by eliminating irrelevant candidates before expensive multimodal processing.

Stage 2: Multimodal Filtering with Visual Context

The surviving candidates undergo multimodal analysis via a vision-capable LLM (e.g., GPT-4V). For each candidate, the system encodes the image using image_path_to_b64 from utils/image.py, creating base-64 data URLs paired with textual descriptions. The LLM processes these through the system_prompt_template_select_reference_images_multimodal system prompt alongside the frame description.

The multimodal stage returns final indices and a text prompt that explicitly references selected images (e.g., "Image 0 should provide the character’s face"), ensuring the subsequent generator receives precise visual instructions.

Prompt Design and Consistency Constraints

Both LLM stages enforce strict consistency constraints through their system prompts:

• Character consistency – Select portraits matching gender, ethnicity, and pose requirements.
• Environmental consistency – Prioritize recent scene images with matching camera angles and lighting conditions.
• Style consistency – Maintain coherent visual styling across all reference materials.

The LLM must output valid JSON conforming to the RefImageIndicesAndTextPrompt structure:

{
  "ref_image_indices": [0, 3, 5],
  "text_prompt": "Create an image ... Image 0 provides Alice's face ..."
}

This structured output guarantees that the generated first frame aligns with narrative context and established visual continuity.

Composing the Final Generation Prompt

After selection, the pipeline constructs the complete generation prompt by prefixing the chosen reference descriptions to the LLM-generated instructions:

prefix_prompt = ""
for i, (image_path, text) in enumerate(reference_image_path_and_text_pairs):
    prefix_prompt += f"Image {i}: {text}\n"
prompt = f"{prefix_prompt}\n{prompt}"

The resulting composite prompt (reference descriptions plus specific instructions) feeds directly into the image generation model, producing an initial frame that respects character identities, environmental context, and directorial intent.

Practical Implementation Example

To invoke the selector directly for debugging or custom workflows, initialize the class with any LangChain-compatible model and execute the selection method:

from agents.reference_image_selector import ReferenceImageSelector
from utils.image import image_path_to_b64
from langchain.chat_models import init_chat_model
import asyncio

# 1. Initialise a chat model

chat_model = init_chat_model(model_name="gpt-4o-mini")

# 2. Create the selector

selector = ReferenceImageSelector(chat_model=chat_model)

# 3. Build candidate list (path, description)

candidates = [
    ("/path/to/alice_front.png", "A front-view portrait of Alice."),
    ("/path/to/bob_side.png", "A side-view portrait of Bob."),
    ("/path/to/scene1.png", "Medium shot of the supermarket aisle."),
]

# 4. Describe the target frame

frame_desc = "[Camera 1] Shot from Alice's over-the-shoulder perspective..."

# 5. Run selector (async)

output = asyncio.run(
    selector.select_reference_images_and_generate_prompt(
        available_image_path_and_text_pairs=candidates,
        frame_description=frame_desc,
    )
)

print("Chosen references:", output["reference_image_path_and_text_pairs"])
print("Generated prompt:", output["text_prompt"])

When integrated into the standard ViMax pipeline, these steps execute automatically for every first-frame generation within generate_frames_for_single_camera.

Summary

• ViMax's reference image selector operates within Script2VideoPipeline to prepare initial frame generation by curating visual references.
• The selector implements two-stage filtering: an optional text-only pass for large candidate sets followed by multimodal processing with base-64 encoded images.
• Maximum eight references are selected to balance context richness with model token limits.
• Output follows the RefImageIndicesAndTextPrompt JSON schema, producing deterministic prompts that reference specific image indices.
• Core implementation resides in agents/reference_image_selector.py, with utility functions in utils/image.py and pipeline integration in pipelines/script2video_pipeline.py.

Frequently Asked Questions

Why does ViMax limit reference images to eight?

The eight-image cap optimizes both cost and model performance. As implemented in agents/reference_image_selector.py, exceeding this threshold triggers the text-only pre-filtering stage, while the final multimodal LLM call receives at most eight base-64 encoded images. This constraint prevents context window overflow and reduces API costs while maintaining sufficient visual context for coherent generation.

What is the purpose of the text-only filtering stage?

The text-only stage serves as a lightweight pre-processor when candidate pools contain eight or more images. By evaluating descriptions against the frame_description using a cheaper text-only LLM, ViMax eliminates irrelevant candidates before incurring the computational expense of encoding and processing images through the multimodal model. This two-tier approach significantly reduces generation latency and API costs.

How does the selector ensure character consistency across frames?

The selector enforces consistency through specialized system prompts (system_prompt_template_select_reference_images_only_text and system_prompt_template_select_reference_images_multimodal) that instruct the LLM to match gender, ethnicity, pose, and lighting conditions. By explicitly selecting character portraits and scene references that align with the target description, and by generating text prompts that reference specific images (e.g., "Image 0 provides the character's face"), the system ensures the generated frame maintains visual continuity with established character representations.

Which files contain the core selection logic?

The primary implementation resides in agents/reference_image_selector.py, containing the ReferenceImageSelector class and its select_reference_images_and_generate_prompt method. Pipeline integration occurs in pipelines/script2video_pipeline.py within the generate_frames_for_single_camera method. Image encoding utilities are located in utils/image.py, specifically the image_path_to_b64 function used for multimodal LLM preparation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}