# How ViMax's Best Image Selector Uses MLLM/VLM to Evaluate and Choose Consistent Video Frames > Discover how ViMax's Best Image Selector uses MLLM/VLM to evaluate and pick consistent video frames by scoring character consistency, spatial layout, and description accuracy against references. Optimize your video generation. - Repository: [✨Data Intelligence Lab@HKU✨/ViMax](https://github.com/HKUDS/ViMax) - Tags: how-to-guide - Published: 2026-05-20 --- **ViMax's BestImageSelector employs a multimodal LLM (MLLM/VLM) to compare generated video frames against reference images and a target description, scoring them on character consistency, spatial layout, and description accuracy to select the best candidate.** The **BestImageSelector** agent in the HKUDS/ViMax repository automates visual quality control in video generation pipelines. Instead of relying on pixel-level similarity metrics, it leverages a Vision Language Model (VLM) to perform holistic visual reasoning, ensuring that selected frames maintain narrative consistency across the generated video sequence. ## How the BestImageSelector Evaluates Frames The selector operates as a LangChain-based agent that orchestrates multimodal inputs and structured outputs. According to the implementation in [`agents/best_image_selector.py`](https://github.com/HKUDS/ViMax/blob/main/agents/best_image_selector.py), the evaluation workflow follows six distinct steps: ### Initializing the Multimodal Model The agent first initializes an OpenAI-compatible chat model using `init_chat_model`. This creates an asynchronous interface to the VLM using the provided `base_url`, `api_key`, and model identifier (e.g., `gpt-4o-mini`). This model handles both text and image inputs through a unified chat completion API. ### Building the Evaluation Prompt The system constructs a detailed prompt that defines the model's role as a *"professional visual assessment expert"*. The prompt structure in [`agents/best_image_selector.py`](https://github.com/HKUDS/ViMax/blob/main/agents/best_image_selector.py) contains three critical components: - **System instructions** that define the three evaluation criteria: **Character Consistency**, **Spatial Consistency**, and **Description Accuracy** - **Pydantic format instructions** ensuring the model outputs valid JSON matching the `BestImageResponse` schema - **Human content** that sequentially feeds reference images and candidate images as base-64 data URIs, followed by the target description wrapped in `` and `` tags ### Encoding Visual Data Raw image files are converted to base-64 data URIs using the `image_path_to_b64` function located in [`utils/image.py`](https://github.com/HKUDS/ViMax/blob/main/utils/image.py). This utility reads each image file and prefixes the encoded string with the correct MIME type (e.g., `data:image/jpeg;base64,...`), allowing the VLM to process actual visual data rather than external URLs. ### Structured Output Parsing The agent uses `PydanticOutputParser` configured with the `BestImageResponse` schema. This schema enforces two required fields: `best_image_index` (integer) and `reason` (string). By binding this parser to the chat model via a LangChain chain (`self.chat_model | parser`), the system guarantees that the VLM returns a deterministic, machine-readable response rather than free-form text. ### Selection and Validation The chain executes via `await chain.ainvoke(messages)`, sending the assembled multimodal payload to the VLM. Upon receiving the response, the selector validates that `best_image_index` falls within the bounds of the provided candidate list. If the index is malformed or out-of-range, the system logs a warning and falls back to selecting the first candidate (index 0). ## Evaluation Criteria Used by the VLM The system prompt explicitly instructs the VLM to assess candidates across three dimensions, enabling nuanced visual reasoning that transcends simple template matching. ### Character Consistency The model compares gender, ethnicity, age, facial features, body shape, outfit, hairstyle, and overall outlook between each candidate frame and the provided reference images. This ensures that the same protagonist appears visually coherent across different shots in the generated video. ### Spatial Consistency The VLM evaluates relative positioning and perspective, verifying that spatial relationships (e.g., "Character A is on the left") and environmental layouts remain coherent between reference and candidate frames. This prevents jarring continuity errors in scene composition. ### Description Accuracy Finally, the model checks whether the candidate image actually depicts the actions, objects, and scene elements described in the `` block. This grounds the visual output in the narrative requirements of the script. ## Implementation Example The following example demonstrates how to instantiate and invoke the selector in an async context: ```python from agents.best_image_selector import BestImageSelector # Initialize the selector with your VLM endpoint selector = BestImageSelector( base_url="https://api.openai.com/v1", api_key="sk-...", # Use environment variables in production chat_model="gpt-4o-mini" ) # Define reference frames with descriptions reference_pairs = [ ("/tmp/ref0.jpg", "A young girl with long brown hair wearing a red dress."), ("/tmp/ref1.jpg", "A close-up of a green-clad robot arm.") ] target_desc = "A teenage girl standing in a park, holding a red balloon." candidates = ["/tmp/cand0.jpg", "/tmp/cand1.jpg", "/tmp/cand2.jpg"] # Execute selection best_path = await selector( reference_image_path_and_text_pairs=reference_pairs, target_description=target_desc, candidate_image_paths=candidates, ) print(f"Selected frame: {best_path}") ``` ## Integration in Video Pipelines In production use within [`pipelines/script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/script2video_pipeline.py), the selector functions as a quality gate between generation and final composition: ```python from pipelines.script2video_pipeline import Script2VideoPipeline pipeline = Script2VideoPipeline(...) # After generating multiple candidates for a scene best_frame = await pipeline.best_image_selector( reference_image_path_and_text_pairs=pipeline.prev_frames, target_description=pipeline.current_description, candidate_image_paths=pipeline.generated_frames, ) pipeline.add_frame(best_frame) ``` This pattern allows the video generation pipeline to produce multiple candidate frames per scene while ensuring only the most consistent, high-quality frame advances to the final video sequence. ## Summary - **BestImageSelector** in [`agents/best_image_selector.py`](https://github.com/HKUDS/ViMax/blob/main/agents/best_image_selector.py) uses a VLM to perform multimodal evaluation of video frames, combining reference images with target descriptions. - The system encodes images as base-64 data URIs via [`utils/image.py`](https://github.com/HKUDS/ViMax/blob/main/utils/image.py), enabling direct visual analysis without external hosting. - Three criteria drive selection: **Character Consistency**, **Spatial Consistency**, and **Description Accuracy**. - Structured output parsing via `PydanticOutputParser` guarantees deterministic index selection from the `BestImageResponse` schema. - Built-in validation and fallback logic ensure pipeline resilience even when VLM responses are malformed. ## Frequently Asked Questions ### How does ViMax handle images when sending them to the VLM? ViMax converts image files to base-64 encoded data URIs using the `image_path_to_b64` utility in [`utils/image.py`](https://github.com/HKUDS/ViMax/blob/main/utils/image.py). This function reads the binary image data, applies base-64 encoding, and prefixes it with the appropriate MIME type (e.g., `data:image/jpeg;base64,...`), allowing the multimodal LLM to process the visual content directly within the chat completion payload. ### What happens if the VLM returns an invalid index for the best image? The selector includes validation logic that checks whether `best_image_index` is a valid integer within the bounds of the candidate list. If the VLM returns an out-of-range index or the parsing fails, the system logs a warning and automatically falls back to selecting the first candidate (index 0), ensuring the pipeline continues without interruption. ### Can I use a different VLM model with the BestImageSelector? Yes, the selector accepts any OpenAI-compatible chat model through the `chat_model` parameter in the constructor. As implemented in [`agents/best_image_selector.py`](https://github.com/HKUDS/ViMax/blob/main/agents/best_image_selector.py), the `init_chat_model` function supports various endpoints by configuring the `base_url` and `api_key`, allowing integration with alternative VLMs such as Claude, Gemini, or local deployments via vLLM or Ollama, provided they support image inputs in the messages array format. ### How does the VLM know what to look for when evaluating frames? The system injects explicit evaluation criteria into the system prompt, defining the VLM's role as a *"professional visual assessment expert"* and specifying three scoring dimensions: character appearance matching, spatial relationship preservation, and adherence to the target description. Additionally, the prompt includes Pydantic formatting instructions to enforce structured JSON output containing both the selected index and a reasoning string.