How ViMax Ensures Character Consistency Across Multiple Video Shots

ViMax guarantees character consistency by binding every visual element to a single, reusable portrait set generated once per character and referenced throughout the entire video-generation pipeline.

Maintaining visual continuity in AI-generated video remains one of the hardest challenges in generative media. The HKUDS/ViMax repository solves this through a strict registry-based architecture that eliminates character drift by reusing identical reference images across every shot.

The Three-Stage Consistency Pipeline

ViMax implements character consistency across multiple video shots through three tightly coupled stages:

1. Character extraction – Parses the script and creates a CharacterInScene model describing each character’s static and dynamic traits.
2. Portrait generation – Builds front, side, and back portraits once per character and stores them in a shared registry.
3. Frame rendering – Every shot pulls appropriate portraits from the registry as reference images, ensuring the same pixel data drives every frame.

Unified Character Schema and Registry

At the heart of the system lies the unified character schema defined in interfaces/character.py. This CharacterInScene model serves as the single source of truth for every downstream component.

class CharacterInScene(BaseModel):
    idx: int
    identifier_in_scene: str               # e.g. "Alice"

    is_visible: bool
    static_features: str                   # immutable traits (hair, eyes, build)

    dynamic_features: Optional[str] = None  # clothing, accessories

The CharacterExtractor agent (located in main/agents/character_extractor.py) parses input scripts and instantiates these models, capturing immutable traits like hair color and facial structure separately from dynamic elements like clothing. This separation ensures that static visual identifiers remain constant while allowing contextual variation in attire.

One-Time Portrait Generation

After extraction, the Script2VideoPipeline checks the character_portraits_registry.json file. If a character is missing from this registry, the pipeline invokes generate_portraits_for_single_character to create three canonical views: front, side, and back.

As implemented in main/pipelines/script2video_pipeline.py, the generation produces a nested dictionary structure:

{
  "Alice": {
    "front": {"path": ".../front.png", "description": "A front view portrait of Alice."},
    "side":  {"path": ".../side.png",  "description": "A side view portrait of Alice."},
    "back":  {"path": ".../back.png",  "description": "A back view portrait of Alice."}
  }
}

These portraits are written once per character and reused for the entire pipeline run. The registry locks visual identity to specific file paths, ensuring that subsequent frames reference identical pixel data regardless of shot changes or camera movements.

Reference Image Injection for Every Frame

During frame generation, ViMax pulls consistent portraits from the registry for every shot type—whether generating first frames, last frames, or transition frames. The generate_frame_for_single_shot function constructs available_image_path_and_text_pairs by looking up visible characters in the registry:

for character_idx in shot_descriptions[first_shot_idx].ff_vis_char_idxs:
    identifier = characters[character_idx].identifier_in_scene
    registry_item = character_portraits_registry[identifier]
    for view, item in registry_item.items():
        available_image_path_and_text_pairs.append((item["path"], item["description"]))

Because the same file paths from character_portraits_registry feed into the image-generation model for every shot featuring that character, the model receives stable visual cues. This prevents the subtle drift in facial features or body proportions that typically occurs when AI video generators recreate characters from text descriptions alone.

Event-Driven Synchronization

To eliminate race conditions between portrait creation and frame consumption, ViMax implements an event-driven synchronization mechanism. Each character has an associated asyncio.Event stored in character_portrait_events[character.idx].

Downstream frame-generation tasks await this event before executing, ensuring portrait files are fully written to disk before any shot attempts to consume them. As seen in main/pipelines/script2video_pipeline.py, this coordination prevents scenarios where a frame generator might read partially written or corrupted portrait data.

Summary

• Single registry architecture: ViMax stores canonical portraits in character_portraits_registry.json and references them by identifier_in_scene throughout generation.
• Three-view consistency: Each character receives front, side, and back portraits generated once by generate_portraits_for_single_character.
• Path-based reference: Frame generation functions like generate_frame_for_single_shot inject exact file paths from the registry, ensuring pixel-perfect consistency across shots.
• Async safety: asyncio.Event objects synchronize portrait generation with frame rendering, preventing race conditions.

Frequently Asked Questions

What is character consistency in video generation?

Character consistency refers to maintaining stable visual attributes—such as facial features, body type, and clothing style—across multiple shots or scenes in a generated video. Without explicit mechanisms to preserve identity, AI models tend to drift, producing characters with varying appearances between cuts.

How does ViMax prevent character drift between shots?

ViMax prevents drift by generating a fixed set of portrait images once per character and storing them in character_portraits_registry.json. Every subsequent shot pulls these exact image files as reference material for the image-generation model. Because the model receives identical pixel data for "Alice" in shot one and shot ten, the character's appearance remains constant.

What are the three portrait views generated for each character?

The system generates front, side, and back views for every character via generate_portraits_for_single_character. These multiple angles allow the rendering pipeline to select the most appropriate reference based on camera positioning while maintaining geometric consistency in the character's proportions and features.

How does ViMax handle synchronization between portrait generation and frame rendering?

ViMax uses asyncio.Event objects mapped to each character index. Frame generation tasks await character_portrait_events[character.idx] before executing, ensuring that portrait files are completely written and ready before any shot attempts to load them. This prevents race conditions where a frame might reference an incomplete or missing portrait file.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}