How ViMax Handles Transitions Between Video Shots From Different Cameras

ViMax treats multi-camera scenes as a hierarchical camera tree, automatically generating intermediate transition videos and extracting new camera images to ensure visual continuity when switching between parent and child camera shots.

The HKUDS/ViMax repository implements a sophisticated pipeline for maintaining visual consistency across shots captured by different cameras. By modeling camera relationships as a tree structure and leveraging LLM-driven video generation, ViMax bridges the gap between disparate camera angles without manual intervention.

Understanding the Camera Tree Architecture

ViMax represents multi-camera scenes as a camera tree, where each node may optionally reference a parent camera whose footage already covers the visual content of the child camera. This hierarchy is established through the CameraImageGenerator.construct_camera_tree method in agents/camera_image_generator.py, which processes camera definitions from interfaces/camera.py.

When a shot's camera has a parent (camera.parent_shot_idx is not None), the system identifies the need for a transitional bridge. The parent shot provides the visual context, while the child shot requires a seamless entry point that matches the new camera's composition, background, and lighting.

The Three-Stage Transition Pipeline

ViMax handles camera transitions through a deterministic three-stage workflow managed by CameraImageGenerator and orchestrated within Script2VideoPipeline.generate_frames_for_single_camera.

Step 1: Generating the Transition Video

The generate_transition_video method constructs a specialized prompt describing both the parent and child shots, then invokes the video generator to produce a short "cut-to" clip.

prompt = (
    "Two shots. The transition between the shots is a cut to. "
    "The style of the two shots should be consistent.\n"
    f"The first shot description: {first_shot_visual_desc}.\n"
    f"The second shot description: {second_shot_visual_desc}."
)

The system supplies the first-shot frame of the parent camera (first_shot_ff_path) as a reference image, ensuring stylistic consistency between the source and destination shots.

Step 2: Deriving the New Camera Image

Once the transition video exists, get_new_camera_image performs scene detection to extract a usable reference frame for the child camera. This method uses ContentDetector from the scenedetect library to identify scene boundaries within the transition clip.

video = open_video(transition_video_path)
scene_manager = SceneManager()
scene_manager.add_detector(ContentDetector())
scene_manager.detect_scenes(video, show_progress=False)
scene_list = scene_manager.get_scene_list()
split_video_ffmpeg(transition_video_path, scene_list, output_dir, show_progress=True)

video_name = os.path.basename(transition_video_path).split('.')[0]
second_video_path = os.path.join(output_dir, f"{video_name}-Scene-002.mp4")
if os.path.exists(second_video_path):
    # use first frame of the second scene

    clip = VideoFileClip(second_video_path)
    frame = Image.fromarray(clip.get_frame(0).astype('uint8'), 'RGB')
else:
    # fallback to the last frame of the whole transition clip

    clip = VideoFileClip(transition_video_path)
    t = max(0, clip.duration - (1 / clip.fps))
    frame = Image.fromarray(clip.get_frame(t).astype('uint8'), 'RGB')
return ImageOutput(fmt="pil", ext="png", data=frame)

The method prioritizes the first frame of the second scene (representing the child camera's view) but falls back to the last frame of the transition clip if scene detection fails. The resulting ImageOutput becomes the reference image for the child's first frame generation.

Step 3: First-Frame Synthesis

In Script2VideoPipeline.generate_frames_for_single_camera, ViMax integrates the extracted new camera image into the generation workflow. The system either uses this image alongside character portraits to select reference images and craft a generation prompt, or copies the generated image directly when the child camera has no missing information.

Implementation in Script2VideoPipeline

The orchestration logic in pipelines/script2video_pipeline.py handles the synchronization between parent and child shots. The pipeline waits for the parent's first frame to be ready before initiating the transition process.

if camera.parent_shot_idx is not None:
    # wait for the parent’s first frame

    await self.frame_events[parent_shot_idx]["first_frame"].wait()
    parent_shot_ff_path = os.path.join(self.working_dir, "shots",
                                      f"{parent_shot_idx}", "first_frame.png")
    transition_path = os.path.join(self.working_dir, "shots",
                                   f"{first_shot_idx}",
                                   f"transition_video_from_shot_{parent_shot_idx}.mp4")

    # ① Create transition video (cut‑to)

    transition_video = await self.camera_image_generator.generate_transition_video(
        first_shot_visual_desc=shot_descriptions[parent_shot_idx].visual_desc,
        second_shot_visual_desc=shot_descriptions[first_shot_idx].visual_desc,
        first_shot_ff_path=parent_shot_ff_path,
    )
    transition_video.save(transition_path)

    # ② Derive a new camera image from the transition clip

    new_camera_img = self.camera_image_generator.get_new_camera_image(
        transition_video_path=transition_path
    )
    new_camera_img.save(os.path.join(self.working_dir, "shots",
                                     f"{first_shot_idx}",
                                     f"new_camera_{camera.idx}.png"))

This implementation ensures that child cameras never generate content in isolation; they always inherit visual context from their parent transitions.

Summary

• ViMax models multi-camera relationships as a camera tree with parent-child hierarchies defined in interfaces/camera.py.
• The construct_camera_tree method in agents/camera_image_generator.py determines which shots require transitional bridges.
• Transition videos are generated using LLM-driven prompts that describe both source and destination shots.
• Scene detection extracts the new camera image from the transition video, providing a contextually appropriate starting frame for the child camera.
• Script2VideoPipeline.generate_frames_for_single_camera orchestrates the workflow, ensuring parent frames are ready before child processing begins.

Frequently Asked Questions

How does ViMax determine which cameras need transition videos?

ViMax checks the parent_shot_idx attribute of each camera object defined in interfaces/camera.py. If this value is not None, the camera has a parent shot, and the system automatically triggers the transition video generation process through CameraImageGenerator.construct_camera_tree.

What happens if the transition video contains only one scene?

The get_new_camera_image method includes fallback logic for single-scene transitions. When the expected second scene file (Scene-002.mp4) does not exist, the system extracts the last frame of the entire transition clip using VideoFileClip.get_frame() with a timestamp calculated as duration - (1 / fps).

Can ViMax handle transitions between more than two cameras?

Yes, the camera tree structure supports arbitrary depth. Each camera can have its own parent, creating chains of transitions. The pipeline processes these sequentially, ensuring that each child camera waits for its parent frame to be generated before beginning its own transition workflow.

Where is the transition prompt constructed in the source code?

The prompt template resides within CameraImageGenerator.generate_transition_video in agents/camera_image_generator.py. The method concatenates a fixed descriptor ("Two shots. The transition between the shots is a cut to...") with the specific visual descriptions of the parent and child shots supplied by the script pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}