# How ViMax Handles Transitions Between Video Shots From Different Cameras

> Discover how ViMax manages transitions between different camera shots. Learn how ViMax's hierarchical camera tree ensures visual continuity with generated intermediate videos.

- Repository: [✨Data Intelligence Lab@HKU✨/ViMax](https://github.com/HKUDS/ViMax)
- Tags: deep-dive
- Published: 2026-05-20

---

**ViMax treats multi-camera scenes as a hierarchical camera tree, automatically generating intermediate transition videos and extracting new camera images to ensure visual continuity when switching between parent and child camera shots.**

The HKUDS/ViMax repository implements a sophisticated pipeline for maintaining visual consistency across shots captured by different cameras. By modeling camera relationships as a tree structure and leveraging LLM-driven video generation, ViMax bridges the gap between disparate camera angles without manual intervention.

## Understanding the Camera Tree Architecture

ViMax represents multi-camera scenes as a **camera tree**, where each node may optionally reference a **parent camera** whose footage already covers the visual content of the child camera. This hierarchy is established through the `CameraImageGenerator.construct_camera_tree` method in [`agents/camera_image_generator.py`](https://github.com/HKUDS/ViMax/blob/main/agents/camera_image_generator.py), which processes camera definitions from [`interfaces/camera.py`](https://github.com/HKUDS/ViMax/blob/main/interfaces/camera.py).

When a shot's camera has a parent (`camera.parent_shot_idx` is not `None`), the system identifies the need for a transitional bridge. The parent shot provides the visual context, while the child shot requires a seamless entry point that matches the new camera's composition, background, and lighting.

## The Three-Stage Transition Pipeline

ViMax handles camera transitions through a deterministic three-stage workflow managed by `CameraImageGenerator` and orchestrated within `Script2VideoPipeline.generate_frames_for_single_camera`.

### Step 1: Generating the Transition Video

The `generate_transition_video` method constructs a specialized prompt describing both the parent and child shots, then invokes the video generator to produce a short "cut-to" clip.

```python
prompt = (
    "Two shots. The transition between the shots is a cut to. "
    "The style of the two shots should be consistent.\n"
    f"The first shot description: {first_shot_visual_desc}.\n"
    f"The second shot description: {second_shot_visual_desc}."
)

```

The system supplies the first-shot frame of the parent camera (`first_shot_ff_path`) as a reference image, ensuring stylistic consistency between the source and destination shots.

### Step 2: Deriving the New Camera Image

Once the transition video exists, `get_new_camera_image` performs scene detection to extract a usable reference frame for the child camera. This method uses `ContentDetector` from the scenedetect library to identify scene boundaries within the transition clip.

```python
video = open_video(transition_video_path)
scene_manager = SceneManager()
scene_manager.add_detector(ContentDetector())
scene_manager.detect_scenes(video, show_progress=False)
scene_list = scene_manager.get_scene_list()
split_video_ffmpeg(transition_video_path, scene_list, output_dir, show_progress=True)

video_name = os.path.basename(transition_video_path).split('.')[0]
second_video_path = os.path.join(output_dir, f"{video_name}-Scene-002.mp4")
if os.path.exists(second_video_path):
    # use first frame of the second scene

    clip = VideoFileClip(second_video_path)
    frame = Image.fromarray(clip.get_frame(0).astype('uint8'), 'RGB')
else:
    # fallback to the last frame of the whole transition clip

    clip = VideoFileClip(transition_video_path)
    t = max(0, clip.duration - (1 / clip.fps))
    frame = Image.fromarray(clip.get_frame(t).astype('uint8'), 'RGB')
return ImageOutput(fmt="pil", ext="png", data=frame)

```

The method prioritizes the first frame of the second scene (representing the child camera's view) but falls back to the last frame of the transition clip if scene detection fails. The resulting `ImageOutput` becomes the reference image for the child's first frame generation.

### Step 3: First-Frame Synthesis

In `Script2VideoPipeline.generate_frames_for_single_camera`, ViMax integrates the extracted new camera image into the generation workflow. The system either uses this image alongside character portraits to select reference images and craft a generation prompt, or copies the generated image directly when the child camera has no missing information.

## Implementation in Script2VideoPipeline

The orchestration logic in [`pipelines/script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/script2video_pipeline.py) handles the synchronization between parent and child shots. The pipeline waits for the parent's first frame to be ready before initiating the transition process.

```python
if camera.parent_shot_idx is not None:
    # wait for the parent’s first frame

    await self.frame_events[parent_shot_idx]["first_frame"].wait()
    parent_shot_ff_path = os.path.join(self.working_dir, "shots",
                                      f"{parent_shot_idx}", "first_frame.png")
    transition_path = os.path.join(self.working_dir, "shots",
                                   f"{first_shot_idx}",
                                   f"transition_video_from_shot_{parent_shot_idx}.mp4")

    # ① Create transition video (cut‑to)

    transition_video = await self.camera_image_generator.generate_transition_video(
        first_shot_visual_desc=shot_descriptions[parent_shot_idx].visual_desc,
        second_shot_visual_desc=shot_descriptions[first_shot_idx].visual_desc,
        first_shot_ff_path=parent_shot_ff_path,
    )
    transition_video.save(transition_path)

    # ② Derive a new camera image from the transition clip

    new_camera_img = self.camera_image_generator.get_new_camera_image(
        transition_video_path=transition_path
    )
    new_camera_img.save(os.path.join(self.working_dir, "shots",
                                     f"{first_shot_idx}",
                                     f"new_camera_{camera.idx}.png"))

```

This implementation ensures that child cameras never generate content in isolation; they always inherit visual context from their parent transitions.

## Summary

- ViMax models multi-camera relationships as a **camera tree** with parent-child hierarchies defined in [`interfaces/camera.py`](https://github.com/HKUDS/ViMax/blob/main/interfaces/camera.py).
- The `construct_camera_tree` method in [`agents/camera_image_generator.py`](https://github.com/HKUDS/ViMax/blob/main/agents/camera_image_generator.py) determines which shots require transitional bridges.
- **Transition videos** are generated using LLM-driven prompts that describe both source and destination shots.
- Scene detection extracts the **new camera image** from the transition video, providing a contextually appropriate starting frame for the child camera.
- `Script2VideoPipeline.generate_frames_for_single_camera` orchestrates the workflow, ensuring parent frames are ready before child processing begins.

## Frequently Asked Questions

### How does ViMax determine which cameras need transition videos?

ViMax checks the `parent_shot_idx` attribute of each camera object defined in [`interfaces/camera.py`](https://github.com/HKUDS/ViMax/blob/main/interfaces/camera.py). If this value is not `None`, the camera has a parent shot, and the system automatically triggers the transition video generation process through `CameraImageGenerator.construct_camera_tree`.

### What happens if the transition video contains only one scene?

The `get_new_camera_image` method includes fallback logic for single-scene transitions. When the expected second scene file (`Scene-002.mp4`) does not exist, the system extracts the last frame of the entire transition clip using `VideoFileClip.get_frame()` with a timestamp calculated as `duration - (1 / fps)`.

### Can ViMax handle transitions between more than two cameras?

Yes, the camera tree structure supports arbitrary depth. Each camera can have its own parent, creating chains of transitions. The pipeline processes these sequentially, ensuring that each child camera waits for its parent frame to be generated before beginning its own transition workflow.

### Where is the transition prompt constructed in the source code?

The prompt template resides within `CameraImageGenerator.generate_transition_video` in [`agents/camera_image_generator.py`](https://github.com/HKUDS/ViMax/blob/main/agents/camera_image_generator.py). The method concatenates a fixed descriptor ("Two shots. The transition between the shots is a cut to...") with the specific visual descriptions of the parent and child shots supplied by the script pipeline.