# ViMax Idea2Video vs Script2Video Pipelines: Architecture and Usage Guide

> Understand ViMax Idea2Video vs Script2Video pipelines. Learn how Idea2Video orchestrates movie creation from concepts while Script2Video handles granular script-to-video rendering and scene composition.

- Repository: [✨Data Intelligence Lab@HKU✨/ViMax](https://github.com/HKUDS/ViMax)
- Tags: architecture
- Published: 2026-05-20

---

**`Idea2VideoPipeline` is a high-level orchestrator that transforms raw concepts into full movies by generating stories and delegating scene rendering to `Script2VideoPipeline`, which handles the granular script-to-video workflow including storyboarding, frame generation, and shot composition.**

The HKUDS/ViMax repository implements a **two-stage video generation architecture** that separates high-level narrative development from low-level visual production. Understanding the distinction between Idea2Video and Script2Video pipelines is essential for effectively using the framework, whether you are generating videos from simple prompts or fine-tuning specific scenes from existing screenplays.

## Architectural Overview

ViMax employs a hierarchical design where `Idea2VideoPipeline` serves as the entry point for end-to-end creation, while `Script2VideoPipeline` functions as the specialized rendering engine. This separation allows users to either automate the entire creative process or intervene at specific stages with custom scripts.

| Aspect | `Idea2VideoPipeline` | `Script2VideoPipeline` |
|--------|----------------------|------------------------|
| **Input** | Raw idea (`str`) + user requirements | Complete script (`str` or structured list) |
| **Core Logic** | Story development, character extraction, scene splitting | Storyboard design, camera tree construction, frame/video generation |
| **Sub-components** | Creates `Screenwriter`, `CharacterExtractor`, and `CharacterPortraitsGenerator` once per pipeline | Creates `StoryboardArtist`, `CameraImageGenerator`, and `ReferenceImageSelector` per instance |
| **Output** | Single `final_video.mp4` concatenating all scenes | Single `final_video.mp4` for one specific scene |
| **Source File** | [`pipelines/idea2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/idea2video_pipeline.py) | [`pipelines/script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/script2video_pipeline.py) |

Both pipelines inherit configuration-driven initialization through `init_from_config()`, loading chat models, image generators, and video backends from YAML specifications.

## How Idea2VideoPipeline Works

The `Idea2VideoPipeline` operates as the **top-level coordinator** in [`pipelines/idea2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/idea2video_pipeline.py). Its `__call__` method implements a six-stage workflow that bridges the gap between abstract concepts and executable scene scripts.

The pipeline executes the following sequence:

1. **Develop Story**: Invokes `develop_story(idea, user_requirement)` to expand the raw concept into a structured narrative.
2. **Extract Characters**: Calls `extract_characters(story)` to identify dramatis personae from the generated narrative.
3. **Generate Portraits**: Uses `generate_character_portraits(characters, None, style)` to create consistent visual references for each character.
4. **Write Scene Scripts**: Executes `write_script_based_on_story(story, user_requirement)` to split the narrative into discrete scene-level scripts.
5. **Delegate Rendering**: Instantiates a `Script2VideoPipeline` for each scene script and executes it asynchronously.
6. **Concatenate Results**: Stitches individual scene videos into the final movie file.

```python

# From pipelines/idea2video_pipeline.py (simplified flow)

story = await self.develop_story(idea, user_requirement)
characters = await self.extract_characters(story)
character_portraits = await self.generate_character_portraits(characters, None, style)
scene_scripts = await self.write_script_based_on_story(story, user_requirement)

for idx, scene_script in enumerate(scene_scripts):
    scene_dir = os.path.join(self.working_dir, f"scene_{idx}")
    script2video = Script2VideoPipeline(
        chat_model=self.chat_model,
        image_generator=self.image_generator,
        video_generator=self.video_generator,
        working_dir=scene_dir,
    )
    final_path = await script2video(
        script=scene_script,
        user_requirement=user_requirement,
        style=style,
        characters=characters,
        character_portraits_registry=character_portraits,
    )

```

Intermediate artifacts including [`story.txt`](https://github.com/HKUDS/ViMax/blob/main/story.txt), [`characters.json`](https://github.com/HKUDS/ViMax/blob/main/characters.json), and portrait registries are cached in the working directory to enable resumable execution.

## How Script2VideoPipeline Works

The `Script2VideoPipeline` in [`pipelines/script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/script2video_pipeline.py) serves as the **mid-level execution engine** that transforms structured scripts into actual video content. This pipeline handles the granular visual decisions required for cinematographic production.

The implementation processes scripts through eight distinct phases:

1. **Character Extraction** (optional): `extract_characters(script)` parses dramatis personae if not provided by the parent pipeline.
2. **Portrait Management**: Loads or generates character portraits via `generate_character_portraits()`.
3. **Storyboard Design**: `design_storyboard(script, characters, user_requirement)` creates shot briefs and scene composition plans.
4. **Visual Decomposition**: `decompose_visual_descriptions(storyboard, characters)` translates narrative shots into detailed visual specifications.
5. **Camera Tree Construction**: `construct_camera_tree(shot_descriptions)` establishes shot ordering, transitions, and camera relationships.
6. **Frame Generation**: `generate_frames_for_single_camera()` produces first and last frames for each shot using the image generator.
7. **Video Generation**: `generate_video_for_single_shot()` creates motion sequences from the generated frames.
8. **Video Assembly**: Concatenates all shot videos into the scene's `final_video.mp4`.

```python

# From pipelines/script2video_pipeline.py (execution flow)

if characters is None:
    characters = await self.extract_characters(script)

storyboard = await self.design_storyboard(script, characters, user_requirement)
shot_descriptions = await self.decompose_visual_descriptions(storyboard, characters)
camera_tree = await self.construct_camera_tree(shot_descriptions)

# Parallel execution of frame and video generation

await self.generate_frames_for_single_camera(...)
await self.generate_video_for_single_shot(...)

final_video_path = os.path.join(self.working_dir, "final_video.mp4")

```

The pipeline utilizes **asyncio** for parallel I/O operations across LLM calls and generative model inference, significantly reducing latency for multi-shot scenes.

## Key Differences and Use Cases

Choosing between these **ViMax pipelines** depends on your starting point and desired level of control.

**Use `Idea2VideoPipeline` when:**
- Starting from a high-level concept (e.g., *"a sci-fi adventure about AI"*)
- Requiring automated story development and character creation
- Generating end-to-end content with minimal manual intervention
- Creating full movies where scene boundaries should be determined algorithmically

**Use `Script2VideoPipeline` when:**
- Working with pre-written screenplays or specific scene scripts
- Fine-tuning individual scenes with custom requirements
- Replacing specific scenes in previously generated content
- Implementing custom narrative structures that bypass automatic story generation

The `Idea2VideoPipeline` essentially automates the pre-production phase that `Script2VideoPipeline` assumes has been completed.

## Practical Implementation Examples

### Running the Full Idea-to-Video Workflow

Use [`main_idea2video.py`](https://github.com/HKUDS/ViMax/blob/main/main_idea2video.py) or instantiate the pipeline directly for end-to-end generation:

```python
import asyncio
from pipelines.idea2video_pipeline import Idea2VideoPipeline

# Initialize from YAML configuration

pipeline = Idea2VideoPipeline.init_from_config("config.yaml")

# Execute full pipeline

final_movie = asyncio.run(
    pipeline(
        idea="A brave explorer discovers an ancient hidden city on Mars",
        user_requirement="5-minute runtime, epic tone, sci-fi style",
        style="cinematic"
    )
)

print(f"Movie saved to: {final_movie}")

```

### Processing Individual Scripts

For scene-specific rendering using [`main_script2video.py`](https://github.com/HKUDS/ViMax/blob/main/main_script2video.py):

```python
import asyncio
from pipelines.script2video_pipeline import Script2VideoPipeline

pipeline = Script2VideoPipeline.init_from_config("config.yaml")

script = [
    {"scene": "INTRO", "dialogue": "Our hero gazes at the red dunes..."},
    {"scene": "RISING_ACTION", "dialogue": "The ground trembles beneath..."}
]

scene_video = asyncio.run(
    pipeline(
        script=script,
        user_requirement="Dramatic pacing, 1080p resolution",
        style="cinematic"
    )
)

print(f"Scene video saved to: {scene_video}")

```

### Hybrid Workflow: Regenerating Specific Scenes

Combine both pipelines to refine specific segments while preserving the overall narrative:

```python
import asyncio
from pipelines.idea2video_pipeline import Idea2VideoPipeline
from pipelines.script2video_pipeline import Script2VideoPipeline

# Generate initial movie

idea_pipe = Idea2VideoPipeline.init_from_config("config.yaml")
movie_path = asyncio.run(
    idea_pipe(
        idea="A detective solves a mystery in a cyberpunk city",
        user_requirement="15-minute runtime, noir vibe",
        style="noir"
    )
)

# Re-render scene 2 with different parameters

script_pipe = Script2VideoPipeline.init_from_config("config.yaml")
scene_2_script = [...]  # Load from previous working_dir/scene_2/

new_scene = asyncio.run(
    script_pipe(
        script=scene_2_script,
        user_requirement="Add rain effects, darker palette",
        style="noir-dark"
    )
)

```

## Summary

The **Idea2Video and Script2Video pipelines** in ViMax represent distinct abstraction layers within a hierarchical video generation system:

- **`Idea2VideoPipeline`** ([`pipelines/idea2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/idea2video_pipeline.py)) orchestrates the complete creative workflow from raw concepts to final movies, automatically handling story development, character extraction, and scene segmentation.
- **`Script2VideoPipeline`** ([`pipelines/script2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/script2video_pipeline.py)) executes the technical rendering pipeline, converting structured scripts into video through storyboarding, camera tree construction, and frame generation.
- **Architectural Relationship**: The high-level pipeline instantiates the mid-level pipeline for each scene, creating a parent-child execution model where `Idea2VideoPipeline` manages narrative coherence while `Script2VideoPipeline` handles cinematographic execution.
- **Configuration**: Both classes utilize `init_from_config()` for dependency injection, sharing the same chat model, image generator, and video generator backends defined in YAML configuration files.

## Frequently Asked Questions

### What is the main entry point file for running Idea2Video in ViMax?

The primary entry point is **[`main_idea2video.py`](https://github.com/HKUDS/ViMax/blob/main/main_idea2video.py)**, which demonstrates how to instantiate `Idea2VideoPipeline` from configuration and execute the full workflow. Alternatively, import `Idea2VideoPipeline` directly from [`pipelines/idea2video_pipeline.py`](https://github.com/HKUDS/ViMax/blob/main/pipelines/idea2video_pipeline.py) for programmatic use.

### Can I use Script2VideoPipeline without first running Idea2VideoPipeline?

Yes. **`Script2VideoPipeline`** is designed to operate independently when provided with a structured script. You can manually construct scene scripts or load existing screenplays and pass them directly to the pipeline's `__call__` method without invoking the story generation logic of `Idea2VideoPipeline`.

### How does ViMax handle character consistency between multiple scenes?

`Idea2VideoPipeline` extracts characters once during initialization and generates portraits via `generate_character_portraits()`, storing them in a `character_portraits_registry`. This registry is passed to each `Script2VideoPipeline` instance, ensuring visual consistency across all scenes in the final movie.

### Where are intermediate files like storyboards and frame images stored?

Each pipeline maintains a dedicated **working directory** specified during initialization. `Idea2VideoPipeline` stores [`story.txt`](https://github.com/HKUDS/ViMax/blob/main/story.txt), [`characters.json`](https://github.com/HKUDS/ViMax/blob/main/characters.json), and per-scene subdirectories. `Script2VideoPipeline` caches storyboard JSONs, shot descriptions, and generated frames within its own working directory, enabling resumable execution and post-generation inspection.