ViMax Idea2Video vs Script2Video Pipelines: Architecture and Usage Guide

Idea2VideoPipeline is a high-level orchestrator that transforms raw concepts into full movies by generating stories and delegating scene rendering to Script2VideoPipeline, which handles the granular script-to-video workflow including storyboarding, frame generation, and shot composition.

The HKUDS/ViMax repository implements a two-stage video generation architecture that separates high-level narrative development from low-level visual production. Understanding the distinction between Idea2Video and Script2Video pipelines is essential for effectively using the framework, whether you are generating videos from simple prompts or fine-tuning specific scenes from existing screenplays.

Architectural Overview

ViMax employs a hierarchical design where Idea2VideoPipeline serves as the entry point for end-to-end creation, while Script2VideoPipeline functions as the specialized rendering engine. This separation allows users to either automate the entire creative process or intervene at specific stages with custom scripts.

Aspect	Idea2VideoPipeline	Script2VideoPipeline
Input	Raw idea (str) + user requirements	Complete script (str or structured list)
Core Logic	Story development, character extraction, scene splitting	Storyboard design, camera tree construction, frame/video generation
Sub-components	Creates Screenwriter, CharacterExtractor, and CharacterPortraitsGenerator once per pipeline	Creates StoryboardArtist, CameraImageGenerator, and ReferenceImageSelector per instance
Output	Single final_video.mp4 concatenating all scenes	Single final_video.mp4 for one specific scene
Source File	pipelines/idea2video_pipeline.py	pipelines/script2video_pipeline.py

Both pipelines inherit configuration-driven initialization through init_from_config(), loading chat models, image generators, and video backends from YAML specifications.

How Idea2VideoPipeline Works

The Idea2VideoPipeline operates as the top-level coordinator in pipelines/idea2video_pipeline.py. Its __call__ method implements a six-stage workflow that bridges the gap between abstract concepts and executable scene scripts.

The pipeline executes the following sequence:

1. Develop Story: Invokes develop_story(idea, user_requirement) to expand the raw concept into a structured narrative.
2. Extract Characters: Calls extract_characters(story) to identify dramatis personae from the generated narrative.
3. Generate Portraits: Uses generate_character_portraits(characters, None, style) to create consistent visual references for each character.
4. Write Scene Scripts: Executes write_script_based_on_story(story, user_requirement) to split the narrative into discrete scene-level scripts.
5. Delegate Rendering: Instantiates a Script2VideoPipeline for each scene script and executes it asynchronously.
6. Concatenate Results: Stitches individual scene videos into the final movie file.

# From pipelines/idea2video_pipeline.py (simplified flow)

story = await self.develop_story(idea, user_requirement)
characters = await self.extract_characters(story)
character_portraits = await self.generate_character_portraits(characters, None, style)
scene_scripts = await self.write_script_based_on_story(story, user_requirement)

for idx, scene_script in enumerate(scene_scripts):
    scene_dir = os.path.join(self.working_dir, f"scene_{idx}")
    script2video = Script2VideoPipeline(
        chat_model=self.chat_model,
        image_generator=self.image_generator,
        video_generator=self.video_generator,
        working_dir=scene_dir,
    )
    final_path = await script2video(
        script=scene_script,
        user_requirement=user_requirement,
        style=style,
        characters=characters,
        character_portraits_registry=character_portraits,
    )

Intermediate artifacts including story.txt, characters.json, and portrait registries are cached in the working directory to enable resumable execution.

How Script2VideoPipeline Works

The Script2VideoPipeline in pipelines/script2video_pipeline.py serves as the mid-level execution engine that transforms structured scripts into actual video content. This pipeline handles the granular visual decisions required for cinematographic production.

The implementation processes scripts through eight distinct phases:

1. Character Extraction (optional): extract_characters(script) parses dramatis personae if not provided by the parent pipeline.
2. Portrait Management: Loads or generates character portraits via generate_character_portraits().
3. Storyboard Design: design_storyboard(script, characters, user_requirement) creates shot briefs and scene composition plans.
4. Visual Decomposition: decompose_visual_descriptions(storyboard, characters) translates narrative shots into detailed visual specifications.
5. Camera Tree Construction: construct_camera_tree(shot_descriptions) establishes shot ordering, transitions, and camera relationships.
6. Frame Generation: generate_frames_for_single_camera() produces first and last frames for each shot using the image generator.
7. Video Generation: generate_video_for_single_shot() creates motion sequences from the generated frames.
8. Video Assembly: Concatenates all shot videos into the scene's final_video.mp4.

# From pipelines/script2video_pipeline.py (execution flow)

if characters is None:
    characters = await self.extract_characters(script)

storyboard = await self.design_storyboard(script, characters, user_requirement)
shot_descriptions = await self.decompose_visual_descriptions(storyboard, characters)
camera_tree = await self.construct_camera_tree(shot_descriptions)

# Parallel execution of frame and video generation

await self.generate_frames_for_single_camera(...)
await self.generate_video_for_single_shot(...)

final_video_path = os.path.join(self.working_dir, "final_video.mp4")

The pipeline utilizes asyncio for parallel I/O operations across LLM calls and generative model inference, significantly reducing latency for multi-shot scenes.

Key Differences and Use Cases

Choosing between these ViMax pipelines depends on your starting point and desired level of control.

Use Idea2VideoPipeline when:

• Starting from a high-level concept (e.g., "a sci-fi adventure about AI")
• Requiring automated story development and character creation
• Generating end-to-end content with minimal manual intervention
• Creating full movies where scene boundaries should be determined algorithmically

Use Script2VideoPipeline when:

• Working with pre-written screenplays or specific scene scripts
• Fine-tuning individual scenes with custom requirements
• Replacing specific scenes in previously generated content
• Implementing custom narrative structures that bypass automatic story generation

The Idea2VideoPipeline essentially automates the pre-production phase that Script2VideoPipeline assumes has been completed.

Practical Implementation Examples

Running the Full Idea-to-Video Workflow

Use main_idea2video.py or instantiate the pipeline directly for end-to-end generation:

import asyncio
from pipelines.idea2video_pipeline import Idea2VideoPipeline

# Initialize from YAML configuration

pipeline = Idea2VideoPipeline.init_from_config("config.yaml")

# Execute full pipeline

final_movie = asyncio.run(
    pipeline(
        idea="A brave explorer discovers an ancient hidden city on Mars",
        user_requirement="5-minute runtime, epic tone, sci-fi style",
        style="cinematic"
    )
)

print(f"Movie saved to: {final_movie}")

Processing Individual Scripts

For scene-specific rendering using main_script2video.py:

import asyncio
from pipelines.script2video_pipeline import Script2VideoPipeline

pipeline = Script2VideoPipeline.init_from_config("config.yaml")

script = [
    {"scene": "INTRO", "dialogue": "Our hero gazes at the red dunes..."},
    {"scene": "RISING_ACTION", "dialogue": "The ground trembles beneath..."}
]

scene_video = asyncio.run(
    pipeline(
        script=script,
        user_requirement="Dramatic pacing, 1080p resolution",
        style="cinematic"
    )
)

print(f"Scene video saved to: {scene_video}")

Hybrid Workflow: Regenerating Specific Scenes

Combine both pipelines to refine specific segments while preserving the overall narrative:

import asyncio
from pipelines.idea2video_pipeline import Idea2VideoPipeline
from pipelines.script2video_pipeline import Script2VideoPipeline

# Generate initial movie

idea_pipe = Idea2VideoPipeline.init_from_config("config.yaml")
movie_path = asyncio.run(
    idea_pipe(
        idea="A detective solves a mystery in a cyberpunk city",
        user_requirement="15-minute runtime, noir vibe",
        style="noir"
    )
)

# Re-render scene 2 with different parameters

script_pipe = Script2VideoPipeline.init_from_config("config.yaml")
scene_2_script = [...]  # Load from previous working_dir/scene_2/

new_scene = asyncio.run(
    script_pipe(
        script=scene_2_script,
        user_requirement="Add rain effects, darker palette",
        style="noir-dark"
    )
)

Summary

The Idea2Video and Script2Video pipelines in ViMax represent distinct abstraction layers within a hierarchical video generation system:

• Idea2VideoPipeline (pipelines/idea2video_pipeline.py) orchestrates the complete creative workflow from raw concepts to final movies, automatically handling story development, character extraction, and scene segmentation.
• Script2VideoPipeline (pipelines/script2video_pipeline.py) executes the technical rendering pipeline, converting structured scripts into video through storyboarding, camera tree construction, and frame generation.
• Architectural Relationship: The high-level pipeline instantiates the mid-level pipeline for each scene, creating a parent-child execution model where Idea2VideoPipeline manages narrative coherence while Script2VideoPipeline handles cinematographic execution.
• Configuration: Both classes utilize init_from_config() for dependency injection, sharing the same chat model, image generator, and video generator backends defined in YAML configuration files.

Frequently Asked Questions

What is the main entry point file for running Idea2Video in ViMax?

The primary entry point is main_idea2video.py, which demonstrates how to instantiate Idea2VideoPipeline from configuration and execute the full workflow. Alternatively, import Idea2VideoPipeline directly from pipelines/idea2video_pipeline.py for programmatic use.

Can I use Script2VideoPipeline without first running Idea2VideoPipeline?

Yes. Script2VideoPipeline is designed to operate independently when provided with a structured script. You can manually construct scene scripts or load existing screenplays and pass them directly to the pipeline's __call__ method without invoking the story generation logic of Idea2VideoPipeline.

How does ViMax handle character consistency between multiple scenes?

Idea2VideoPipeline extracts characters once during initialization and generates portraits via generate_character_portraits(), storing them in a character_portraits_registry. This registry is passed to each Script2VideoPipeline instance, ensuring visual consistency across all scenes in the final movie.

Where are intermediate files like storyboards and frame images stored?

Each pipeline maintains a dedicated working directory specified during initialization. Idea2VideoPipeline stores story.txt, characters.json, and per-scene subdirectories. Script2VideoPipeline caches storyboard JSONs, shot descriptions, and generated frames within its own working directory, enabling resumable execution and post-generation inspection.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}