ViMax Novel2Video Pipeline: Technical Deep Dive into Long-Form Narrative Compression

ViMax converts full-length novels into short cinematic videos by first compressing source text into a condensed "story-core" that preserves essential plot points, character arcs, and dialogue while eliminating redundant description and internal monologue.

ViMax is an open-source narrative-to-video system hosted in the HKUDS/ViMax repository. The Novel2Video pipeline, orchestrated in pipelines/novel2movie_pipeline.py, solves the fundamental challenge of processing book-length texts that exceed standard LLM context windows by implementing a sophisticated three-stage compression workflow.

Three-Stage Novel2Video Compression Architecture

The Novel2Video pipeline processes raw novel text through a structured workflow implemented in the NovelCompressor class located in agents/novel_compressor.py.

Stage 1: Intelligent Chunking and Parallel Compression

The pipeline begins by splitting the novel into manageable fragments. The NovelCompressor.split method initializes a RecursiveCharacterTextSplitter with parameters optimized for long-form narrative preservation:

self.splitter = RecursiveCharacterTextSplitter(
    chunk_size=65536,  # 65KB chunks

    chunk_overlap=8192  # 8KB overlap maintains continuity

)
novel_chunks = self.splitter.split_text(novel_text)

Each chunk is processed by compress_single_novel_chunk, which wraps text in XML tags (<NOVEL_CHUNK_START> and <NOVEL_CHUNK_END>) and sends it to the LLM with specialized system prompts (system_prompt_template_compress_novel_chunk and human_prompt_template_compress_novel_chunk). These prompts instruct the model to retain major plot points, character actions, and dialogue while stripping non-essential narration.

To maximize throughput, the pipeline limits concurrent API calls to five simultaneous requests using asyncio.Semaphore:

sem = asyncio.Semaphore(max_concurrent_tasks)
tasks = [self.compress_single_novel_chunk(sem, i, chunk) 
         for i, chunk in index_chunk_pairs]
compressed_novel_chunks = await asyncio.gather(*tasks)

Stage 2: Aggregation and Deduplication

After parallel compression, the aggregate method combines individual chunks into a single coherent narrative. This stage uses system_prompt_template_aggregate to instruct the LLM to seamlessly merge sequential text fragments while intelligently handling overlapping or duplicated content between chunks. The pipeline demarcates chunks with indexed XML tags (<CHUNK_N_START> and <CHUNK_N_END>) to preserve temporal sequence during merging.

Stage 3: Downstream Video Synthesis

The resulting aggregated_novel serves as the compressed story-core that drives subsequent pipeline stages, including event extraction (EventExtractor), scene generation, character portrait creation, and final video synthesis. The system stores both compressed chunks and the aggregated story on disk, enabling incremental reuse—the pipeline automatically skips recompression if output files already exist, making it safe to resume after interruption.

Technical Implementation Details

Prompt Engineering Strategy

The compression logic relies on carefully crafted prompts defined in agents/novel_compressor.py (lines 11-38). The system prompt establishes the LLM as a "text-compression assistant," while the human prompt provides strict formatting instructions using XML delimiters. This structured approach ensures consistent output formatting across all chunks, which is critical for successful aggregation.

Separation of Concerns

The architecture deliberately separates compression from knowledge-base construction. This design allows the same compressed short story to power both semantic search indexing (using FAISS) and script generation workflows, maximizing utility while minimizing API token consumption and costs.

Practical Code Examples

Compress a Novel Stand-Alone

To use the compression engine independently of the full video pipeline:

from agents.novel_compressor import NovelCompressor
import asyncio

# Initialize with LLM credentials

compressor = NovelCompressor(
    api_key="YOUR_API_KEY",
    base_url="https://api.openai.com/v1",
    chat_model="gpt-4o-mini"
)

# Load and split novel

with open("my_novel.txt", "r", encoding="utf-8") as f:
    novel_text = f.read()
chunks = compressor.split(novel_text)

# Parallel compression (max 5 concurrent tasks)

compressed_chunks = await compressor.compress(
    index_chunk_pairs=list(enumerate(chunks)),
    max_concurrent_tasks=5,
)

# Aggregate into final short story

short_story = compressor.aggregate(
    [c for _, c in sorted(compressed_chunks)]
)
print(short_story)

Run the Complete Novel2Video Pipeline

For end-to-end novel-to-video generation:

import asyncio
from pipelines.novel2movie_pipeline import Novel2MoviePipeline
from agents.novel_compressor import NovelCompressor
from agents.event_extractor import EventExtractor

# ... other agent imports ...

pipeline = Novel2MoviePipeline(
    working_dir="workdir/novel2movie",
    novel_compressor=NovelCompressor(
        api_key="...",
        base_url="...",
        chat_model="gpt-4o-mini"
    ),
    event_extractor=EventExtractor(...),
    # ... other agents (scene_extractor, etc.) ...

)

novel_text = open("my_novel.txt", "r", encoding="utf-8").read()
await pipeline(novel_text=novel_text, style="cinematic realistic style")

Summary

• The Novel2Video pipeline in pipelines/novel2movie_pipeline.py compresses novels using a three-stage workflow: chunking, parallel LLM compression, and intelligent aggregation.
• Chunking parameters use 65KB segments with 8KB overlap to balance context preservation with API constraints.
• Concurrent processing limits API calls to five simultaneous requests using asyncio.Semaphore for efficient throughput.
• Prompt engineering leverages XML tagging and role-based system prompts to ensure narrative coherence and consistent formatting across fragments.
• Incremental reuse stores compressed outputs on disk, enabling pipeline resumption and cost savings across multiple video generation attempts.

Frequently Asked Questions

How does ViMax handle novels that exceed standard LLM context windows?

ViMax implements recursive character splitting with substantial overlap (65KB chunks with 8KB overlap) to ensure narrative continuity across fragment boundaries. The NovelCompressor class processes these chunks in parallel with a concurrency limit of five, then aggregates them using a dedicated LLM prompt designed to seamlessly merge overlapping content while preserving chronological flow and character consistency.

What specific narrative elements does the compression preserve?

According to the system prompts in agents/novel_compressor.py, the compression explicitly retains major plot points, character actions, and dialogue while stripping redundant description, internal monologue, and non-narrative text. The aggregation stage further refines this by deduplicating content that appears in multiple chunk windows, ensuring the final "story-core" maintains the emotional and structural integrity of the original novel.

Can compressed outputs be reused for different video generation attempts?

Yes. The pipeline writes compressed chunks and the aggregated short story to disk in the specified working directory. On subsequent runs, the system checks for existing files and skips the compression phase entirely, allowing users to experiment with different visual styles, scene selections, or event extractions without incurring additional LLM costs for text compression.

What is the performance impact of the 65KB chunk size selection?

The 65KB chunk size represents an optimization between context window utilization and API reliability. This size provides sufficient context for the LLM to understand character relationships and plot developments within each fragment while remaining well within typical token limits, preventing request timeouts and ensuring consistent response formatting across the parallel compression workers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}