How ViMax Extracts Characters from Scripts and Novels: A Complete Technical Guide

ViMax uses two distinct LLM-powered pipelines to extract characters from scripts and novels—direct Pydantic parsing for screenplays via CharacterExtractor and incremental scene-level merging via GlobalInformationPlanner for novels.

ViMax is an open-source visual storytelling framework developed by HKUDS that transforms text into cinematic storyboards. To extract characters from scripts and novels reliably, the codebase implements specialized agents in the agents/ directory that leverage LangChain chat models, structured prompting, and Pydantic output parsing for type-safe character metadata extraction.

Script Character Extraction

For movie scripts, ViMax employs a single-pass extraction strategy that parses the entire screenplay in one LLM invocation.

The CharacterExtractor Agent

The CharacterExtractor class in agents/character_extractor.py (lines 15-55) handles script processing through a structured prompt engineering approach:

1. System prompt – Defines the extractor’s role, output format requirements, and distinguishes between static features (physical attributes) and dynamic features (emotional states, visibility).
2. Human prompt – Wraps the raw script content within <SCRIPT> … </SCRIPT> XML tags to delimit the input context.
3. Pydantic validation – Uses PydanticOutputParser expecting an ExtractCharactersResponse containing a list of CharacterInScene objects (defined at lines 57-61).

The extracted characters are persisted to working_dir/characters.json for downstream pipeline stages, as implemented in pipelines/script2video_pipeline.py (lines 25-40) where self.character_extractor.extract_characters(script) is invoked.

from agents.character_extractor import CharacterExtractor
from langchain.chat_models import init_chat_model

chat = init_chat_model(model="gpt-4o-mini")
extractor = CharacterExtractor(chat_model=chat)

script = """
INT. CAFE – DAY
A young woman sips coffee, eyes distant...
"""
characters = await extractor.extract_characters(script)
print(characters)   # → List[CharacterInScene] with static/dynamic features

Novel Character Extraction

Novel processing requires a more complex, multi-stage approach because source material lacks explicit scene boundaries and dialogue markers.

Multi-Stage Architecture

The GlobalInformationPlanner in agents/global_information_planner.py orchestrates character discovery through four sequential phases:

1. Novel compression – NovelCompressor splits and compresses large texts to fit context windows.
2. Event extraction – Identifies narrative events from compressed content.
3. Scene generation – SceneExtractor converts events into screenplay-formatted scenes.
4. Hierarchical merging – Combines character references across scenes and events into a canonical novel-wide list.

Merging Characters Across Scenes and Events

For each narrative event, merge_characters_across_scenes_in_event (lines 58-96) resolves character identity across fragmented scene references. The method serializes scenes using <SCENE_N_START> … </SCENE_N_END> tags and prompts the LLM to identify which character mentions refer to the same entity, resolving ambiguous naming and evolving descriptions.

Global Novel-Level Consolidation

The merge_characters_to_existing_characters_in_novel method (lines 20-72) performs the final integration step. It constructs structured prompts containing <EXISTING_CHARACTERS> and <EVENT_CHARACTERS> sections, enabling the LLM to update the global character catalogue with new entries or merge features when the same character appears in multiple events. This logic is invoked iteratively within pipelines/novel2movie_pipeline.py (lines 96-124).

from agents.global_information_planner import GlobalInformationPlanner
from langchain.chat_models import init_chat_model

chat = init_chat_model(model="gpt-4o-mini")
planner = GlobalInformationPlanner(
    api_key="YOUR_KEY",
    base_url="https://api.openai.com/v1",
    chat_model="gpt-4o-mini",
)

# Merge characters within a single event from multiple scenes

merged_event_chars = await planner.merge_characters_across_scenes_in_event(
    event_idx=0,
    scenes=scenes,
)

# Integrate event-level characters into the novel-wide roster

novel_chars = await planner.merge_characters_to_existing_characters_in_novel(
    event_idx=0,
    existing_characters_in_novel=novel_chars,
    characters_in_event=merged_event_chars,
)

print(novel_chars)   # → List[CharacterInNovel] representing the whole novel

Key Differences: Script vs. Novel Processing

ViMax maintains separate extraction paths because the input characteristics differ fundamentally:

• Script extraction is immediate. Screenplays already delineate dialogue headers (e.g., INT. CAFE – DAY) and character names, allowing CharacterExtractor to map mentions directly to CharacterInScene objects in a single LLM pass.
• Novel extraction is discover. Prose lacks explicit scene boundaries, requiring GlobalInformationPlanner to first identify events, generate scenes, and then merge fragmented character references across the narrative timeline to maintain consistency.

Both implementations rely on LangChain chat models and Pydantic for type-safe output validation, but the novel pipeline adds layers of indirection to handle the ambiguity and scale of book-length content.

Summary

• Script extraction uses CharacterExtractor (agents/character_extractor.py) with <SCRIPT> XML tags and direct Pydantic parsing into CharacterInScene objects.
• Novel extraction relies on GlobalInformationPlanner (agents/global_information_planner.py) to merge characters hierarchically across scenes and events using structured tags like <SCENE_N_START> and <EXISTING_CHARACTERS>.
• Pipeline integration occurs in script2video_pipeline.py and novel2movie_pipeline.py, with results cached for downstream video generation stages.
• Type safety is enforced through PydanticOutputParser and response models including ExtractCharactersResponse and CharacterInNovel.

Frequently Asked Questions

What is the difference between CharacterInScene and CharacterInNovel?

CharacterInScene (defined in agents/character_extractor.py, lines 57-61) represents character metadata extracted from a specific screenplay segment, capturing immediate context like visibility and emotional state. CharacterInNovel represents the canonical character entity maintained across the entire book, containing accumulated attributes from multiple events and scenes through the merging process in GlobalInformationPlanner.

How does ViMax handle character consistency across multiple novel chapters?

ViMax maintains consistency through incremental merging. The merge_characters_to_existing_characters_in_novel method compares new event characters against the existing novel-wide roster using <EXISTING_CHARACTERS> and <EVENT_CHARACTERS> prompt sections. The LLM determines whether to create new entries or update existing character profiles when encountering alternate names, physical description changes, or new narrative roles across different chapters.

Why does ViMax use PydanticOutputParser instead of raw JSON?

According to the HKUDS/ViMax source code, PydanticOutputParser guarantees type-safe deserialization of LLM outputs into structured Python objects like ExtractCharactersResponse. This prevents runtime errors from malformed JSON, enforces schema validation for required fields (e.g., character names, static features), and enables IDE autocomplete and type checking throughout the pipeline in script2video_pipeline.py and novel2movie_pipeline.py.

Can I use custom LLM models with ViMax's character extraction?

Yes. Both CharacterExtractor and GlobalInformationPlanner accept any LangChain-compatible chat model through the chat_model parameter or init_chat_model() configuration. You can substitute gpt-4o-mini with other OpenAI models, local LLMs via Ollama, or alternative providers by adjusting the model string and API configuration in the constructor calls.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}