ViMax Working Directory Caching: Resume Interrupted Generation Processes

ViMax uses a file-system-based working directory structure to cache intermediate artifacts, allowing generation pipelines to resume automatically from the last completed step after interruptions.

ViMax (HKUDS/ViMax) implements deterministic caching through its working directory architecture, enabling long-running video generation pipelines to survive crashes and manual stops. When instantiating pipelines like Novel2MoviePipeline, Script2VideoPipeline, or Idea2VideoPipeline, you specify a persistent directory where every intermediate artifact—from embedding vectors to scene JSON—is stored. This design allows the system to check for existing files and skip already-completed stages, seamlessly resuming generation from exactly where it left off.

How ViMax Working Directory Caching Works

ViMax partitions the user-provided working directory into semantic sub-folders, each responsible for persisting specific artifact types across pipeline stages.

Directory Structure and Organization

When you initialize a pipeline, ViMax creates a structured hierarchy inside your specified working_dir:

• knowledge_base/ – Stores CacheBackedEmbeddings using a langchain.storage.LocalFileStore for persistent embedding vector caching
• relevant_chunks/ – Contains the top-k text chunks selected for each event after retrieval and reranking
• events/ – Holds JSON files representing extracted narrative events
• scenes/ – Stores scene-level JSON configurations
• character_portraits/ – Caches generated character images
• videos/ – Contains final rendered video outputs

This organization appears in pipelines such as Novel2MoviePipeline, where the constructor establishes these paths using os.makedirs(..., exist_ok=True) to ensure idempotent directory creation.

File-Based Checkpoint Mechanism

The resumability relies on a simple but effective pattern: check-before-compute. Before executing any expensive operation, ViMax verifies whether the expected output files already exist using conditional if os.path.exists(...) checks. If the files are present, the stage prints a skip message (e.g., "⏭️ Skipping event extraction...") and continues to the next phase.

In pipelines/novel2movie_pipeline.py, this pattern appears in the event extraction loop (lines 123–133), where the code checks for existing events/event_<n>.json files before invoking the LLM. Similarly, the chunk retrieval logic (lines 98–110) verifies whether relevant_chunks/event_<n>/ contains files before calling the reranker, reusing saved chunks when available.

Resuming Interrupted Generation in ViMax

The file-system caching enables robust interruption recovery without requiring database snapshots or complex state management.

Automatic Resume Behavior

To resume an interrupted generation:

1. First run – The pipeline creates sub-folders and writes every artifact to disk
2. Interruption – If the process stops (crash, manual stop, or timeout), files already written persist on disk
3. Resume – Re-run the pipeline with the identical working_dir argument; ViMax detects existing artifacts and automatically jumps to the first unfinished step

This behavior is implemented across all pipeline stages. For example, in pipelines/script2video_pipeline.py (lines 30–44), the initialization establishes the working directory with exist_ok=True, allowing subsequent runs to access previously cached data immediately.

Forcing Recomputation of Specific Stages

Because caching is file-system based, you can manually invalidate specific stages by deleting their corresponding sub-directories. Removing a folder forces ViMax to recompute that specific stage while preserving other cached artifacts:

import shutil
import os

# Delete only the knowledge-base cache to force embedding recomputation

shutil.rmtree(os.path.join(working_dir, "knowledge_base"))

# Re-run – ViMax rebuilds the embedding cache but skips other existing stages

pipeline.run(input_novel_path="my_novel.txt")

This granular control allows you to correct errors in specific pipeline phases without regenerating the entire video sequence.

Implementation Details from the Source Code

The caching mechanism relies on specific implementation patterns found in the ViMax repository.

Embedding Cache with LocalFileStore

In pipelines/novel2movie_pipeline.py (lines 152–158), ViMax wraps LangChain embeddings with CacheBackedEmbeddings.from_bytes_store, providing a LocalFileStore(root_path=working_dir_knowledge_base) instance. Once an embedding is computed, it writes to disk; subsequent runs read from this local store instead of recomputing vectors, significantly reducing API costs and latency for repeated generations.

Stage-Specific Skipping Logic

The event extraction module demonstrates the skip pattern explicitly. Before processing event n, the code checks events/event_<n>.json (lines 102–112). If present, the pipeline loads the existing JSON and proceeds. The chunk retrieval module applies identical logic to relevant_chunks/event_<n>/ directories (lines 98–110), avoiding redundant LLM calls when cached chunks satisfy the retrieval criteria.

Practical Code Examples

Running a Full Novel-to-Movie Generation

from pipelines.novel2movie_pipeline import Novel2MoviePipeline

# Choose a persistent directory for this run

working_dir = "/tmp/vimax_run_2026_05_20"

pipeline = Novel2MoviePipeline(
    working_dir=working_dir,
    # other configuration arguments …

)

# First execution – everything will be created

pipeline.run(input_novel_path="my_novel.txt")

Resuming After an Interruption

# Simply call `run` again with the same `working_dir`.

# All previously generated files are kept, and the pipeline

# continues from the first step that has missing artifacts.

pipeline.run(input_novel_path="my_novel.txt")

Forcing a Recompute of the Knowledge Base

import shutil
import os

# Delete only the knowledge-base cache (embeddings)

shutil.rmtree(os.path.join(working_dir, "knowledge_base"))

# Re-run – the pipeline rebuilds the embedding cache,

# but skips all other steps that already have outputs.

pipeline.run(input_novel_path="my_novel.txt")

Summary

• ViMax working directory caching stores all intermediate artifacts—embeddings, chunks, events, and scenes—in a user-specified directory structure using os.makedirs(..., exist_ok=True).
• Resume capability is achieved through conditional os.path.exists(...) checks at each pipeline stage, automatically skipping completed steps when files are detected.
• Granular invalidation allows manual deletion of specific sub-folders (e.g., knowledge_base/) to force recomputation of individual stages without regenerating the entire pipeline.
• Embedding persistence uses CacheBackedEmbeddings with LocalFileStore in pipelines/novel2movie_pipeline.py (lines 152–158) to avoid redundant vector computation across runs.

Frequently Asked Questions

How does ViMax detect which generation steps to skip?

ViMax uses explicit file existence checks before each computational stage. For example, in pipelines/novel2movie_pipeline.py (lines 102–112), the event extraction loop checks for events/event_<n>.json files. If present, it prints a skip message and loads the existing data; if absent, it extracts the event via LLM and writes the JSON file for future runs.

Can I resume generation after deleting specific cache directories?

Yes. Because ViMax caches are file-system based, deleting a sub-directory like knowledge_base/ or relevant_chunks/ forces recomputation of only that specific stage. The pipeline will recreate the deleted files on the next run while skipping all other stages that still have intact output files in the working directory.

Where are embedding vectors cached in ViMax?

Embedding vectors are cached in the knowledge_base/ sub-directory using LangChain's CacheBackedEmbeddings configured with a LocalFileStore. According to the implementation in pipelines/novel2movie_pipeline.py (lines 152–158), this setup persists computed embeddings to disk, allowing subsequent pipeline runs to load vectors directly rather than recomputing them through the embedding API.

What happens if I change the working_dir between runs?

Changing the working_dir argument creates an entirely new cache namespace. ViMax treats the new directory as a fresh generation, recomputing all artifacts from scratch. To resume a previous run, you must use the exact same working_dir path that contains the existing events/, scenes/, and knowledge_base/ folders.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}