How to Use DurationHead for Automatic Frame Count Prediction in LTX-2

The DurationHead is a lightweight regression module that predicts shot duration from frozen connector token streams, eliminating the need to run the full diffusion pipeline for frame-count estimation.

The DurationHead in LTX-2 enables intelligent, content-aware video length prediction directly from encoded caption representations. This article explains how to load, configure, and invoke the DurationHead for automatic frame count prediction based on the official Lightricks/LTX-2 source code.

---

What Is the DurationHead?

The DurationHead is a specialized nn.Module that performs regression on audio and video connector tokens to estimate natural shot duration in seconds. According to the LTX-2 implementation in [packages/ltx-core/src/ltx_core/duration_head/duration_head.py](https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-core/src/ltx_core/duration_head/duration_head.py), it operates on frozen encoder outputs without requiring diffusion model inference.

The architecture follows a cross-attention pooling design:

• Projection layers: Video tokens (4096 → hidden_dim) and audio tokens (2048 → hidden_dim) are projected to shared space
• Modality embeddings: Learned embeddings distinguish audio from video streams
• Learnable query token: Single query cross-attends against concatenated tokens
• Shallow MLP: Maps pooled representation to log-duration; exponentiation yields seconds

This design keeps the module lightweight while capturing semantic cues from both modalities.

---

Loading the DurationPredictor

The DurationPredictor class in [packages/ltx-pipelines/src/ltx_pipelines/utils/blocks.py](https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks.py) provides the user-facing API. It wraps a DurationHead with validation and conversion utilities.

Checkpoint Loading

DurationHead weights are available starting from LTX 2.5 / Gemma 4. The loader uses non-strict reconstruction to detect missing weights early:

import torch
from ltx_pipelines.utils.blocks import DurationPredictor

ckpt_path = "path/to/checkpoint.pt"          # must contain DurationHead weights (LTX 2.5+)

dtype = torch.float16
device = torch.device("cuda")

# Load the predictor; returns `None` if the checkpoint lacks DurationHead

duration_predictor = DurationPredictor.from_checkpoint(
    checkpoint_path=ckpt_path,
    dtype=dtype,
    device=device,
)

if duration_predictor is None:
    raise RuntimeError("Checkpoint does not contain DurationHead – auto-duration unavailable")

The from_checkpoint() method delegates to DurationHeadConfigurator in [packages/ltx-core/src/ltx_core/duration_head/model_configurator.py](https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-core/src/ltx_core/duration_head/model_configurator.py) for deserialization.

---

Obtaining Connector Token Tensors

Before prediction, you must encode your caption through the PromptEncoder to extract connector tokens:

from ltx_pipelines.utils.blocks import PromptEncoder

# Assume `prompt_encoder` is already built from the same checkpoint

caption = "A sunny beach with waves crashing"
video_tokens, audio_tokens = prompt_encoder.encode(caption)   # returns two tensors (or None)

The returned tensors have shapes:

• video_tokens: (B, T_v, 4096) — video connector hidden states
• audio_tokens: (B, T_a, 2048) — audio connector hidden states (may be None)

These tensors remain frozen; no gradients flow through them during duration prediction.

---

Predicting and Clamping Frame Counts

The DurationPredictor.__call__() method handles the full prediction pipeline:

frame_rate = 30.0           # fps used for the generation

min_sec = 1.0               # safety lower bound

max_sec = 12.0              # safety upper bound

num_frames = duration_predictor(
    video_encoding=video_tokens,
    audio_encoding=audio_tokens,
    frame_rate=frame_rate,
    min_seconds=min_sec,
    max_seconds=max_sec,
)

print(f"Predicted {num_frames} frames ({num_frames / frame_rate:.2f} s)")

Internal Processing Steps

1. Head inference: Calls DurationHead.forward() to obtain log-duration
2. Exponentiation: Converts to raw seconds via exp()
3. Batch validation: Asserts batch size is (1,) — per-sample prediction only
4. Range clamping: Applies min_seconds and max_seconds bounds
5. Grid snapping: Converts to frame count matching VAE temporal requirements (8k + 1)

The helper seconds_to_clamped_num_frames() in blocks.py performs the final conversion with proper temporal grid alignment.

---

Using AutoDuration for Seamless Integration

For pipeline integration, LTX-2 provides the AutoDuration dataclass and resolve_num_frames() helper:

from ltx_pipelines.utils.blocks import AutoDuration, resolve_num_frames

# User requested automatic duration with explicit bounds

num_frames_param = AutoDuration(min_seconds=1.0, max_seconds=20.0)

# After encoding caption into `video_tokens` / `audio_tokens`

final_num_frames = resolve_num_frames(
    num_frames=num_frames_param,
    duration_predictor=duration_predictor,
    video_encoding=video_tokens,
    audio_encoding=audio_tokens,
    frame_rate=frame_rate,
)

# `final_num_frames` is now ready for the generation pipeline

This pattern decouples user intent (automatic duration with safety bounds) from implementation details (head inference and grid snapping).

---

CLI Integration

Command-line interfaces can enable automatic duration prediction via arguments defined in [packages/ltx-pipelines/src/ltx_pipelines/utils/args.py](https://github.com/Lightricks/LTX-2/blob/main/packages/ltx-pipelines/src/ltx_pipelines/utils/args.py):

• --auto-duration — Enable DurationHead-based prediction
• --num-frames — Manual override (skips prediction when specified)

When --auto-duration is set without --num-frames, the pipeline automatically invokes resolve_num_frames() with default bounds.

---

Summary

• Load once: Use DurationPredictor.from_checkpoint() with LTX 2.5+ checkpoints containing DurationHead weights
• Encode captions: Extract frozen connector tokens via PromptEncoder.encode()
• Predict safely: Call the predictor with explicit min_seconds/max_seconds bounds for content-aware duration
• Integrate cleanly: Use AutoDuration and resolve_num_frames() for pipeline-compatible automatic frame count prediction

---

Frequently Asked Questions

What checkpoint versions include DurationHead weights?

DurationHead weights are available starting from LTX 2.5 / Gemma 4. Earlier checkpoints return None from DurationPredictor.from_checkpoint(). Always validate the predictor before attempting automatic duration prediction.

Can DurationHead predict duration without audio tokens?

Yes. The implementation in duration_head.py handles audio_encoding=None gracefully. Audio tokens contribute additional semantic signal when present, but video-only prediction remains functional.

Why does the predictor clamp to 8k + 1 frames?

The VAE temporal grid requirement in LTX-2 uses 8k + 1 frame alignment for proper latent space encoding. The seconds_to_clamped_num_frames() helper ensures generated videos decode correctly without temporal artifacts.

How accurate is DurationHead compared to manual frame selection?

DurationHead predicts natural shot duration based on caption semantics rather than arbitrary fixed lengths. Accuracy depends on training data distribution, but the clamping mechanism guarantees outputs remain within generation-safe bounds regardless of prediction quality.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}