How to Implement Multi-Shot Attention Sink for Generating Long Videos with LongLive

LongLive implements multi-shot attention sink by pinning KV-cache regions at scene cuts, combining a static global sink with a movable pinned block to maintain attention across arbitrarily long video sequences.

The LongLive framework by NVIDIA Labs introduces a multi-shot attention sink mechanism that enables autoregressive diffusion models to generate arbitrarily long videos without quality degradation. Unlike traditional single-shot sinks that only preserve initial frames, this technique creates movable "anchor" regions across multiple shots while efficiently managing the KV-cache. This guide walks through the exact implementation using source code from the NVlabs/LongLive repository.

Understanding the Multi-Shot Attention Sink Architecture

LongLive 2.0 replaces the single-shot sink with a three-component system that balances memory efficiency and long-range attention.

The Three-Component System

The architecture combines distinct cache management strategies inside the causal attention module:

• Global static sink: A fixed block of initial frames (size defined by sink_size) that never rolls out of the cache.
• Pinned local sink: A movable anchor region that shifts forward when scene cuts occur, allowing the model to attend to recent keyframes across shots.
• Dynamic rolling window: All tokens outside the pinned regions shift normally during autoregressive generation.

This design lives primarily in wan_5b/modules/causal_model.py, where the _effective_sink method calculates the combined size of static and pinned regions.

Per-Shot RoPE Offset Management

Temporal consistency across shots requires careful handling of positional embeddings. The implementation uses a per-shot RoPE offset that shifts the temporal frequencies in linear_rope_freqs (located in utils/position_embedding_utils.py). This ensures that positional encodings remain aligned for the pinned anchor region even as the rolling window advances.

Configuration Flags

Enable the multi-shot behavior through three YAML configuration parameters:

Flag	Purpose
inference.multi_shot_sink	Activates the pinned-region logic for scene-based anchoring.
inference.multi_shot_rope_offset	Enables temporal RoPE adjustments required for cross-shot attention.
inference.sink_size	Defines the frame count for the static global sink.

Configuration File Setup

Add these settings to your inference configuration:

# configs/inference.yaml

inference:
  sink_size: 8                 # Static global sink size in frames

  multi_shot_sink: true        # Enable pinned region logic

  multi_shot_rope_offset: true # Enable per-shot RoPE offset

Running End-to-End Inference

The CausalDiffusionInferencePipeline automatically handles sink management when the configuration flags are active.

Complete Inference Example

import torch
from omegaconf import OmegaConf
from pipeline import CausalDiffusionInferencePipeline
from utils.config import normalize_config
from utils.inference_utils import (
    load_generator_checkpoint,
    prepare_single_prompt_inputs,
    save_video,
)

# Load configuration with multi-shot flags enabled

cfg = normalize_config(OmegaConf.load("configs/inference.yaml"))
device = torch.device("cuda")

# Initialize pipeline

pipe = CausalDiffusionInferencePipeline(cfg, device=device)

# Load LongLive-2.0-5B checkpoint (BF16 or NVFP4 format)

load_generator_checkpoint(pipe.generator, "LongLive-2.0-5B/model_bf16.pt")
pipe = pipe.to(device=device, dtype=torch.bfloat16)
pipe.generator.model.eval().requires_grad_(False)

# Prepare inputs and generate

prompt = "A sunrise over a bustling futuristic city, time-lapse style."
noise, prompts = prepare_single_prompt_inputs(cfg, prompt, device)
video = pipe.inference(noise=noise, text_prompts=prompts)

# Save the long video with preserved anchor attention

save_video(video[0], "outputs/multi_shot_demo.mp4", fps=24)

During generation, the pipeline calls _effective_sink (lines 212-228 in causal_model.py) to determine the cache boundary, then rolls only tokens after this boundary (lines 248-260).

Advanced: Manual KV Cache Pinning

For custom shot boundaries or specific frame anchoring, manually trigger the pinning mechanism used during training.

Pinning at Specific Frames

# Access the KV cache from the inference pipeline

kv_cache = pipe.generator.model.kv_cache
frame_idx = 120  # Anchor after frame 120

# Pin the cache to create a multi-shot sink anchor

pipe.generator.model.pin_kv_on_scene_cut(kv_cache, frame_idx)

This invokes the same logic found in pipeline/self_forcing_training.py (lines 390-398) that the self-forcing training pipeline uses when detecting automatic scene cuts.

Core Implementation Details

The multi-shot sink relies on specific methods across the LongLive codebase:

wan_5b/modules/causal_model.py

• _effective_sink: Calculates global_sink + pinned_len to determine which tokens remain fixed during rolling (lines 212-228).
• forward: Uses the effective sink to decide KV-cache shift boundaries (lines 248-260).
• pin_kv_on_scene_cut: Creates the pinned anchor block when scene boundaries are detected.

wan_5b/modules/causal_model_sp_ulysses.py

• Sequence-parallel variant implementing identical sink logic for distributed inference.

utils/position_embedding_utils.py

• linear_rope_freqs: Applies per-shot offsets to RoPE temporal frequencies to maintain positional coherence across anchored shots (lines 10-66).

pipeline/self_forcing_training.py

• Contains the automatic scene-cut detection and pinning logic used during training (lines 390-398).

pipeline/causal_diffusion_inference.py

• Respects the multi_shot_sink and multi_shot_rope_offset configuration flags during the inference loop.

Summary

• Multi-shot attention sink extends LongLive's KV-cache management by adding movable pinned regions to the traditional static sink.
• Three configuration flags control the feature: multi_shot_sink, multi_shot_rope_offset, and sink_size.
• Automatic pinning occurs during inference via CausalDiffusionInferencePipeline, while manual control is available through pin_kv_on_scene_cut.
• RoPE offset management in position_embedding_utils.py ensures temporal consistency across shot boundaries.
• Core logic resides in causal_model.py, specifically the _effective_sink calculation and the causal attention forward loop.

Frequently Asked Questions

What is the difference between single-shot and multi-shot attention sink?

A single-shot sink only preserves the first sink_size frames throughout the entire generation, limiting attention to initial content. The multi-shot sink creates additional pinned regions when scene cuts occur, allowing the model to anchor attention to the most recent keyframes of each new shot while discarding intermediate frames.

How does the per-shot RoPE offset maintain temporal consistency?

The RoPE offset shifts the rotary positional embedding frequencies for tokens following a pinned region. According to the implementation in utils/position_embedding_utils.py, this offset ensures that the temporal positional encodings of the pinned anchor frames remain consistent relative to new tokens, preventing attention degradation across shot boundaries.

When should I manually pin the KV cache instead of using automatic scene detection?

Manual pinning via pin_kv_on_scene_cut is recommended when you have pre-defined shot boundaries in your generation script (e.g., specific frame counts where scene changes occur) or when the automatic detection in self_forcing_training.py fails to capture subtle visual transitions that you want to treat as shot boundaries.

What sink size should I configure for optimal long video generation?

The default 8 frames for sink_size provides a good balance between memory usage and attention stability for most scenarios. For videos requiring stronger consistency with opening scenes, increase to 16-24 frames; for memory-constrained environments, 4 frames may suffice, though this reduces the model's ability to reference initial content.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}