# How to Implement Multi-Shot Attention Sink for Generating Long Videos with LongLive

> Learn how to implement multi-shot attention sink for long video generation with LongLive. Pin KV-cache regions to maintain attention across long sequences.

- Repository: [NVIDIA Research Projects/LongLive](https://github.com/NVlabs/LongLive)
- Tags: how-to-guide
- Published: 2026-05-24

---

**LongLive implements multi-shot attention sink by pinning KV-cache regions at scene cuts, combining a static global sink with a movable pinned block to maintain attention across arbitrarily long video sequences.**

The LongLive framework by NVIDIA Labs introduces a **multi-shot attention sink** mechanism that enables autoregressive diffusion models to generate arbitrarily long videos without quality degradation. Unlike traditional single-shot sinks that only preserve initial frames, this technique creates movable "anchor" regions across multiple shots while efficiently managing the KV-cache. This guide walks through the exact implementation using source code from the NVlabs/LongLive repository.

## Understanding the Multi-Shot Attention Sink Architecture

LongLive 2.0 replaces the single-shot sink with a three-component system that balances memory efficiency and long-range attention.

### The Three-Component System

The architecture combines distinct cache management strategies inside the causal attention module:

- **Global static sink**: A fixed block of initial frames (size defined by `sink_size`) that never rolls out of the cache.
- **Pinned local sink**: A movable anchor region that shifts forward when scene cuts occur, allowing the model to attend to recent keyframes across shots.
- **Dynamic rolling window**: All tokens outside the pinned regions shift normally during autoregressive generation.

This design lives primarily in [`wan_5b/modules/causal_model.py`](https://github.com/NVlabs/LongLive/blob/main/wan_5b/modules/causal_model.py), where the `_effective_sink` method calculates the combined size of static and pinned regions.

### Per-Shot RoPE Offset Management

Temporal consistency across shots requires careful handling of positional embeddings. The implementation uses a **per-shot RoPE offset** that shifts the temporal frequencies in `linear_rope_freqs` (located in [`utils/position_embedding_utils.py`](https://github.com/NVlabs/LongLive/blob/main/utils/position_embedding_utils.py)). This ensures that positional encodings remain aligned for the pinned anchor region even as the rolling window advances.

## Configuration Flags

Enable the multi-shot behavior through three YAML configuration parameters:

| Flag | Purpose |
|------|---------|
| `inference.multi_shot_sink` | Activates the pinned-region logic for scene-based anchoring. |
| `inference.multi_shot_rope_offset` | Enables temporal RoPE adjustments required for cross-shot attention. |
| `inference.sink_size` | Defines the frame count for the static global sink. |

### Configuration File Setup

Add these settings to your inference configuration:

```yaml

# configs/inference.yaml

inference:
  sink_size: 8                 # Static global sink size in frames

  multi_shot_sink: true        # Enable pinned region logic

  multi_shot_rope_offset: true # Enable per-shot RoPE offset

```

## Running End-to-End Inference

The `CausalDiffusionInferencePipeline` automatically handles sink management when the configuration flags are active.

### Complete Inference Example

```python
import torch
from omegaconf import OmegaConf
from pipeline import CausalDiffusionInferencePipeline
from utils.config import normalize_config
from utils.inference_utils import (
    load_generator_checkpoint,
    prepare_single_prompt_inputs,
    save_video,
)

# Load configuration with multi-shot flags enabled

cfg = normalize_config(OmegaConf.load("configs/inference.yaml"))
device = torch.device("cuda")

# Initialize pipeline

pipe = CausalDiffusionInferencePipeline(cfg, device=device)

# Load LongLive-2.0-5B checkpoint (BF16 or NVFP4 format)

load_generator_checkpoint(pipe.generator, "LongLive-2.0-5B/model_bf16.pt")
pipe = pipe.to(device=device, dtype=torch.bfloat16)
pipe.generator.model.eval().requires_grad_(False)

# Prepare inputs and generate

prompt = "A sunrise over a bustling futuristic city, time-lapse style."
noise, prompts = prepare_single_prompt_inputs(cfg, prompt, device)
video = pipe.inference(noise=noise, text_prompts=prompts)

# Save the long video with preserved anchor attention

save_video(video[0], "outputs/multi_shot_demo.mp4", fps=24)

```

During generation, the pipeline calls `_effective_sink` (lines 212-228 in [`causal_model.py`](https://github.com/NVlabs/LongLive/blob/main/causal_model.py)) to determine the cache boundary, then rolls only tokens after this boundary (lines 248-260).

## Advanced: Manual KV Cache Pinning

For custom shot boundaries or specific frame anchoring, manually trigger the pinning mechanism used during training.

### Pinning at Specific Frames

```python

# Access the KV cache from the inference pipeline

kv_cache = pipe.generator.model.kv_cache
frame_idx = 120  # Anchor after frame 120

# Pin the cache to create a multi-shot sink anchor

pipe.generator.model.pin_kv_on_scene_cut(kv_cache, frame_idx)

```

This invokes the same logic found in [`pipeline/self_forcing_training.py`](https://github.com/NVlabs/LongLive/blob/main/pipeline/self_forcing_training.py) (lines 390-398) that the self-forcing training pipeline uses when detecting automatic scene cuts.

## Core Implementation Details

The multi-shot sink relies on specific methods across the LongLive codebase:

**[`wan_5b/modules/causal_model.py`](https://github.com/NVlabs/LongLive/blob/main/wan_5b/modules/causal_model.py)**
- **`_effective_sink`**: Calculates `global_sink + pinned_len` to determine which tokens remain fixed during rolling (lines 212-228).
- **`forward`**: Uses the effective sink to decide KV-cache shift boundaries (lines 248-260).
- **`pin_kv_on_scene_cut`**: Creates the pinned anchor block when scene boundaries are detected.

**[`wan_5b/modules/causal_model_sp_ulysses.py`](https://github.com/NVlabs/LongLive/blob/main/wan_5b/modules/causal_model_sp_ulysses.py)**
- Sequence-parallel variant implementing identical sink logic for distributed inference.

**[`utils/position_embedding_utils.py`](https://github.com/NVlabs/LongLive/blob/main/utils/position_embedding_utils.py)**
- **`linear_rope_freqs`**: Applies per-shot offsets to RoPE temporal frequencies to maintain positional coherence across anchored shots (lines 10-66).

**[`pipeline/self_forcing_training.py`](https://github.com/NVlabs/LongLive/blob/main/pipeline/self_forcing_training.py)**
- Contains the automatic scene-cut detection and pinning logic used during training (lines 390-398).

**[`pipeline/causal_diffusion_inference.py`](https://github.com/NVlabs/LongLive/blob/main/pipeline/causal_diffusion_inference.py)**
- Respects the `multi_shot_sink` and `multi_shot_rope_offset` configuration flags during the inference loop.

## Summary

- **Multi-shot attention sink** extends LongLive's KV-cache management by adding movable pinned regions to the traditional static sink.
- **Three configuration flags** control the feature: `multi_shot_sink`, `multi_shot_rope_offset`, and `sink_size`.
- **Automatic pinning** occurs during inference via `CausalDiffusionInferencePipeline`, while manual control is available through `pin_kv_on_scene_cut`.
- **RoPE offset management** in [`position_embedding_utils.py`](https://github.com/NVlabs/LongLive/blob/main/position_embedding_utils.py) ensures temporal consistency across shot boundaries.
- **Core logic** resides in [`causal_model.py`](https://github.com/NVlabs/LongLive/blob/main/causal_model.py), specifically the `_effective_sink` calculation and the causal attention `forward` loop.

## Frequently Asked Questions

### What is the difference between single-shot and multi-shot attention sink?

A **single-shot sink** only preserves the first `sink_size` frames throughout the entire generation, limiting attention to initial content. The **multi-shot sink** creates additional pinned regions when scene cuts occur, allowing the model to anchor attention to the most recent keyframes of each new shot while discarding intermediate frames.

### How does the per-shot RoPE offset maintain temporal consistency?

The **RoPE offset** shifts the rotary positional embedding frequencies for tokens following a pinned region. According to the implementation in [`utils/position_embedding_utils.py`](https://github.com/NVlabs/LongLive/blob/main/utils/position_embedding_utils.py), this offset ensures that the temporal positional encodings of the pinned anchor frames remain consistent relative to new tokens, preventing attention degradation across shot boundaries.

### When should I manually pin the KV cache instead of using automatic scene detection?

Manual pinning via `pin_kv_on_scene_cut` is recommended when you have **pre-defined shot boundaries** in your generation script (e.g., specific frame counts where scene changes occur) or when the automatic detection in [`self_forcing_training.py`](https://github.com/NVlabs/LongLive/blob/main/self_forcing_training.py) fails to capture subtle visual transitions that you want to treat as shot boundaries.

### What sink size should I configure for optimal long video generation?

The default **8 frames** for `sink_size` provides a good balance between memory usage and attention stability for most scenarios. For videos requiring stronger consistency with opening scenes, increase to 16-24 frames; for memory-constrained environments, 4 frames may suffice, though this reduces the model's ability to reference initial content.