How to Handle Sequences Exceeding the RoPE Training Range in LingBot-Map

LingBot-Map handles sequences exceeding the RoPE training range by deriving frequency table sizes from the token count (positions.shape[1]) rather than the maximum position index, enabling dynamic caching that supports arbitrary sequence lengths without breaking CUDA-graph compatibility.

LingBot-Map implements a 2-D Rotary Position Embedding (RoPE) mechanism to encode spatial coordinates of image patches in vision-language workflows. When inference sequences grow longer than those seen during training, the system must generate valid sinusoidal frequencies for new positions without triggering costly host-device synchronizations or violating Torch-Compile constraints. The repository solves this through a dynamic caching strategy in lingbot_map/layers/rope.py that treats sequence length as a static property known at trace time.

Understanding the 2D RoPE Architecture

Spatial Position Encoding

The foundation of LingBot-Map's approach begins with the PositionGetter class, which generates coordinate tensors for image patches. Located in lingbot_map/layers/rope.py (lines 41‑53), this utility creates a tensor of shape (batch, n_tokens, 2) containing (y, x) coordinates for every patch in the grid. These coordinates feed into the RotaryPositionEmbedding2D module, which applies separate 1-D rotary embeddings to the vertical and horizontal halves of the token features.

The RotaryPositionEmbedding2D class splits input features into two halves—one for the y-axis and one for the x-axis—then applies sinusoidal position encoding to each dimension independently before concatenating the results back together (lines 91‑98).

Dynamic Frequency Caching for Arbitrary Sequence Lengths

Deriving Size from Token Count

The critical innovation for handling sequences exceeding the RoPE training range appears in the forward method at lines 87‑89. Instead of computing max_position = positions.max() + 1, which would require a host-device synchronization and break CUDA-graph capture, the implementation uses:

max_position = positions.shape[1]

This makes the frequency table size a static property determined by the tensor shape, known at compilation time. Because shape[1] represents the sequence length (number of tokens), the embedding automatically adapts to any input size presented during inference.

Cache Keying by Sequence Length

The _compute_frequency_components method (lines 102‑108) implements a caching mechanism keyed by the tuple (dim, seq_len, device, dtype). When a longer sequence arrives, the cache misses on the new seq_len, triggering computation of new cosine and sine tables sized exactly to the current token count. Subsequent forward passes with the same sequence length reuse the cached tensors, eliminating redundant computation.

This design allows the model to transparently handle any sequence length: the first forward pass with a longer input automatically populates a new cache entry, and all following passes reuse it without host overhead.

Implementing Long-Sequence Handling in Practice

Automatic Cache Generation During Forward Pass

The following example demonstrates processing a high-resolution image with 4096 tokens (64×64 patches), which likely exceeds the original training resolution:

import torch
from lingbot_map.layers.rope import RotaryPositionEmbedding2D, PositionGetter

# Create a dummy batch with larger token count than training

batch_size = 2
height, width = 64, 64          # 4096 tokens (larger than typical 16×16)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Generate spatial positions (y, x) for every patch

position_getter = PositionGetter()
positions = position_getter(batch_size, height, width, device)  # shape: (B, H*W, 2)

# Dummy token tensor: (B, n_heads, n_tokens, dim)

n_heads = 8
dim = 128                       # Must be divisible by 4 (vertical + horizontal halves)

tokens = torch.randn(batch_size, n_heads, height * width, dim, device=device)

# Instantiate the 2-D RoPE module

rope = RotaryPositionEmbedding2D(frequency=100.0, scaling_factor=1.0).to(device)

# Forward pass automatically caches frequency table for current token count

out = rope(tokens, positions)   # shape: (B, n_heads, H*W, dim)

print(out.shape)  # → torch.Size([2, 8, 4096, 128])

Pre-populating Frequency Tables for High-Resolution Inputs

If you want to avoid the one-time cost of building frequency tables during the first forward pass, you can manually prime the cache using the private _compute_frequency_components method. This is useful when you know you'll be processing very high-resolution inputs and want to pre-allocate GPU memory:

# Pre-compute for known maximum sequence length

max_seq_len = 8192
dummy_dim = 128
rope._compute_frequency_components(
    dummy_dim // 2, 
    max_seq_len, 
    device, 
    torch.float32
)

After this call, any forward pass with up to 8192 tokens will reuse the pre-computed tables, ensuring consistent latency across all inference batches.

Integration with the Attention Pipeline

The RotaryPositionEmbedding2D module integrates seamlessly with the multi-head attention system defined in lingbot_map/layers/attention.py. The attention layer calls rope.forward() after projecting input tokens into query and key vectors, applying the rotary embeddings to encode relative spatial positions before the scaled dot-product computation.

Because the RoPE module handles arbitrary sequence lengths internally, the attention mechanism in attention.py requires no modifications to support high-resolution inputs. The patch embedding layer in lingbot_map/layers/patch_embed.py simply produces more tokens for higher resolution images, and the RoPE cache expands automatically to accommodate them.

Summary

• Dynamic sizing: LingBot-Map determines frequency table dimensions from positions.shape[1] rather than computing the maximum position value, preserving CUDA-graph compatibility when handling sequences exceeding the RoPE training range.
• Automatic caching: Frequency components are cached by the tuple (dim, seq_len, device, dtype), allowing transparent support for longer sequences without code changes.
• Zero host overhead: The design avoids positions.max()+1 calculations that would force costly host-device synchronizations.
• Optional pre-population: Developers can call _compute_frequency_components ahead of time to eliminate first-pass latency for known large sequence lengths.

Frequently Asked Questions

What happens when a sequence exceeds the original RoPE training length in LingBot-Map?

The forward method automatically detects the new sequence length from the input tensor shape and generates appropriately sized frequency tables. The first pass with a longer sequence populates a new cache entry keyed by the new length, and subsequent passes reuse this cached data. This happens transparently without requiring model retraining or architecture changes.

Why does LingBot-Map use positions.shape[1] instead of positions.max()+1?

Using positions.max() would require transferring data from GPU to CPU to compute the maximum value, breaking CUDA-graph capture and Torch-Compile optimization. By deriving the size from the tensor shape (shape[1]), the value becomes a compile-time constant, allowing the graph compiler to treat the operation as static and maintain hardware acceleration performance.

Can I pre-compute RoPE frequencies for very high-resolution images?

Yes. You can call rope._compute_frequency_components(dim // 2, target_seq_len, device, dtype) before your first forward pass to pre-populate the cache. This is particularly useful for deployment scenarios where you know the maximum resolution ahead of time and want to eliminate the initial computation overhead when processing the first high-resolution batch.

Where is the RoPE implementation located in the repository?

The core implementation resides in lingbot_map/layers/rope.py, containing the RotaryPositionEmbedding2D and PositionGetter classes. The integration with the transformer architecture occurs in lingbot_map/layers/attention.py, which calls the RoPE module during the attention computation. Patch token generation, which determines the sequence length fed to RoPE, is handled in lingbot_map/layers/patch_embed.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}