DeepSeek-V3 RoPE Scaling Factor (`rope_factor=40`) and YaRN Extrapolation for 128K Context

DeepSeek-V3 extends its 4K training context to 128K tokens by combining a RoPE scaling factor of 40 with YaRN extrapolation, implemented in inference/model.py through frequency scaling and linear ramp correction.

DeepSeek-V3 uses Rotary Positional Embeddings (RoPE) to encode token positions, but its original training length was limited to 4,096 tokens (original_seq_len = 4096). To support 128K context windows without retraining, the model employs a dual mechanism: a fixed scaling factor that stretches rotary frequencies and YaRN extrapolation that smooths the transition between trained and extended positions. These techniques are implemented in the deepseek-ai/DeepSeek-V3 repository.

Understanding RoPE Scaling in DeepSeek-V3

The rope_factor=40 Mechanism

The RoPE scaling factor (rope_factor = 40) uniformly stretches the sinusoidal period of rotary embeddings. In inference/model.py, the base frequencies are computed from rope_theta = 10000.0, then divided by this factor when extrapolating beyond the original sequence length:

# From inference/model.py lines 81-84

if seqlen > args.original_seq_len:
    factor = args.rope_factor  # 40

    # Frequencies are scaled by 1/factor for extrapolated positions

This scaling effectively multiplies the context capacity by 40, enabling theoretical support for up to 163,840 tokens (4096 × 40), though DeepSeek-V3 specifically targets 128K contexts for practical deployment.

YaRN Extrapolation for Extended Context

Linear Ramp and Correction Range

While simple scaling stretches frequencies uniformly, YaRN (Yet another RoPE extension method) prevents attention instability by gradually adjusting frequencies beyond the original training length. The implementation in inference/model.py uses three helper functions inside get_freqs_cis:

• find_correction_dim: Calculates dimension-specific correction boundaries
• find_correction_range: Determines the low and high frequency bounds
• linear_ramp_factor: Creates a smooth interpolation weight between 0 and 1

When seqlen > args.original_seq_len, the code computes a correction range (low, high) based on the YaRN hyperparameters, then generates a linear ramp (smooth) that weights the scaled frequencies against the original frequencies:

# Conceptual implementation from inference/model.py lines 14-63

smooth = linear_ramp_factor(low, high, dim_indices)

# Final frequency blend: scaled frequencies outside correction range,

# original frequencies inside

freqs = freqs / factor * (1 - smooth) + freqs * smooth

YaRN Hyperparameters (beta_fast and beta_slow)

The YaRN configuration in DeepSeek-V3 uses specific rotation bounds to define the correction range:

• beta_fast = 32: Sets the upper bound for "fast" rotation dimensions
• beta_slow = 1: Sets the lower bound for "slow" rotation dimensions

These values are defined in inference/model.py alongside the rope_factor:

# From inference/model.py lines 84-85

beta_fast: int = 32
beta_slow: int = 1

The combination of beta_fast=32 and beta_slow=1 ensures that high-frequency dimensions (which capture fine positional details) are gradually adjusted while low-frequency dimensions (capturing coarse patterns) transition more aggressively, maintaining attention stability across the 128K context window.

Attention Scaling with mscale

When extrapolating to 128K tokens, DeepSeek-V3 adjusts the attention softmax scale to compensate for the extended context. The mscale parameter derives from the same rope_factor and rescales the query-key dot products to prevent attention scores from becoming too diffuse:

# From inference/model.py lines 34-38

mscale = 0.1 * args.mscale * math.log(args.rope_factor) + 1.0
self.softmax_scale = self.softmax_scale * mscale * mscale

Here, self.softmax_scale starts as qk_head_dim ** -0.5 (the standard attention temperature). The additional mscale factor, calculated from the logarithm of rope_factor=40, ensures that attention weights remain sharp and well-calibrated even when attending across 128,000 positions.

Implementation in inference/model.py

The complete RoPE and YaRN implementation resides in inference/model.py. The get_freqs_cis function generates the rotary frequency tensors, while the ModelArgs dataclass holds the configuration:

# Complete configuration example from inference/model.py

@dataclass
class ModelArgs:
    original_seq_len: int = 4096
    max_seq_len: int = 128000  # 128K context

    rope_theta: float = 10000.0
    rope_factor: int = 40      # RoPE scaling factor

    beta_fast: int = 32        # YaRN fast rotation bound

    beta_slow: int = 1         # YaRN slow rotation bound

    mscale: float = 1.0        # Attention scaling multiplier

def get_freqs_cis(args: ModelArgs, seqlen: int) -> torch.Tensor:
    # Computes rotary frequencies with YaRN extrapolation

    # Includes find_correction_dim, linear_ramp_factor, etc.

    pass

Configuration Example

To enable 128K context generation, the model configuration JSON (such as inference/configs/config_236B.json) specifies these parameters:

{
  "original_seq_len": 4096,
  "max_seq_len": 128000,
  "rope_theta": 10000.0,
  "rope_factor": 40,
  "beta_fast": 32,
  "beta_slow": 1,
  "mscale": 1.0
}

When loading the model with these settings, DeepSeekModel automatically applies the RoPE scaling and YaRN extrapolation in the attention layers, allowing the model to process inputs up to 128,000 tokens without positional aliasing or attention degradation.

Summary

• RoPE scaling factor (rope_factor=40) stretches the rotary embedding frequencies by a factor of 40, enabling support for context lengths up to 128K tokens from a 4K training base.
• YaRN extrapolation uses beta_fast=32 and beta_slow=1 to compute a smooth linear ramp between original and scaled frequencies, preventing sudden transitions that destabilize attention.
• Attention scaling (mscale) compensates for extended contexts by adjusting the softmax temperature based on the logarithm of rope_factor.
• Implementation location: All mechanisms reside in inference/model.py, specifically within get_freqs_cis and the ModelArgs configuration class.

Frequently Asked Questions

What does rope_factor=40 do in DeepSeek-V3?

The rope_factor=40 parameter scales the rotary positional embedding frequencies by a factor of 40, effectively stretching the sinusoidal period to accommodate sequences up to 128,000 tokens. This allows the model to generalize from its original 4,096-token training length to much longer contexts without requiring additional training on positional embeddings.

How does YaRN differ from standard RoPE scaling?

While standard RoPE scaling uniformly stretches all frequency dimensions by a fixed factor, YaRN (Yet another RoPE extension method) applies a smooth, dimension-aware transition. It uses beta_fast=32 and beta_slow=1 to define a correction range where high-frequency dimensions gradually shift from original to scaled values via a linear ramp, preventing attention instability at the boundary between trained and extrapolated positions.

Why is mscale necessary for 128K context windows?

The mscale parameter adjusts the attention softmax scaling factor to compensate for the statistical effects of longer sequences. As context length increases, dot-product attention scores become more diffuse; mscale (calculated as 0.1 * mscale * log(rope_factor) + 1.0) rescales the query-key products to maintain sharp, stable attention distributions across 128,000 positions.

Where are these RoPE and YaRN settings configured?

All RoPE scaling and YaRN parameters are defined in inference/model.py within the ModelArgs dataclass (lines 81-85) and implemented in the get_freqs_cis function (lines 14-63). Pre-configured values for the 236B parameter model are also stored in inference/configs/config_236B.json, which sets rope_factor=40, beta_fast=32, and beta_slow=1 for 128K context generation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}