# DeepSeek-V3 RoPE Scaling Factor (`rope_factor=40`) and YaRN Extrapolation for 128K Context

> Discover how DeepSeek-V3 achieves 128K context with rope_factor=40 and YaRN extrapolation. Learn about frequency scaling and linear ramp correction for enhanced performance.

- Repository: [DeepSeek/DeepSeek-V3](https://github.com/deepseek-ai/DeepSeek-V3)
- Tags: deep-dive
- Published: 2026-02-26

---

**DeepSeek-V3 extends its 4K training context to 128K tokens by combining a RoPE scaling factor of 40 with YaRN extrapolation, implemented in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) through frequency scaling and linear ramp correction.**

DeepSeek-V3 uses **Rotary Positional Embeddings (RoPE)** to encode token positions, but its original training length was limited to 4,096 tokens (`original_seq_len = 4096`). To support **128K context windows** without retraining, the model employs a dual mechanism: a fixed scaling factor that stretches rotary frequencies and YaRN extrapolation that smooths the transition between trained and extended positions. These techniques are implemented in the `deepseek-ai/DeepSeek-V3` repository.

## Understanding RoPE Scaling in DeepSeek-V3

### The `rope_factor=40` Mechanism

The **RoPE scaling factor** (`rope_factor = 40`) uniformly stretches the sinusoidal period of rotary embeddings. In [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py), the base frequencies are computed from `rope_theta = 10000.0`, then divided by this factor when extrapolating beyond the original sequence length:

```python

# From inference/model.py lines 81-84

if seqlen > args.original_seq_len:
    factor = args.rope_factor  # 40

    # Frequencies are scaled by 1/factor for extrapolated positions

```

This scaling effectively multiplies the context capacity by 40, enabling theoretical support for up to 163,840 tokens (4096 × 40), though DeepSeek-V3 specifically targets **128K contexts** for practical deployment.

## YaRN Extrapolation for Extended Context

### Linear Ramp and Correction Range

While simple scaling stretches frequencies uniformly, **YaRN (Yet another RoPE extension method)** prevents attention instability by gradually adjusting frequencies beyond the original training length. The implementation in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) uses three helper functions inside `get_freqs_cis`:

- `find_correction_dim`: Calculates dimension-specific correction boundaries
- `find_correction_range`: Determines the low and high frequency bounds
- `linear_ramp_factor`: Creates a smooth interpolation weight between 0 and 1

When `seqlen > args.original_seq_len`, the code computes a correction range (`low`, `high`) based on the YaRN hyperparameters, then generates a linear ramp (`smooth`) that weights the scaled frequencies against the original frequencies:

```python

# Conceptual implementation from inference/model.py lines 14-63

smooth = linear_ramp_factor(low, high, dim_indices)

# Final frequency blend: scaled frequencies outside correction range,

# original frequencies inside

freqs = freqs / factor * (1 - smooth) + freqs * smooth

```

### YaRN Hyperparameters (`beta_fast` and `beta_slow`)

The YaRN configuration in DeepSeek-V3 uses specific rotation bounds to define the correction range:

- **`beta_fast = 32`**: Sets the upper bound for "fast" rotation dimensions
- **`beta_slow = 1`**: Sets the lower bound for "slow" rotation dimensions

These values are defined in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py) alongside the `rope_factor`:

```python

# From inference/model.py lines 84-85

beta_fast: int = 32
beta_slow: int = 1

```

The combination of `beta_fast=32` and `beta_slow=1` ensures that high-frequency dimensions (which capture fine positional details) are gradually adjusted while low-frequency dimensions (capturing coarse patterns) transition more aggressively, maintaining attention stability across the 128K context window.

## Attention Scaling with `mscale`

When extrapolating to 128K tokens, DeepSeek-V3 adjusts the attention **softmax scale** to compensate for the extended context. The `mscale` parameter derives from the same `rope_factor` and rescales the query-key dot products to prevent attention scores from becoming too diffuse:

```python

# From inference/model.py lines 34-38

mscale = 0.1 * args.mscale * math.log(args.rope_factor) + 1.0
self.softmax_scale = self.softmax_scale * mscale * mscale

```

Here, `self.softmax_scale` starts as `qk_head_dim ** -0.5` (the standard attention temperature). The additional `mscale` factor, calculated from the logarithm of `rope_factor=40`, ensures that attention weights remain sharp and well-calibrated even when attending across 128,000 positions.

## Implementation in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py)

The complete RoPE and YaRN implementation resides in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py). The `get_freqs_cis` function generates the rotary frequency tensors, while the `ModelArgs` dataclass holds the configuration:

```python

# Complete configuration example from inference/model.py

@dataclass
class ModelArgs:
    original_seq_len: int = 4096
    max_seq_len: int = 128000  # 128K context

    rope_theta: float = 10000.0
    rope_factor: int = 40      # RoPE scaling factor

    beta_fast: int = 32        # YaRN fast rotation bound

    beta_slow: int = 1         # YaRN slow rotation bound

    mscale: float = 1.0        # Attention scaling multiplier

def get_freqs_cis(args: ModelArgs, seqlen: int) -> torch.Tensor:
    # Computes rotary frequencies with YaRN extrapolation

    # Includes find_correction_dim, linear_ramp_factor, etc.

    pass

```

## Configuration Example

To enable 128K context generation, the model configuration JSON (such as [`inference/configs/config_236B.json`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/configs/config_236B.json)) specifies these parameters:

```json
{
  "original_seq_len": 4096,
  "max_seq_len": 128000,
  "rope_theta": 10000.0,
  "rope_factor": 40,
  "beta_fast": 32,
  "beta_slow": 1,
  "mscale": 1.0
}

```

When loading the model with these settings, `DeepSeekModel` automatically applies the RoPE scaling and YaRN extrapolation in the attention layers, allowing the model to process inputs up to 128,000 tokens without positional aliasing or attention degradation.

## Summary

- **RoPE scaling factor (`rope_factor=40`)** stretches the rotary embedding frequencies by a factor of 40, enabling support for context lengths up to 128K tokens from a 4K training base.
- **YaRN extrapolation** uses `beta_fast=32` and `beta_slow=1` to compute a smooth linear ramp between original and scaled frequencies, preventing sudden transitions that destabilize attention.
- **Attention scaling (`mscale`)** compensates for extended contexts by adjusting the softmax temperature based on the logarithm of `rope_factor`.
- **Implementation location**: All mechanisms reside in [`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py), specifically within `get_freqs_cis` and the `ModelArgs` configuration class.

## Frequently Asked Questions

### What does `rope_factor=40` do in DeepSeek-V3?

The `rope_factor=40` parameter scales the rotary positional embedding frequencies by a factor of 40, effectively stretching the sinusoidal period to accommodate sequences up to 128,000 tokens. This allows the model to generalize from its original 4,096-token training length to much longer contexts without requiring additional training on positional embeddings.

### How does YaRN differ from standard RoPE scaling?

While standard RoPE scaling uniformly stretches all frequency dimensions by a fixed factor, **YaRN (Yet another RoPE extension method)** applies a smooth, dimension-aware transition. It uses `beta_fast=32` and `beta_slow=1` to define a correction range where high-frequency dimensions gradually shift from original to scaled values via a linear ramp, preventing attention instability at the boundary between trained and extrapolated positions.

### Why is `mscale` necessary for 128K context windows?

The `mscale` parameter adjusts the attention softmax scaling factor to compensate for the statistical effects of longer sequences. As context length increases, dot-product attention scores become more diffuse; `mscale` (calculated as `0.1 * mscale * log(rope_factor) + 1.0`) rescales the query-key products to maintain sharp, stable attention distributions across 128,000 positions.

### Where are these RoPE and YaRN settings configured?

All RoPE scaling and YaRN parameters are defined in **[`inference/model.py`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/model.py)** within the `ModelArgs` dataclass (lines 81-85) and implemented in the `get_freqs_cis` function (lines 14-63). Pre-configured values for the 236B parameter model are also stored in **[`inference/configs/config_236B.json`](https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/configs/config_236B.json)**, which sets `rope_factor=40`, `beta_fast=32`, and `beta_slow=1` for 128K context generation.