# Extending Llama 3 Context Length to 32K or 96K: A Config-Based Approach

> Extend Llama 3 context length to 32K or 96K tokens by modifying the config json file for longer text processing and enhanced capabilities.

- Repository: [Xinlu Lai/llama3-chinese-chat](https://github.com/crazyboym/llama3-chinese-chat)
- Tags: how-to-guide
- Published: 2026-02-28

---

**You can stretch Llama 3’s default 8K context window to 32K or 96K tokens by editing `max_position_embeddings` and `rope_theta` in the model’s [`config.json`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/config.json) file, then reloading the model with a runtime that respects these limits.**

Llama 3 ships with a native 8K token limit, but many Chinese language applications require processing significantly longer documents. The `crazyboym/llama3-chinese-chat` repository documents a configuration-based "stretching" technique that expands the usable window without retraining or architectural changes. This guide walks through the exact edits to [`config.json`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/config.json) and the deployment steps needed to unlock extended context lengths.

## Understanding the Configuration Method

The extension relies on modifying two critical parameters in the model's configuration file. According to the README section "llama3 上下文长度简单扩张法（32K、96K）" in `crazyboym/llama3-chinese-chat` (lines 588-603), you adjust how the model interprets positional information rather than changing underlying transformer weights.

### Increasing max_position_embeddings

The `max_position_embeddings` field controls the maximum positional index the transformer can address. Set this value to `32768` for 32K context or `98304` for 96K context. This expansion treats the positional embeddings as a reusable lookup table, allowing the model to attend to longer sequences while preserving the original 8K training distribution for the first 8K tokens.

### Adjusting rope_theta for RoPE Scaling

RoPE (Rotary Positional Embedding) normally applies frequency scaling that degrades attention scores beyond the training window. Setting `rope_theta` to a very large number—such as `1000000` or `4000000`—effectively neutralizes this scaling factor. As noted in the repository documentation, this prevents the attention mechanism from losing coherence when attending to positions far beyond the original 8K limit.

## Editing the Model Configuration

Locate the [`config.json`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/config.json) file in your downloaded model directory. Update the following fields to match your desired context length:

```json
{
  "max_position_embeddings": 32768,
  "rope_theta": 1000000,
  "_name_or_path": "meta-llama/Meta-Llama-3-8B-Instruct"
}

```

For 96K context, use `98304` instead of `32768`. These changes are lossless for the original 8K window and require no additional fine-tuning.

## Loading the Extended Model in Python

After editing the configuration, instantiate the model using the Hugging Face Transformers library. The updated `max_position_embeddings` automatically applies during initialization:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "path/to/llama3-instruct-8b"
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="auto",
    trust_remote_code=True,
    torch_dtype="auto"
)

# Verify the extension worked

print(f"Context window: {model.config.max_position_embeddings} tokens")
print(f"RoPE theta: {model.config.rope_theta}")

```

The model now accepts inputs up to the new limit without throwing index errors.

## Deploying with vLLM

When serving the extended model with vLLM, you must align the runtime's allocation with your edited configuration. According to [`deploy/vLLM/README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/vLLM/README.md) (lines 60-67), pass the `--max-model-len` flag to ensure sufficient KV-cache memory allocation:

```bash
python -m vllm.entrypoints.openai.api_server \
  --model /path/to/llama3-instruct-8b \
  --served-model-name "llama3-32k" \
  --max-model-len 32768 \
  --api-key="YOUR_API_KEY"

```

Mismatch between the [`config.json`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/config.json) value and `--max-model-len` causes either silent truncation or out-of-memory errors during inference.

### Testing the Extended Endpoint

Verify the deployment by sending a request that exceeds the original 8K limit:

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="YOUR_API_KEY")
response = client.chat.completions.create(
    model="llama3-32k",
    messages=[{"role": "user", "content": "请写一篇5000字的长文..."}],
    max_tokens=4000
)
print(response.choices[0].message.content)

```

## Memory and Performance Considerations

Extending the context window increases GPU memory consumption linearly due to KV-cache growth. The 32K configuration typically fits on 24GB GPUs for moderate batch sizes, while 96K requires significantly more VRAM or aggressive batch size reduction. This method works specifically for Llama 3 Instruct variants (as confirmed in README.md lines 596-603). Note that Llama 3.1 models ship with native 128K support, making manual extension unnecessary for newer checkpoints.

## Summary

- **Edit [`config.json`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/config.json)**: Set `max_position_embeddings` to `32768` (32K) or `98304` (96K) and `rope_theta` to `1000000` or higher.
- **Align runtime parameters**: When using vLLM, set `--max-model-len` to match your configuration exactly.
- **Preserve weights**: This method reuses existing positional embeddings without fine-tuning or architectural changes.
- **Monitor memory**: Longer contexts require proportionally more KV-cache memory; 96K contexts need substantial GPU resources.

## Frequently Asked Questions

### Does extending the context window require retraining the model?

No. The method described in `crazyboym/llama3-chinese-chat` modifies only the configuration file. The underlying weights remain unchanged, and the model treats the extension as a lookup table expansion for positional indices beyond the original 8K limit.

### Why does increasing rope_theta disable RoPE scaling?

RoPE uses theta to determine the rotation frequency of positional encodings. Setting `rope_theta` to 1,000,000 or higher effectively flattens the frequency curve, preventing the dot-product attention scores from decaying beyond the original 8K training boundary. This allows the model to maintain coherence on 32K or 96K sequences without additional training data.

### Can I use this method with Llama 3.1 models?

No. Llama 3.1 already includes native 128K context support with proper long-context training, so manual extension is unnecessary and potentially suboptimal. This technique applies specifically to the original Llama 3 (8K) base and instruct variants.

### What happens if I set --max-model-len lower than my config.json value?

vLLM will truncate inputs to the `--max-model-len` value regardless of your [`config.json`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/config.json) setting. For full 32K or 96K utilization, these values must match exactly as documented in [`deploy/vLLM/README.md`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/vLLM/README.md) to prevent silent context window reduction.