# Understanding the Llama3 Chat Template Format: A Complete Guide

> Master the Llama3 chat template format with our guide. Learn how control tokens structure conversations for system, user, and assistant roles. Improve model understanding.

- Repository: [Xinlu Lai/llama3-chinese-chat](https://github.com/crazyboym/llama3-chinese-chat)
- Tags: deep-dive
- Published: 2026-02-28

---

**The Llama3 chat template format uses special control tokens like `<|begin_of_text|>`, `<|start_header_id|>`, and `<|eot_id|>` to structure conversations into system, user, and assistant roles, enabling the model to distinguish speakers and conversation boundaries.**

The `crazyboym/llama3-chinese-chat` repository implements this Llama3 chat template format for Chinese conversational AI, providing both manual prompt construction utilities and integration with Hugging Face's automated tokenizer methods. Understanding these token conventions is essential for customizing prompts and extending the model to new conversational scenarios.

## Core Template Architecture and Special Tokens

The template system defines how raw text transforms into structured model inputs through specific delimiter tokens that mark role boundaries and turn endings.

### Template Definition in chat_demo.py

The repository centralizes template configuration in **[`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py)** using the `register_template` function:

```python
register_template(
    template_name='llama3',
    system_format='<|begin_of_text|><<SYS>>\n{content}\n<</SYS>>\n\n',
    user_format='<|start_header_id|>user<|end_header_id|>\n\n{content}<|eot_id|>',
    assistant_format='<|start_header_id|>assistant<|end_header_id|>\n\n{content}<|end_of_text|>\n',
    system="You are a helpful, excellent and smart assistant.",
    stop_word='<|end_of_text|>'
)

```

This configuration creates a **ChatML-style** structure where:
- **`<|begin_of_text|>`** marks the conversation start
- **`<<SYS>>`** delimiters wrap system instructions (establishing persistent context)
- **`<|start_header_id|>` and `<|end_header_id|>`** bound role identifiers (user/assistant)
- **`<|eot_id|>`** (end-of-turn) terminates user inputs
- **`<|end_of_text|>`** terminates assistant responses and serves as the generation stop token

## Registering and Accessing Templates

The `register_template` function stores a **Template** dataclass instance in the global `template_dict` dictionary, keyed by the template name (`'llama3'`). This registry pattern allows consistent retrieval across deployment scripts:

```python
template = template_dict['llama3']

```

This centralized storage ensures that UI layers and inference pipelines use identical formatting rules without duplicating template logic.

## Building Prompts: Two Implementation Approaches

The repository supports dual methodologies for applying the Llama3 chat template format: a legacy manual builder for fine-grained control and a modern automated approach leveraging the tokenizer.

### Manual Construction with build_prompt

The `build_prompt` function (lines 88-115 in **[`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py)**) constructs token sequences through explicit string manipulation:

1. **System injection**: Applies `system_format` to the system prompt content
2. **History iteration**: Processes conversation history, applying `user_format` or `assistant_format` based on the role field
3. **Tokenization**: Encodes the concatenated string with `tokenizer.encode()`, disabling automatic special token addition to preserve template integrity

This approach returns a `torch.Tensor` of token IDs ready for the model's `generate` method.

### Automated Formatting with apply_chat_template

Modern UI implementations in **[`deploy/streamlit/web_llama3_chat.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/streamlit/web_llama3_chat.py)** leverage the Hugging Face Transformers built-in helper:

```python

# deploy/streamlit/web_llama3_chat.py (line 124)

formatted_chat = tokenizer.apply_chat_template(
    chat, 
    tokenize=False, 
    add_generation_prompt=True
)

```

Setting `add_generation_prompt=True` appends the assistant header (`<|start_header_id|>assistant<|end_header_id|>`) without a closing token, signaling the model to begin generating immediately. The `chat` parameter expects a list of dictionaries with `role` and `content` keys:

```python
chat = [
    {"role": "system", "content": st.session_state.system_prompt_content},
    *st.session_state.messages
]

```

## Stop Token Management and Generation Boundaries

The **stop_word** parameter (`<|end_of_text|>`, corresponding to token ID 128001 in the Llama 3 vocabulary) defines the generation boundary. The implementation extracts the stop token ID using:

```python
stop_token_id = tokenizer.encode(template.stop_word, add_special_tokens=True)

```

During generation, this ID is passed as the `eos_token_id` parameter, ensuring the model halts immediately when it produces the end-of-sequence marker rather than continuing indefinitely.

## Practical Implementation Examples

### Complete Manual Pipeline

```python
from deploy.python.chat_demo import template_dict, build_prompt, load_model, load_tokenizer
import copy
import torch

# Initialize model and tokenizer

model = load_model('shareAI/llama3-Chinese-chat-8b')
tokenizer = load_tokenizer('shareAI/llama3-Chinese-chat-8b')
template = template_dict['llama3']

# Prepare conversation history

history = [
    {"role": "user", "message": "请用中文解释一下量子纠缠。"},
    {"role": "assistant", "message": "量子纠缠是…"}
]

# Build prompt for next query

query = "它会怎样影响加密技术？"
input_ids = build_prompt(tokenizer, template, query, copy.deepcopy(history))
input_ids = input_ids.to(model.device)

# Generate with explicit stop token control

stop_id = tokenizer.encode(template.stop_word)[0]
output = model.generate(
    input_ids, 
    max_new_tokens=200, 
    eos_token_id=stop_id
)
response = tokenizer.decode(output[0][len(input_ids[0]):])

```

### Streamlit Integration with Tokenizer Helpers

```python
import streamlit as st
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "shareAI/llama3-Chinese-chat-8b"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name, 
    device_map="auto", 
    torch_dtype=torch.float16
)

# Collect user input

if user_input := st.chat_input("提问"):
    st.session_state.messages.append({"role": "user", "content": user_input})
    
    # Construct chat array with system prompt

    chat = [
        {"role": "system", "content": "You are a helpful Chinese assistant."},
        *st.session_state.messages
    ]
    
    # Apply template automatically

    formatted = tokenizer.apply_chat_template(
        chat, 
        tokenize=False, 
        add_generation_prompt=True
    )
    inputs = tokenizer(formatted, return_tensors="pt").to(model.device)
    
    # Generate response

    outputs = model.generate(**inputs, max_new_tokens=300, eos_token_id=128001)

```

## Summary

- The **Llama3 chat template format** relies on special tokens (`<|begin_of_text|>`, `<|start_header_id|>`, `<|eot_id|>`, `<|end_of_text|>`) to delineate system instructions, user turns, and assistant replies.
- Template definitions reside in **[`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py)**, registered via `register_template` and stored in the global `template_dict` for centralized access.
- The **legacy `build_prompt` method** (lines 88-115) offers manual control over prompt construction and tokenization, while **`apply_chat_template`** (as used in **[`deploy/streamlit/web_llama3_chat.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/streamlit/web_llama3_chat.py)**) provides streamlined, standardized formatting.
- **Stop tokens** (specifically `<|end_of_text|>`, token ID 128001) terminate generation and are encoded using `tokenizer.encode(template.stop_word)` to set the `eos_token_id` parameter correctly.

## Frequently Asked Questions

### What is the difference between build_prompt and apply_chat_template?

The `build_prompt` function manually concatenates formatted strings using the `Template` dataclass fields and tokenizes them separately, giving explicit control over special token handling, while `apply_chat_template` leverages the tokenizer's built-in chat template logic to handle formatting automatically according to the model's configuration file.

### How does the template handle system instructions?

The **system_format** wraps system instructions with `<|begin_of_text|>` and `<<SYS>>` delimiters, placing them at the absolute start of the conversation context to establish persistent behavioral guidelines that influence all subsequent assistant responses.

### Why is the stop_word parameter essential in the template?

The **stop_word** (`<|end_of_text|>`) serves as the definitive generation boundary; the UI extracts its token ID (128001) and passes it as `eos_token_id` to the model's `generate` method, ensuring the model halts immediately upon completing its response rather than hallucinating additional turns.

### Can I customize the chat template for specific use cases?

Yes, you can modify the parameters in `register_template` within **[`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py)** to adjust system prompts, change header formats, or implement few-shot prompting patterns by altering the `system_format`, `user_format`, and `assistant_format` strings while maintaining the core Llama3 token vocabulary for compatibility.