# Configuring 4-Bit and 8-Bit Quantization for Llama3 in the Chinese Chat Repository

> Discover how to configure 4-bit and 8-bit quantization for Llama3 in the llama3-chinese-chat repository. Deploy Llama3 on low VRAM GPUs with high-quality Chinese generation.

- Repository: [Xinlu Lai/llama3-chinese-chat](https://github.com/crazyboym/llama3-chinese-chat)
- Tags: how-to-guide
- Published: 2026-02-28

---

**The crazyboym/llama3-chinese-chat repository implements 4-bit quantization through a reusable `load_model` helper function that integrates `BitsAndBytesConfig`, enabling Llama3 deployment on GPUs with as little as 4GB VRAM while maintaining high-quality Chinese text generation.**

Configuring 4-bit and 8-bit quantization for Llama3 models is essential when deploying large language models on consumer hardware. The repository provides a centralized `load_model` utility across multiple deployment scripts that automatically handles BitsAndBytes configuration. This architecture allows you to switch between full precision (FP16), 4-bit, and 8-bit modes by passing a single boolean parameter to the model loader.

## How 4-Bit Quantization Is Implemented in the Repository

The quantization logic is encapsulated in a cached helper function defined in [`deploy/web_streamlit_for_v1.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/web_streamlit_for_v1.py) at lines 71-92. Identical implementations exist in [`deploy/streamlit/web_llama3_chat.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/streamlit/web_llama3_chat.py) (lines 34-55) and [`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py) (lines 44-55), ensuring consistent behavior across all deployment interfaces.

When `load_in_4bit=True` is passed to the helper, the function constructs a `BitsAndBytesConfig` with optimized settings for the Llama3 architecture:

```python
@st.cache_resource
def load_model(model_name_or_path,
               adapter_name_or_path=None,
               load_in_4bit=False):
    if load_in_4bit:
        quantization_config = BitsAndBytesConfig(
            load_in_4bit=True,
            bnb_4bit_compute_dtype=torch.float16,
            bnb_4bit_use_double_quant=True,
            bnb_4bit_quant_type="nf4",
            llm_int8_threshold=6.0,
            llm_int8_has_fp16_weight=False,
        )
    else:
        quantization_config = None

    model = AutoModelForCausalLM.from_pretrained(
        model_name_or_path,
        load_in_4bit=load_in_4bit,
        trust_remote_code=True,
        low_cpu_mem_usage=True,
        torch_dtype=torch.float16,
        device_map="auto",
        quantization_config=quantization_config,
    )
    # ... tokenizer loading and return statement

    return model, tokenizer

```

The `device_map="auto"` parameter automatically distributes model layers across available GPUs and CPU offloading buffers, while `low_cpu_mem_usage=True` prevents RAM spikes during the loading process.

### Key Configuration Parameters Explained

The **BitsAndBytes** configuration uses four critical parameters to balance memory efficiency and computational accuracy:

- **`bnb_4bit_compute_dtype=torch.float16`** – Specifies FP16 arithmetic for matrix multiplications during inference, providing a optimal trade-off between speed and precision.
- **`bnb_4bit_use_double_quant=True`** – Enables double quantization, which compresses the quantization constants themselves to further reduce memory footprint.
- **`bnb_4bit_quant_type="nf4"`** – Selects the Normal Float 4 (NF4) data type, which is theoretically optimal for normally distributed weights and provides superior quality to standard INT4 quantization.
- **`llm_int8_threshold=6.0`** – Retained for API compatibility with 8-bit quantization workflows; values above this threshold trigger FP16 computation for outlier features.

## Enabling 8-Bit Quantization Support

While the current implementation focuses on 4-bit mode, you can extend the `load_model` helper to support **8-bit quantization** by adding a conditional branch and the `load_in_8bit` parameter. This modification applies uniformly across all three deployment files.

```python
@st.cache_resource
def load_model(model_name_or_path,
               adapter_name_or_path=None,
               load_in_4bit=False,
               load_in_8bit=False):                     # ← new flag

    if load_in_4bit:
        quantization_config = BitsAndBytesConfig(
            load_in_4bit=True,
            bnb_4bit_compute_dtype=torch.float16,
            bnb_4bit_use_double_quant=True,
            bnb_4bit_quant_type="nf4",
            llm_int8_threshold=6.0,
            llm_int8_has_fp16_weight=False,
        )
    elif load_in_8bit:                                 # ← 8-bit branch

        quantization_config = BitsAndBytesConfig(
            load_in_8bit=True,
            bnb_8bit_compute_dtype=torch.float16,
        )
    else:
        quantization_config = None

    model = AutoModelForCausalLM.from_pretrained(
        model_name_or_path,
        load_in_4bit=load_in_4bit,
        load_in_8bit=load_in_8bit,                     # ← forward flag

        trust_remote_code=True,
        low_cpu_mem_usage=True,
        torch_dtype=torch.float16,
        device_map="auto",
        quantization_config=quantization_config,
    )
    # ... return model, tokenizer

```

The 8-bit configuration typically uses `bnb_8bit_compute_dtype=torch.float16` and can optionally include `bnb_8bit_use_double_quant=True` for additional memory savings at a modest computational cost.

## Selecting the Optimal Quantization Strategy

Choose your quantization precision based on available GPU memory and quality requirements:

- **4-bit (`load_in_4bit=True`)** – Reduces model size to approximately 25-50% of FP16 baseline. Ideal for GPUs with ≤ 4GB VRAM. Best for production deployment where memory is severely constrained.
- **8-bit (`load_in_8bit=True`)** – Compresses to roughly 50% of original size with minimal perceptible quality loss. Suitable for 8GB VRAM cards where generation quality remains a priority.
- **Full FP16 (default)** – No quantization overhead; requires approximately 14-16GB VRAM for the 8B parameter Llama3 model. Use when memory is abundant and maximum coherence is required.

## Practical Usage Examples

Launch the Streamlit interface with different precision modes using these command patterns:

```bash

# 4-bit quantization (reduces 8B model to ~4GB VRAM)

streamlit run deploy/web_streamlit_for_v1.py -- --model_path your-model --load_in_4bit

# 8-bit quantization (requires the patched load_model shown above)

streamlit run deploy/web_streamlit_for_v1.py -- --model_path your-model --load_in_8bit

# Full FP16 precision (highest quality, ~16GB VRAM required)

streamlit run deploy/web_streamlit_for_v1.py -- --model_path your-model

```

For the Python CLI demo in [`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py), pass the same flags directly to the script:

```bash
python deploy/python/chat_demo.py --model_path your-model --load_in_4bit

```

## Summary

- The **crazyboym/llama3-chinese-chat** repository centralizes quantization logic in the `load_model` helper function, implemented identically across [`deploy/web_streamlit_for_v1.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/web_streamlit_for_v1.py), [`deploy/streamlit/web_llama3_chat.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/streamlit/web_llama3_chat.py), and [`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py).
- **4-bit quantization** is fully implemented using `BitsAndBytesConfig` with NF4 data type and double quantization, activated by setting `load_in_4bit=True`.
- **8-bit quantization** requires minimal code modification to add a `load_in_8bit` parameter and corresponding configuration branch in the loader function.
- All quantization modes support `device_map="auto"` for automatic layer distribution across heterogeneous hardware configurations.
- Switching between precision modes requires only a flag change and application restart; no model conversion or preprocessing is necessary.

## Frequently Asked Questions

### What is the difference between 4-bit and 8-bit quantization in this repository?

**4-bit quantization** uses the NF4 (Normal Float 4) data type and reduces the model footprint to roughly one-quarter of FP16 size, making it viable for 4GB GPUs but with slightly higher perplexity. **8-bit quantization** maintains more representational precision, consuming approximately half the original memory while delivering generation quality nearly indistinguishable from full precision for Chinese dialogue tasks.

### How do I enable double quantization for additional memory savings?

Double quantization is enabled by default in the repository's 4-bit configuration via `bnb_4bit_use_double_quant=True`. This setting quantizes the quantization constants themselves, typically saving an additional 10-15% GPU memory without impacting inference speed. For 8-bit mode, you can add `bnb_8bit_use_double_quant=True` to the configuration if your version of BitsAndBytes supports it.

### Can I switch between quantization modes without restarting the application?

No, quantization precision is determined at model load time when `AutoModelForCausalLM.from_pretrained()` initializes the weights. To switch from 4-bit to 8-bit or FP16, you must restart the Streamlit or CLI process with the appropriate flag. The `@st.cache_resource` decorator in the Streamlit demos means you cannot reload the model with different quantization settings within the same session.

### Which file should I modify to add 8-bit support to all demos simultaneously?

Modify [`deploy/web_streamlit_for_v1.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/web_streamlit_for_v1.py) at lines 71-92 to add the `load_in_8bit` parameter and conditional logic. Since the repository uses duplicated code across files, you must apply the same changes to [`deploy/streamlit/web_llama3_chat.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/streamlit/web_llama3_chat.py) (lines 34-55) and [`deploy/python/chat_demo.py`](https://github.com/crazyboym/llama3-chinese-chat/blob/main/deploy/python/chat_demo.py) (lines 44-55) to ensure consistent 8-bit support across the web interface, alternate Streamlit UI, and CLI demo.