VRAM Optimization Techniques for Llama3 Inference: 5 Ways to Run 8B Models on 10GB GPUs

The crazyboym/llama3-chinese-chat repository reduces GPU memory usage by up to 75% using 4-bit quantization, FP16 precision, and explicit cache management, enabling Llama3-8B inference on consumer GPUs with as little as 4GB of VRAM.

The crazyboym/llama3-chinese-chat repository demonstrates practical VRAM optimization techniques for Llama3 inference that make large language models accessible on modest hardware. By implementing bitsandbytes quantization, half-precision tensors, and strategic memory clearing, the codebase allows a full 8-billion-parameter model to run comfortably on single GPUs with 10GB of memory—or even less when aggressive optimization flags are enabled.

Core VRAM Reduction Strategies

4-Bit Quantization with BitsAndBytes

The most aggressive memory saver uses the bitsandbytes library to store weights in 4-bit format while performing computation in float16. This cuts the model weight size to approximately one-quarter of standard FP16 storage. In the repository’s load_model implementation, this is configured via BitsAndBytesConfig with bnb_4bit_quant_type="nf4" and bnb_4bit_use_double_quant=True for additional memory savings.

You can see this configuration in deploy/web_streamlit_for_v1.py (lines 171–190), where the load_in_4bit flag triggers the quantization setup before model loading.

FP16 Half-Precision Mode

Forcing the model to use 16-bit floating point tensors (torch.float16) halves the memory required for activations and intermediate buffers compared to FP32. The codebase passes torch_dtype=torch.float16 to AutoModelForCausalLM.from_pretrained in every loading function, ensuring consistent half-precision arithmetic across inference pipelines.

Explicit CUDA Cache Clearing

Long-running inference sessions suffer from gradual VRAM fragmentation. The repository addresses this by calling torch.cuda.empty_cache() immediately after each generation step. This pattern appears in deploy/web_streamlit_for_v1.py at line 295, where the function releases temporary GPU buffers back to the allocator before processing the next user input.

LoRA Adapter-Only Loading

When using fine-tuned models, the code loads only the LoRA adapter weights via PeftModel.from_pretrained rather than duplicating the full base model. This technique, found in the load_model functions across multiple deployment scripts, keeps the large pretrained weights frozen while applying low-rank updates, significantly reducing the memory footprint for custom models.

Runtime Quantization Toggles

The implementation exposes a load_in_4bit boolean parameter in function signatures (such as def load_model(..., load_in_4bit=False)), allowing users to toggle 4-bit mode without modifying source code. This flag appears in CLI tools like tools/convert_gguf.py (lines 5–12) and the various Streamlit deployment interfaces.

Where These Techniques Live in the Codebase

The VRAM optimization patterns are consistently implemented across the following files:

• deploy/web_streamlit_for_v1.py – Contains the primary load_model implementation (lines 171–190) with quantization configuration, FP16 dtype settings, and explicit cache clearing (lines 248–295).
• deploy/web_streamlit_for_instruct.py – Replicates the same memory-saving logic for Instruct-tuned variants (lines 170–190, 256–303).
• deploy/web_streamlit_for_instruct_v2.py – Updates the UI with system-prompt support while preserving identical VRAM management (lines 170–190, 250–297).
• deploy/streamlit/web_llama3_chat.py – Minimal demo showing the load_model routine (lines 34–52) and cache management (lines 87–159).
• deploy/python/chat_demo.py – Pure-Python example demonstrating loading functions (lines 44–63) and cache clearing (lines 124–137).
• tools/convert_gguf.py – Shows the configurable load_in_4bit flag for offline conversion workflows (lines 5–12).

Complete Working Example

Below is a minimal implementation combining all five VRAM optimization techniques. This pattern mirrors the load_model functions found throughout the repository:

from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel
import torch

def load_optimized_llama3(
    model_path: str, 
    adapter_path: str | None = None, 
    load_in_4bit: bool = False
):
    # 1. Configure 4-bit quantization if requested

    quant_config = None
    if load_in_4bit:
        quant_config = BitsAndBytesConfig(
            load_in_4bit=True,
            bnb_4bit_compute_dtype=torch.float16,
            bnb_4bit_use_double_quant=True,
            bnb_4bit_quant_type="nf4",
        )
    
    # 2. Load base model with FP16 dtype and optional quantization

    model = AutoModelForCausalLM.from_pretrained(
        model_path,
        device_map="auto",
        torch_dtype=torch.float16,  # Force half-precision

        quantization_config=quant_config,
    )
    
    # 3. Load LoRA adapter only (optional)

    if adapter_path:
        model = PeftModel.from_pretrained(model, adapter_path)
    
    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
    return model, tokenizer

# Usage with maximum VRAM savings

model, tokenizer = load_optimized_llama3(
    model_path="shareAI/llama3-8b-instruct-dpo-zh",
    adapter_path=None,
    load_in_4bit=True,  # Enable 4-bit mode

)

# After generation, clear cache to prevent memory creep

outputs = model.generate(**inputs, max_new_tokens=256)
torch.cuda.empty_cache()

Summary

• 4-bit quantization via BitsAndBytesConfig reduces weight storage by 75% when load_in_4bit=True.
• FP16 precision (torch_dtype=torch.float16) cuts activation memory in half across all inference operations.
• Explicit cache clearing with torch.cuda.empty_cache() prevents gradual memory leaks during long sessions.
• LoRA adapter loading avoids duplicating base model weights when using fine-tuned checkpoints.
• Runtime flags allow dynamic switching between standard and quantized modes without code changes.

These techniques, as implemented in deploy/web_streamlit_for_v1.py and companion files, collectively enable high-quality Llama3 inference on consumer GPUs previously considered too small for 8B parameter models.

Frequently Asked Questions

How much VRAM is required to run Llama3-8B with these optimizations?

With load_in_4bit=True enabled, the model consumes approximately 4GB of VRAM for the 8B parameter variant, though exact usage varies by GPU architecture and sequence length. Without 4-bit quantization but with FP16 mode, expect around 10GB of VRAM, making these techniques viable for RTX 3080, RTX 2080 Ti, and equivalent consumer cards.

Does 4-bit quantization significantly reduce model quality?

According to the repository’s implementation using NF4 (Normal Float 4) quantization with double quantization, the quality degradation is minimal for Chinese chat applications. The bnb_4bit_compute_dtype=torch.float16 setting ensures computation occurs in higher precision while only storage uses 4-bit representation, preserving most of the model's reasoning capabilities.

Where should I call torch.cuda.empty_cache() in my inference pipeline?

The repository places this call immediately after model.generate() completes, as seen in deploy/python/chat_demo.py (lines 124–137). This placement ensures that temporary activation buffers allocated during the forward pass are released before the next user interaction or batch, preventing out-of-memory errors during long-running chat sessions.

Can I use these techniques with fine-tuned LoRA adapters?

Yes. The codebase specifically supports this via PeftModel.from_pretrained, which loads only the small adapter weights (typically megabytes) onto the base model (several gigabytes). This pattern appears in load_model functions across multiple deployment scripts, allowing you to switch between different fine-tuned versions without reloading the entire base model into VRAM.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}