# How to Use Local LLM Models (LLaMA, ChatGLM) with GPT Academic

> Learn to use local LLM models like LLaMA and ChatGLM with GPT Academic. Configure config.py and launch main.py for seamless integration and Gradio interface access. Get started today!

- Repository: [binary-husky/gpt_academic](https://github.com/binary-husky/gpt_academic)
- Tags: how-to-guide
- Published: 2026-03-02

---

**Configure `LLM_MODEL` in [`config.py`](https://github.com/binary-husky/gpt_academic/blob/main/config.py) to a local model key like `"chatglm3"` or `"llama2"`, install the required dependencies, and launch [`main.py`](https://github.com/binary-husky/gpt_academic/blob/main/main.py)—GPT Academic will automatically spawn an isolated subprocess to load the model via `transformers` and stream responses through the Gradio interface.**

GPT Academic, the open-source academic paper analysis toolkit from `binary-husky/gpt_academic`, supports running **local** large language models alongside cloud APIs. By leveraging a unified plugin architecture, you can run LLaMA, ChatGLM 3/4, and other transformer-based models directly on your own hardware while maintaining the same interactive chat experience.

## Architecture of Local LLM Support in GPT Academic

The framework implements a **subprocess isolation pattern** to prevent the heavy PyTorch/Transformers runtime from blocking the main Gradio UI. When you select a local model, GPT Academic delegates inference to a dedicated child process that communicates via pipes.

### The Subprocess Base Class

At the core of this system is `LocalLLMHandle`, defined in [`request_llms/local_llm_class.py`](https://github.com/binary-husky/gpt_academic/blob/main/request_llms/local_llm_class.py). This abstract base class inherits from `multiprocessing.Process` and provides:

- **Dual pipe channels** for bidirectional communication between the main process and the model worker.
- **Dependency validation** via `check_dependency`, which verifies that required libraries (e.g., `transformers`, `torch`) are installed before attempting to load weights.
- **Thread-safe streaming** through `stream_chat`, which yields tokens as they are generated by the model.

The `GetSingletonHandle` utility (lines 31–44 in the same file) ensures that only **one** instance of a given local model class exists per Python process, preventing memory exhaustion from duplicate model loads.

### Model-Specific Bridge Implementations

Concrete model support is provided by bridge modules that subclass `LocalLLMHandle` and implement three required methods:

1. **`load_model_info`** – Sets `self.model_name` and `self.cmd_to_install` (the pip command shown to users if dependencies are missing).
2. **`load_model_and_tokenizer`** – Loads the actual weights and tokenizer using `transformers` or model-specific libraries.
3. **`llm_stream_generator`** – Converts chat history into a model-specific prompt format and streams output using `TextIteratorStreamer` or the model’s native streaming API.

For example, [`request_llms/bridge_llama2.py`](https://github.com/binary-husky/gpt_academic/blob/main/request_llms/bridge_llama2.py) implements the LLaMA 2/3 workflow using Hugging Face `transformers`, while [`request_llms/bridge_chatglm3.py`](https://github.com/binary-husky/gpt_academic/blob/main/request_llms/bridge_chatglm3.py) targets ChatGLM 3’s `stream_chat` method and supports quantized variants (`INT4`, `INT8`).

## Configuring Local LLM Models

To activate a local model, you must modify [`config.py`](https://github.com/binary-husky/gpt_academic/blob/main/config.py) before launching the application.

### Setting the Active Model

Locate the `LLM_MODEL` variable and assign the key corresponding to your desired local bridge:

```python

# config.py

LLM_MODEL = "chatglm3"  # Options include "chatglm4", "llama2", "llama3", etc.

```

Ensure the model key exists in `AVAIL_LLM_MODELS`. If you are adding a custom local model, append its identifier to this list:

```python
AVAIL_LLM_MODELS = ["gpt-3.5-turbo", "chatglm3", "llama2", "jittorllms"]

```

### ChatGLM-Specific Configuration

For ChatGLM 3/4, additional variables control hardware placement and quantization:

```python
CHATGLM_LOCAL_MODEL_PATH = "/path/to/chatglm3-6b"  # Override default HuggingFace cache

LOCAL_MODEL_DEVICE = "cuda"                        # or "cpu"

LOCAL_MODEL_QUANT = "INT4"                         # "FP16", "INT8", or "INT4"

```

These settings are consumed by [`bridge_chatglm3.py`](https://github.com/binary-husky/gpt_academic/blob/main/bridge_chatglm3.py) during `load_model_and_tokenizer` to initialize the model with the appropriate `trust_remote_code=True` flag and quantization configuration.

## Running Local Models: Step-by-Step

Once configured, you can interact with local models through the web interface or programmatically.

### Launching via the Gradio UI

Start the application from the repository root:

```bash
python main.py

```

The console will display the local URL (typically `http://localhost:12345`). In the web interface:

1. Open the **Model** dropdown in the top-left corner.
2. Select your configured local model (e.g., **ChatGLM3 (local)**).
3. Type a message and submit.

The first request will trigger the subprocess initialization. You will see status messages such as:

```

ChatGLM3 尚未加载，加载需要一段时间。...
`依赖检测通过`
`尝试加载模型`
`准备就绪`

```

Subsequent requests reuse the loaded model via the singleton handle, providing immediate responses.

### Programmatic Access Without UI

For batch processing or integration into other tools, import the bridge functions directly:

```python
from request_llms.bridge_chatglm3 import predict_no_ui_long_connection

# Stream the response without Gradio

for response_chunk in predict_no_ui_long_connection(
    inputs="Summarize the theory of relativity.",
    llm_kwargs={"max_length": 512, "top_p": 0.9, "temperature": 0.7},
    history=[],
    sys_prompt="You are a physics professor.",
    observe_window=[None, None]
):
    print(response_chunk, end="", flush=True)

```

This interface is identical across all local bridges (`bridge_llama2`, `bridge_chatglm4`, etc.), allowing you to swap models by changing the import path while keeping the same calling convention.

## Key Implementation Files

Understanding the source structure helps with debugging and customization.

| File | Purpose |
|------|---------|
| **[`config.py`](https://github.com/binary-husky/gpt_academic/blob/main/config.py)** | Central configuration for model selection (`LLM_MODEL`), device placement (`LOCAL_MODEL_DEVICE`), and quantization (`LOCAL_MODEL_QUANT`). |
| **[`request_llms/local_llm_class.py`](https://github.com/binary-husky/gpt_academic/blob/main/request_llms/local_llm_class.py)** | Abstract base `LocalLLMHandle` managing subprocess isolation, pipe communication, and dependency validation via `check_dependency`. |
| **[`request_llms/bridge_llama2.py`](https://github.com/binary-husky/gpt_academic/blob/main/request_llms/bridge_llama2.py)** | Concrete implementation for LLaMA 2/3 using Hugging Face `transformers` and `TextIteratorStreamer`. |
| **[`request_llms/bridge_chatglm3.py`](https://github.com/binary-husky/gpt_academic/blob/main/request_llms/bridge_chatglm3.py)** | Bridge for ChatGLM 3 with support for `INT4`/`INT8` quantization and native `stream_chat` API. |
| **[`request_llms/bridge_chatglm4.py`](https://github.com/binary-husky/gpt_academic/blob/main/request_llms/bridge_chatglm4.py)** | Updated bridge for ChatGLM 4 with similar architecture to version 3. |
| **[`main.py`](https://github.com/binary-husky/gpt_academic/blob/main/main.py)** | Application entry point that initializes the Gradio UI and routes requests to the selected bridge based on `config.LLM_MODEL`. |

## Summary

- **Configure** your target model by setting `LLM_MODEL` in [`config.py`](https://github.com/binary-husky/gpt_academic/blob/main/config.py) to a local bridge key like `"chatglm3"` or `"llama2"`.
- **Subprocess isolation** via `LocalLLMHandle` in [`request_llms/local_llm_class.py`](https://github.com/binary-husky/gpt_academic/blob/main/request_llms/local_llm_class.py) keeps the UI responsive while heavy PyTorch models load in background workers.
- **Hardware flexibility** is provided through `LOCAL_MODEL_DEVICE` and `LOCAL_MODEL_QUANT` variables, supporting GPU acceleration and quantized inference for resource-constrained environments.
- **Dual interfaces** allow interaction through the standard Gradio web UI or programmatically via `predict_no_ui_long_connection` for batch processing pipelines.

## Frequently Asked Questions

### What hardware requirements are needed for running local LLMs with GPT Academic?

Local model execution requires sufficient RAM or VRAM to hold the model weights. For **ChatGLM3-6B**, you need at least 8 GB of GPU memory for FP16 inference, or 4-6 GB if using `LOCAL_MODEL_QUANT = "INT4"` or `"INT8"`. **LLaMA 2/3** models vary by size—7B parameter variants typically require 6-8 GB VRAM for 4-bit quantized loading. CPU inference is supported via `LOCAL_MODEL_DEVICE = "cpu"` but is significantly slower.

### How do I switch between multiple local models in the same session?

To switch models, change the `LLM_MODEL` value in [`config.py`](https://github.com/binary-husky/gpt_academic/blob/main/config.py) and restart the application. The `GetSingletonHandle` utility in [`request_llms/local_llm_class.py`](https://github.com/binary-husky/gpt_academic/blob/main/request_llms/local_llm_class.py) ensures only one model instance exists per process, preventing memory conflicts. If you need to run multiple models simultaneously, launch separate GPT Academic instances on different ports using the `--port` argument, each with its own configuration.

### Why does the first request to a local model take significantly longer than subsequent ones?

The delay occurs because `LocalLLMHandle` uses **lazy loading**—the model weights are not loaded into memory until the first request arrives. When you send the initial prompt, the subprocess executes `load_model_and_tokenizer`, which downloads weights (if not cached) and initializes the neural network on GPU/CPU. This initialization can take 10–60 seconds depending on model size and storage speed. Once loaded, the singleton handle keeps the model resident in memory, making subsequent requests instantaneous.

### Can I use custom quantized models or fine-tuned variants with GPT Academic?

Yes, you can point to custom checkpoints by setting model-specific path variables in [`config.py`](https://github.com/binary-husky/gpt_academic/blob/main/config.py). For **ChatGLM**, use `CHATGLM_LOCAL_MODEL_PATH` to specify the directory containing your fine-tuned or quantized weights. For **LLaMA**, modify the model path within [`request_llms/bridge_llama2.py`](https://github.com/binary-husky/gpt_academic/blob/main/request_llms/bridge_llama2.py) or set the `TRANSFORMERS_OFFLINE` environment variable to use cached local snapshots. Ensure your quantized models are compatible with the expected loader—ChatGLM bridges support AutoGPTQ and native INT4/INT8 formats, while LLaMA bridges expect standard Hugging Face `transformers` checkpoints or GGUF variants loaded via appropriate adapters.