How to Use Local LLM Models (LLaMA, ChatGLM) with GPT Academic

Configure LLM_MODEL in config.py to a local model key like "chatglm3" or "llama2", install the required dependencies, and launch main.py—GPT Academic will automatically spawn an isolated subprocess to load the model via transformers and stream responses through the Gradio interface.

GPT Academic, the open-source academic paper analysis toolkit from binary-husky/gpt_academic, supports running local large language models alongside cloud APIs. By leveraging a unified plugin architecture, you can run LLaMA, ChatGLM 3/4, and other transformer-based models directly on your own hardware while maintaining the same interactive chat experience.

Architecture of Local LLM Support in GPT Academic

The framework implements a subprocess isolation pattern to prevent the heavy PyTorch/Transformers runtime from blocking the main Gradio UI. When you select a local model, GPT Academic delegates inference to a dedicated child process that communicates via pipes.

The Subprocess Base Class

At the core of this system is LocalLLMHandle, defined in request_llms/local_llm_class.py. This abstract base class inherits from multiprocessing.Process and provides:

• Dual pipe channels for bidirectional communication between the main process and the model worker.
• Dependency validation via check_dependency, which verifies that required libraries (e.g., transformers, torch) are installed before attempting to load weights.
• Thread-safe streaming through stream_chat, which yields tokens as they are generated by the model.

The GetSingletonHandle utility (lines 31–44 in the same file) ensures that only one instance of a given local model class exists per Python process, preventing memory exhaustion from duplicate model loads.

Model-Specific Bridge Implementations

Concrete model support is provided by bridge modules that subclass LocalLLMHandle and implement three required methods:

1. load_model_info – Sets self.model_name and self.cmd_to_install (the pip command shown to users if dependencies are missing).
2. load_model_and_tokenizer – Loads the actual weights and tokenizer using transformers or model-specific libraries.
3. llm_stream_generator – Converts chat history into a model-specific prompt format and streams output using TextIteratorStreamer or the model’s native streaming API.

For example, request_llms/bridge_llama2.py implements the LLaMA 2/3 workflow using Hugging Face transformers, while request_llms/bridge_chatglm3.py targets ChatGLM 3’s stream_chat method and supports quantized variants (INT4, INT8).

Configuring Local LLM Models

To activate a local model, you must modify config.py before launching the application.

Setting the Active Model

Locate the LLM_MODEL variable and assign the key corresponding to your desired local bridge:

# config.py

LLM_MODEL = "chatglm3"  # Options include "chatglm4", "llama2", "llama3", etc.

Ensure the model key exists in AVAIL_LLM_MODELS. If you are adding a custom local model, append its identifier to this list:

AVAIL_LLM_MODELS = ["gpt-3.5-turbo", "chatglm3", "llama2", "jittorllms"]

ChatGLM-Specific Configuration

For ChatGLM 3/4, additional variables control hardware placement and quantization:

CHATGLM_LOCAL_MODEL_PATH = "/path/to/chatglm3-6b"  # Override default HuggingFace cache

LOCAL_MODEL_DEVICE = "cuda"                        # or "cpu"

LOCAL_MODEL_QUANT = "INT4"                         # "FP16", "INT8", or "INT4"

These settings are consumed by bridge_chatglm3.py during load_model_and_tokenizer to initialize the model with the appropriate trust_remote_code=True flag and quantization configuration.

Running Local Models: Step-by-Step

Once configured, you can interact with local models through the web interface or programmatically.

Launching via the Gradio UI

Start the application from the repository root:

python main.py

The console will display the local URL (typically http://localhost:12345). In the web interface:

1. Open the Model dropdown in the top-left corner.
2. Select your configured local model (e.g., ChatGLM3 (local)).
3. Type a message and submit.

The first request will trigger the subprocess initialization. You will see status messages such as:

ChatGLM3 尚未加载，加载需要一段时间。...
`依赖检测通过`
`尝试加载模型`
`准备就绪`

Subsequent requests reuse the loaded model via the singleton handle, providing immediate responses.

Programmatic Access Without UI

For batch processing or integration into other tools, import the bridge functions directly:

from request_llms.bridge_chatglm3 import predict_no_ui_long_connection

# Stream the response without Gradio

for response_chunk in predict_no_ui_long_connection(
    inputs="Summarize the theory of relativity.",
    llm_kwargs={"max_length": 512, "top_p": 0.9, "temperature": 0.7},
    history=[],
    sys_prompt="You are a physics professor.",
    observe_window=[None, None]
):
    print(response_chunk, end="", flush=True)

This interface is identical across all local bridges (bridge_llama2, bridge_chatglm4, etc.), allowing you to swap models by changing the import path while keeping the same calling convention.

Key Implementation Files

Understanding the source structure helps with debugging and customization.

File	Purpose
config.py	Central configuration for model selection (LLM_MODEL), device placement (LOCAL_MODEL_DEVICE), and quantization (LOCAL_MODEL_QUANT).
request_llms/local_llm_class.py	Abstract base LocalLLMHandle managing subprocess isolation, pipe communication, and dependency validation via check_dependency.
request_llms/bridge_llama2.py	Concrete implementation for LLaMA 2/3 using Hugging Face transformers and TextIteratorStreamer.
request_llms/bridge_chatglm3.py	Bridge for ChatGLM 3 with support for INT4/INT8 quantization and native stream_chat API.
request_llms/bridge_chatglm4.py	Updated bridge for ChatGLM 4 with similar architecture to version 3.
main.py	Application entry point that initializes the Gradio UI and routes requests to the selected bridge based on config.LLM_MODEL.

Summary

• Configure your target model by setting LLM_MODEL in config.py to a local bridge key like "chatglm3" or "llama2".
• Subprocess isolation via LocalLLMHandle in request_llms/local_llm_class.py keeps the UI responsive while heavy PyTorch models load in background workers.
• Hardware flexibility is provided through LOCAL_MODEL_DEVICE and LOCAL_MODEL_QUANT variables, supporting GPU acceleration and quantized inference for resource-constrained environments.
• Dual interfaces allow interaction through the standard Gradio web UI or programmatically via predict_no_ui_long_connection for batch processing pipelines.

Frequently Asked Questions

What hardware requirements are needed for running local LLMs with GPT Academic?

Local model execution requires sufficient RAM or VRAM to hold the model weights. For ChatGLM3-6B, you need at least 8 GB of GPU memory for FP16 inference, or 4-6 GB if using LOCAL_MODEL_QUANT = "INT4" or "INT8". LLaMA 2/3 models vary by size—7B parameter variants typically require 6-8 GB VRAM for 4-bit quantized loading. CPU inference is supported via LOCAL_MODEL_DEVICE = "cpu" but is significantly slower.

How do I switch between multiple local models in the same session?

To switch models, change the LLM_MODEL value in config.py and restart the application. The GetSingletonHandle utility in request_llms/local_llm_class.py ensures only one model instance exists per process, preventing memory conflicts. If you need to run multiple models simultaneously, launch separate GPT Academic instances on different ports using the --port argument, each with its own configuration.

Why does the first request to a local model take significantly longer than subsequent ones?

The delay occurs because LocalLLMHandle uses lazy loading—the model weights are not loaded into memory until the first request arrives. When you send the initial prompt, the subprocess executes load_model_and_tokenizer, which downloads weights (if not cached) and initializes the neural network on GPU/CPU. This initialization can take 10–60 seconds depending on model size and storage speed. Once loaded, the singleton handle keeps the model resident in memory, making subsequent requests instantaneous.

Can I use custom quantized models or fine-tuned variants with GPT Academic?

Yes, you can point to custom checkpoints by setting model-specific path variables in config.py. For ChatGLM, use CHATGLM_LOCAL_MODEL_PATH to specify the directory containing your fine-tuned or quantized weights. For LLaMA, modify the model path within request_llms/bridge_llama2.py or set the TRANSFORMERS_OFFLINE environment variable to use cached local snapshots. Ensure your quantized models are compatible with the expected loader—ChatGLM bridges support AutoGPTQ and native INT4/INT8 formats, while LLaMA bridges expect standard Hugging Face transformers checkpoints or GGUF variants loaded via appropriate adapters.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}