BERT vs GPT-2 vs XGBoost vs LSTM: Comparing Sentiment Analysis Models in Bettafish

The four sentiment analysis models in the 666ghj/bettafish repository represent distinct machine learning paradigms: BERT uses a frozen encoder with a trainable classifier, GPT-2 applies LoRA adapters to a decoder-only transformer, XGBoost relies on gradient-boosted trees with bag-of-words features, and LSTM implements a bidirectional recurrent network with custom Word2Vec embeddings.

The 666ghj/bettafish repository provides a comprehensive comparison of modern and classical approaches to Chinese sentiment analysis. Each sentiment analysis model targets different resource constraints and accuracy requirements, from lightweight tree-based methods to large transformer architectures. Understanding these architectural differences helps developers select the optimal approach for their specific deployment environment.

Architectural Overview of the Four Sentiment Analysis Models

BERT: Encoder-Only Transformer

The BERT implementation leverages the bert-base-chinese pre-trained weights through the BertModel_Custom class defined in SentimentAnalysisModel/WeiboSentiment_MachineLearning/bert_train.py. This encoder-only transformer processes input text through 12 layers of bidirectional self-attention, producing 768-dimensional contextualized embeddings. The architecture freezes all BERT parameters during training, optimizing only a single linear classification head that maps the [CLS] token representation to sentiment probabilities.

GPT-2 with LoRA: Decoder-Only Transformer

The GPT-2 implementation utilizes a decoder-only architecture based on gpt2-chinese-cluecorpussmall, enhanced with Parameter-Efficient Fine-Tuning (PEFT) using LoRA adapters. Located in SentimentAnalysisModel/WeiboSentiment_Finetuned/GPT2-Lora/train.py, this approach wraps GPT2ForSequenceClassification with LoraConfig to inject trainable low-rank matrices into the attention projection layers. This design preserves the pre-trained generative knowledge while reducing trainable parameters by over 90% compared to full fine-tuning.

XGBoost: Gradient-Boosted Decision Trees

The XGBoost implementation represents the classical machine learning approach, implemented in SentimentAnalysisModel/WeiboSentiment_MachineLearning/xgboost_train.py through the XGBoostModel class. This method constructs a sparse bag-of-words representation using CountVectorizer with a default vocabulary size of 2000 features. The algorithm trains an ensemble of decision trees using gradient boosting, optimizing a logistic objective for binary sentiment classification without requiring any neural network dependencies or GPU resources.

LSTM: Bidirectional Recurrent Network

The LSTM implementation provides a deep learning approach without transformer architecture, defined in SentimentAnalysisModel/WeiboSentiment_MachineLearning/lstm_train.py via the LSTMModel class. This bidirectional LSTM network processes sequences of Word2Vec embeddings (default 64-dimensional vectors) through two recurrent layers, capturing temporal dependencies in both forward and backward directions. The architecture pre-trains embeddings on the corpus using Word2Vec before freezing them during the classification phase.

Key Technical Differences

Feature Representation Strategies

Each sentiment analysis model employs distinct feature extraction methodologies. BERT utilizes contextualized subword embeddings from 12 transformer layers, capturing complex linguistic relationships through self-attention mechanisms. GPT-2 leverages causal language modeling representations from its decoder layers, modified through LoRA adapters to adapt to classification tasks. XGBoost relies on high-dimensional sparse vectors from CountVectorizer, treating text as unordered term frequencies. LSTM uses dense Word2Vec embeddings trained specifically on the Weibo sentiment corpus, preserving semantic relationships in continuous vector space.

Training Regimes and Resource Requirements

The training methodologies vary significantly across implementations. The BERT model in bert_train.py freezes the transformer backbone (self.bert.eval() at line 69), training only the linear classifier head, which enables fast convergence on modest GPU memory. The GPT-2 implementation applies full fine-tuning exclusively to LoRA adapter parameters while maintaining the base model in evaluation mode, utilizing AdamW with linear learning rate scheduling and gradient clipping to prevent explosion. XGBoost trains entirely on CPU using multithreading support (nthread parameter), requiring no gradient descent or backpropagation. The LSTM model trains end-to-end through standard backpropagation through time, with Word2Vec vectors pre-trained and subsequently frozen during the classification training phase.

Inference Speed and Deployment Characteristics

Runtime performance characteristics differ based on architectural complexity. BERT inference requires a full forward pass through the 12-layer encoder plus the classification head, though the frozen backbone allows for potential optimization through TorchScript or ONNX conversion. GPT-2 necessitates processing through the decoder transformer architecture; while LoRA adapters minimize trainable parameters, the full model must still execute during inference, making it slightly heavier than BERT. XGBoost delivers the fastest CPU inference, traversing the trained tree ensemble with logarithmic complexity relative to tree depth, making it ideal for high-throughput, resource-constrained environments. LSTM processes sequences token-by-token through recurrent connections, creating inherent sequential dependency that prevents parallelization across the time dimension, resulting in moderate speed that scales linearly with sequence length.

Code Examples: Loading and Running Each Model

BERT Implementation

from SentimentAnalysisModel.WeiboSentiment_MachineLearning.bert_train import BertModel_Custom
from SentimentAnalysisModel.WeiboSentiment_MachineLearning.base_model import BaseModel

# Load data (train_path / test_path are examples)

train_data, test_data = BaseModel.load_data('./data/weibo2018/train.txt',
                                           './data/weibo2018/test.txt')

# Initialise and train

model = BertModel_Custom('./model/chinese_wwm_pytorch')
model.train(train_data, num_epochs=5, batch_size=64, learning_rate=2e-3)

# Predict a list of sentences

texts = ["今天天气真好，心情很棒", "这部电影太无聊了，浪费时间"]
preds = model.predict(texts)
print(preds)   # [1, 0] → 1: 正面, 0: 负面

GPT-2 with LoRA Implementation

from transformers import GPT2ForSequenceClassification, BertTokenizer
from peft import LoraConfig, get_peft_model
import torch

# Load tokenizer and model (already fine‑tuned in "./best_weibo_sentiment_lora")

tokenizer = BertTokenizer.from_pretrained('./models/gpt2-chinese')
model = GPT2ForSequenceClassification.from_pretrained('./best_weibo_sentiment_lora')
model.eval()

def predict(text):
    enc = tokenizer(text, truncation=True, padding='max_length',
                    max_length=128, return_tensors='pt')
    with torch.no_grad():
        logits = model(**{k: v.to(model.device) for k, v in enc.items()}).logits
    prob = torch.softmax(logits, dim=-1)[0, 1].item()   # probability of positive class

    label = int(prob > 0.5)
    return label, prob

print(predict("我很喜欢这部电影"))   # (1, 0.87)

XGBoost Implementation

from SentimentAnalysisModel.WeiboSentiment_MachineLearning.xgboost_train import XGBoostModel
from SentimentAnalysisModel.WeiboSentiment_MachineLearning.base_model import BaseModel

train_data, test_data = BaseModel.load_data('./data/weibo2018/train.txt',
                                           './data/weibo2018/test.txt')

model = XGBoostModel()
model.train(train_data, max_features=2000, max_depth=6, eta=0.3,
            num_boost_round=200)

# Single prediction

label, confidence = model.predict_single("这个产品真的很垃圾")
print(label, confidence)   # 0 0.92  (negative)

LSTM Implementation

from SentimentAnalysisModel.WeiboSentiment_MachineLearning.lstm_train import LSTMModel
from SentimentAnalysisModel.WeiboSentiment_MachineLearning.base_model import BaseModel

train_data, test_data = BaseModel.load_data('./data/weibo2018/train.txt',
                                           './data/weibo2018/test.txt')

model = LSTMModel()
model.train(train_data, embed_size=64, hidden_size=64,
            num_layers=2, learning_rate=5e-4, num_epochs=5)

label, confidence = model.predict_single("这家餐厅的服务太差了")
print(label, confidence)   # 0 0.81  (negative)

File Structure and Implementation Details

The 666ghj/bettafish repository organizes these sentiment analysis models into distinct modules, each implementing a unified interface through the BaseModel abstract class.

• SentimentAnalysisModel/WeiboSentiment_MachineLearning/bert_train.py – Contains the BertModel_Custom class with auto-download logic for bert-base-chinese weights and the frozen backbone training strategy.

• SentimentAnalysisModel/WeiboSentiment_Finetuned/GPT2-Lora/train.py – Implements the GPT2ForSequenceClassification wrapper with LoraConfig for parameter-efficient fine-tuning, including gradient clipping and linear learning rate scheduling.

• SentimentAnalysisModel/WeiboSentiment_MachineLearning/xgboost_train.py – Houses the XGBoostModel class utilizing CountVectorizer for sparse feature extraction and XGBoost's gradient boosting with configurable tree depth and learning rate.

• SentimentAnalysisModel/WeiboSentiment_MachineLearning/lstm_train.py – Defines the LSTMModel class with custom Word2Vec embedding training (lines 90-107) and a bidirectional LSTM architecture with configurable hidden sizes and layer counts.

• SentimentAnalysisModel/WeiboSentiment_MachineLearning/base_model.py – Provides the abstract BaseModel class standardizing load_data, evaluate, and prediction interfaces across all implementations.

Summary

• BERT delivers high accuracy through contextualized transformer embeddings while maintaining reasonable GPU requirements by freezing the backbone and training only a linear classifier head.

• GPT-2 with LoRA leverages generative pre-training through decoder-only architecture while minimizing fine-tuning costs via low-rank adapters, making large transformer technology accessible with limited compute.

• XGBoost provides the fastest CPU inference through gradient-boosted decision trees on sparse bag-of-words features, requiring no deep learning dependencies or GPU resources.

• LSTM offers a middle ground with custom Word2Vec embeddings and recurrent processing, suitable for educational purposes or environments where transformer models are impractical but neural approaches are preferred.

Frequently Asked Questions

Which sentiment analysis model provides the best accuracy for Chinese text?

The BERT implementation generally provides the highest accuracy due to its deep bidirectional contextualization through 12 transformer layers and pre-training on extensive Chinese corpora. However, the GPT-2 with LoRA approach can achieve comparable performance when properly fine-tuned, leveraging its generative pre-training knowledge while adapting specifically to sentiment classification through parameter-efficient fine-tuning.

Can I run these sentiment analysis models without a GPU?

XGBoost and LSTM run efficiently on CPU-only environments, with XGBoost providing the fastest inference through optimized tree traversal and multithreading support. BERT and GPT-2 benefit significantly from GPU acceleration due to transformer matrix operations, though the BERT implementation minimizes GPU memory requirements by freezing backbone parameters, while GPT-2's LoRA adapters keep the trainable footprint small even during inference.

How does the LoRA fine-tuning in GPT-2 differ from the frozen BERT approach?

The GPT-2 implementation uses LoRA (Low-Rank Adaptation) to inject small trainable matrices into the attention layers while keeping the base model frozen, allowing the model to learn sentiment-specific adaptations without catastrophic forgetting of generative knowledge. In contrast, the BERT approach completely freezes all 12 encoder layers and trains only a new linear classifier on top of the [CLS] token representation, treating the transformer as a fixed feature extractor rather than an adaptable model.

Which model should I choose for production deployment with limited resources?

For resource-constrained production environments, XGBoost offers the optimal balance of speed and accuracy, requiring only CPU resources and providing interpretable predictions through its tree-based structure. If neural approaches are necessary but GPU resources are limited, the LSTM model with 64-dimensional Word2Vec embeddings provides reasonable accuracy with moderate computational requirements, though it lacks the contextual depth of transformer-based sentiment analysis models.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}