# BERT vs GPT-2 vs XGBoost vs LSTM: Comparing Sentiment Analysis Models in Bettafish

> Compare BERT GPT-2 XGBoost and LSTM sentiment analysis models in bettafish. Understand their distinct machine learning paradigms and find the best approach for your needs.

- Repository: [BaiFu/bettafish](https://github.com/666ghj/bettafish)
- Tags: comparison
- Published: 2026-02-23

---

**The four sentiment analysis models in the 666ghj/bettafish repository represent distinct machine learning paradigms: BERT uses a frozen encoder with a trainable classifier, GPT-2 applies LoRA adapters to a decoder-only transformer, XGBoost relies on gradient-boosted trees with bag-of-words features, and LSTM implements a bidirectional recurrent network with custom Word2Vec embeddings.**

The 666ghj/bettafish repository provides a comprehensive comparison of modern and classical approaches to Chinese sentiment analysis. Each sentiment analysis model targets different resource constraints and accuracy requirements, from lightweight tree-based methods to large transformer architectures. Understanding these architectural differences helps developers select the optimal approach for their specific deployment environment.

## Architectural Overview of the Four Sentiment Analysis Models

### BERT: Encoder-Only Transformer

The BERT implementation leverages the `bert-base-chinese` pre-trained weights through the `BertModel_Custom` class defined in [`SentimentAnalysisModel/WeiboSentiment_MachineLearning/bert_train.py`](https://github.com/666ghj/bettafish/blob/main/SentimentAnalysisModel/WeiboSentiment_MachineLearning/bert_train.py). This encoder-only transformer processes input text through 12 layers of bidirectional self-attention, producing 768-dimensional contextualized embeddings. The architecture freezes all BERT parameters during training, optimizing only a single linear classification head that maps the `[CLS]` token representation to sentiment probabilities.

### GPT-2 with LoRA: Decoder-Only Transformer

The GPT-2 implementation utilizes a decoder-only architecture based on `gpt2-chinese-cluecorpussmall`, enhanced with Parameter-Efficient Fine-Tuning (PEFT) using LoRA adapters. Located in [`SentimentAnalysisModel/WeiboSentiment_Finetuned/GPT2-Lora/train.py`](https://github.com/666ghj/bettafish/blob/main/SentimentAnalysisModel/WeiboSentiment_Finetuned/GPT2-Lora/train.py), this approach wraps `GPT2ForSequenceClassification` with `LoraConfig` to inject trainable low-rank matrices into the attention projection layers. This design preserves the pre-trained generative knowledge while reducing trainable parameters by over 90% compared to full fine-tuning.

### XGBoost: Gradient-Boosted Decision Trees

The XGBoost implementation represents the classical machine learning approach, implemented in [`SentimentAnalysisModel/WeiboSentiment_MachineLearning/xgboost_train.py`](https://github.com/666ghj/bettafish/blob/main/SentimentAnalysisModel/WeiboSentiment_MachineLearning/xgboost_train.py) through the `XGBoostModel` class. This method constructs a sparse bag-of-words representation using `CountVectorizer` with a default vocabulary size of 2000 features. The algorithm trains an ensemble of decision trees using gradient boosting, optimizing a logistic objective for binary sentiment classification without requiring any neural network dependencies or GPU resources.

### LSTM: Bidirectional Recurrent Network

The LSTM implementation provides a deep learning approach without transformer architecture, defined in [`SentimentAnalysisModel/WeiboSentiment_MachineLearning/lstm_train.py`](https://github.com/666ghj/bettafish/blob/main/SentimentAnalysisModel/WeiboSentiment_MachineLearning/lstm_train.py) via the `LSTMModel` class. This bidirectional LSTM network processes sequences of Word2Vec embeddings (default 64-dimensional vectors) through two recurrent layers, capturing temporal dependencies in both forward and backward directions. The architecture pre-trains embeddings on the corpus using Word2Vec before freezing them during the classification phase.

## Key Technical Differences

### Feature Representation Strategies

Each sentiment analysis model employs distinct feature extraction methodologies. **BERT** utilizes contextualized subword embeddings from 12 transformer layers, capturing complex linguistic relationships through self-attention mechanisms. **GPT-2** leverages causal language modeling representations from its decoder layers, modified through LoRA adapters to adapt to classification tasks. **XGBoost** relies on high-dimensional sparse vectors from `CountVectorizer`, treating text as unordered term frequencies. **LSTM** uses dense Word2Vec embeddings trained specifically on the Weibo sentiment corpus, preserving semantic relationships in continuous vector space.

### Training Regimes and Resource Requirements

The training methodologies vary significantly across implementations. The **BERT** model in [`bert_train.py`](https://github.com/666ghj/bettafish/blob/main/bert_train.py) freezes the transformer backbone (`self.bert.eval()` at line 69), training only the linear classifier head, which enables fast convergence on modest GPU memory. The **GPT-2** implementation applies full fine-tuning exclusively to LoRA adapter parameters while maintaining the base model in evaluation mode, utilizing AdamW with linear learning rate scheduling and gradient clipping to prevent explosion. **XGBoost** trains entirely on CPU using multithreading support (`nthread` parameter), requiring no gradient descent or backpropagation. The **LSTM** model trains end-to-end through standard backpropagation through time, with Word2Vec vectors pre-trained and subsequently frozen during the classification training phase.

### Inference Speed and Deployment Characteristics

Runtime performance characteristics differ based on architectural complexity. **BERT** inference requires a full forward pass through the 12-layer encoder plus the classification head, though the frozen backbone allows for potential optimization through TorchScript or ONNX conversion. **GPT-2** necessitates processing through the decoder transformer architecture; while LoRA adapters minimize trainable parameters, the full model must still execute during inference, making it slightly heavier than BERT. **XGBoost** delivers the fastest CPU inference, traversing the trained tree ensemble with logarithmic complexity relative to tree depth, making it ideal for high-throughput, resource-constrained environments. **LSTM** processes sequences token-by-token through recurrent connections, creating inherent sequential dependency that prevents parallelization across the time dimension, resulting in moderate speed that scales linearly with sequence length.

## Code Examples: Loading and Running Each Model

### BERT Implementation

```python
from SentimentAnalysisModel.WeiboSentiment_MachineLearning.bert_train import BertModel_Custom
from SentimentAnalysisModel.WeiboSentiment_MachineLearning.base_model import BaseModel

# Load data (train_path / test_path are examples)

train_data, test_data = BaseModel.load_data('./data/weibo2018/train.txt',
                                           './data/weibo2018/test.txt')

# Initialise and train

model = BertModel_Custom('./model/chinese_wwm_pytorch')
model.train(train_data, num_epochs=5, batch_size=64, learning_rate=2e-3)

# Predict a list of sentences

texts = ["今天天气真好，心情很棒", "这部电影太无聊了，浪费时间"]
preds = model.predict(texts)
print(preds)   # [1, 0] → 1: 正面, 0: 负面

```

### GPT-2 with LoRA Implementation

```python
from transformers import GPT2ForSequenceClassification, BertTokenizer
from peft import LoraConfig, get_peft_model
import torch

# Load tokenizer and model (already fine‑tuned in "./best_weibo_sentiment_lora")

tokenizer = BertTokenizer.from_pretrained('./models/gpt2-chinese')
model = GPT2ForSequenceClassification.from_pretrained('./best_weibo_sentiment_lora')
model.eval()

def predict(text):
    enc = tokenizer(text, truncation=True, padding='max_length',
                    max_length=128, return_tensors='pt')
    with torch.no_grad():
        logits = model(**{k: v.to(model.device) for k, v in enc.items()}).logits
    prob = torch.softmax(logits, dim=-1)[0, 1].item()   # probability of positive class

    label = int(prob > 0.5)
    return label, prob

print(predict("我很喜欢这部电影"))   # (1, 0.87)

```

### XGBoost Implementation

```python
from SentimentAnalysisModel.WeiboSentiment_MachineLearning.xgboost_train import XGBoostModel
from SentimentAnalysisModel.WeiboSentiment_MachineLearning.base_model import BaseModel

train_data, test_data = BaseModel.load_data('./data/weibo2018/train.txt',
                                           './data/weibo2018/test.txt')

model = XGBoostModel()
model.train(train_data, max_features=2000, max_depth=6, eta=0.3,
            num_boost_round=200)

# Single prediction

label, confidence = model.predict_single("这个产品真的很垃圾")
print(label, confidence)   # 0 0.92  (negative)

```

### LSTM Implementation

```python
from SentimentAnalysisModel.WeiboSentiment_MachineLearning.lstm_train import LSTMModel
from SentimentAnalysisModel.WeiboSentiment_MachineLearning.base_model import BaseModel

train_data, test_data = BaseModel.load_data('./data/weibo2018/train.txt',
                                           './data/weibo2018/test.txt')

model = LSTMModel()
model.train(train_data, embed_size=64, hidden_size=64,
            num_layers=2, learning_rate=5e-4, num_epochs=5)

label, confidence = model.predict_single("这家餐厅的服务太差了")
print(label, confidence)   # 0 0.81  (negative)

```

## File Structure and Implementation Details

The 666ghj/bettafish repository organizes these sentiment analysis models into distinct modules, each implementing a unified interface through the `BaseModel` abstract class.

- **[`SentimentAnalysisModel/WeiboSentiment_MachineLearning/bert_train.py`](https://github.com/666ghj/bettafish/blob/main/SentimentAnalysisModel/WeiboSentiment_MachineLearning/bert_train.py)** – Contains the `BertModel_Custom` class with auto-download logic for `bert-base-chinese` weights and the frozen backbone training strategy.

- **[`SentimentAnalysisModel/WeiboSentiment_Finetuned/GPT2-Lora/train.py`](https://github.com/666ghj/bettafish/blob/main/SentimentAnalysisModel/WeiboSentiment_Finetuned/GPT2-Lora/train.py)** – Implements the `GPT2ForSequenceClassification` wrapper with `LoraConfig` for parameter-efficient fine-tuning, including gradient clipping and linear learning rate scheduling.

- **[`SentimentAnalysisModel/WeiboSentiment_MachineLearning/xgboost_train.py`](https://github.com/666ghj/bettafish/blob/main/SentimentAnalysisModel/WeiboSentiment_MachineLearning/xgboost_train.py)** – Houses the `XGBoostModel` class utilizing `CountVectorizer` for sparse feature extraction and XGBoost's gradient boosting with configurable tree depth and learning rate.

- **[`SentimentAnalysisModel/WeiboSentiment_MachineLearning/lstm_train.py`](https://github.com/666ghj/bettafish/blob/main/SentimentAnalysisModel/WeiboSentiment_MachineLearning/lstm_train.py)** – Defines the `LSTMModel` class with custom Word2Vec embedding training (lines 90-107) and a bidirectional LSTM architecture with configurable hidden sizes and layer counts.

- **[`SentimentAnalysisModel/WeiboSentiment_MachineLearning/base_model.py`](https://github.com/666ghj/bettafish/blob/main/SentimentAnalysisModel/WeiboSentiment_MachineLearning/base_model.py)** – Provides the abstract `BaseModel` class standardizing `load_data`, `evaluate`, and prediction interfaces across all implementations.

## Summary

- **BERT** delivers high accuracy through contextualized transformer embeddings while maintaining reasonable GPU requirements by freezing the backbone and training only a linear classifier head.

- **GPT-2 with LoRA** leverages generative pre-training through decoder-only architecture while minimizing fine-tuning costs via low-rank adapters, making large transformer technology accessible with limited compute.

- **XGBoost** provides the fastest CPU inference through gradient-boosted decision trees on sparse bag-of-words features, requiring no deep learning dependencies or GPU resources.

- **LSTM** offers a middle ground with custom Word2Vec embeddings and recurrent processing, suitable for educational purposes or environments where transformer models are impractical but neural approaches are preferred.

## Frequently Asked Questions

### Which sentiment analysis model provides the best accuracy for Chinese text?

The **BERT** implementation generally provides the highest accuracy due to its deep bidirectional contextualization through 12 transformer layers and pre-training on extensive Chinese corpora. However, the **GPT-2 with LoRA** approach can achieve comparable performance when properly fine-tuned, leveraging its generative pre-training knowledge while adapting specifically to sentiment classification through parameter-efficient fine-tuning.

### Can I run these sentiment analysis models without a GPU?

**XGBoost** and **LSTM** run efficiently on CPU-only environments, with XGBoost providing the fastest inference through optimized tree traversal and multithreading support. **BERT** and **GPT-2** benefit significantly from GPU acceleration due to transformer matrix operations, though the BERT implementation minimizes GPU memory requirements by freezing backbone parameters, while GPT-2's LoRA adapters keep the trainable footprint small even during inference.

### How does the LoRA fine-tuning in GPT-2 differ from the frozen BERT approach?

The **GPT-2** implementation uses **LoRA (Low-Rank Adaptation)** to inject small trainable matrices into the attention layers while keeping the base model frozen, allowing the model to learn sentiment-specific adaptations without catastrophic forgetting of generative knowledge. In contrast, the **BERT** approach completely freezes all 12 encoder layers and trains only a new linear classifier on top of the `[CLS]` token representation, treating the transformer as a fixed feature extractor rather than an adaptable model.

### Which model should I choose for production deployment with limited resources?

For **resource-constrained production environments**, **XGBoost** offers the optimal balance of speed and accuracy, requiring only CPU resources and providing interpretable predictions through its tree-based structure. If neural approaches are necessary but GPU resources are limited, the **LSTM** model with 64-dimensional Word2Vec embeddings provides reasonable accuracy with moderate computational requirements, though it lacks the contextual depth of transformer-based sentiment analysis models.