README.md

---
base_model: google/gemma-2-2b-it
library_name: peft
tags:
- sentiment-analysis
- weighted-loss
- LoRA
- Korean
---

# Model Card for Fine-Tuned `gemma-2-2b-it` on Custom Korean Sentiment Dataset

## Model Summary

This model is a fine-tuned version of `google/gemma-2-2b-it`, trained to classify sentiment in Korean text into four categories: **무감정** (neutral), **슬픔** (sadness), **기쁨** (joy), and **분노** (anger). The model utilizes **LoRA (Low-Rank Adaptation)** for efficient fine-tuning and **4-bit quantization (NF4)** for memory efficiency using **BitsAndBytes**. A custom weighted loss function was applied to handle class imbalance within the dataset.

The model is suitable for multi-class sentiment classification in Korean and is optimized for environments with limited computational resources due to the quantization.

## Model Details

### Developed By:
This model was fine-tuned by [Your Name or Organization] using Hugging Face's `peft` and `transformers` libraries with a custom Korean sentiment dataset.

### Model Type:
This is a transformer-based model for **multi-class sentiment classification** in the Korean language.

### Language:
- **Language(s)**: Korean

### License:
[Add relevant license here]

### Finetuned From:
- **Base Model**: `google/gemma-2-2b-it`

### Framework Versions:
- **Transformers**: 4.44.2
- **PEFT**: 0.12.0
- **Datasets**: 3.0.1
- **PyTorch**: 2.4.1+cu121

## Intended Uses & Limitations

### Intended Use:
This model is suitable for applications requiring multi-class sentiment classification in Korean, such as chatbots, social media monitoring, or customer feedback analysis.

### Out-of-Scope Use:
The model may not perform optimally for tasks requiring multi-language support, sentiment classification with additional classes, or outside the specific context of Korean language data.

### Limitations:
- **Bias**: As the model is trained on a custom dataset, it may reflect specific biases inherent in that data.
- **Generalization**: Performance may vary when applied to datasets outside the scope of the initial training data, such as other forms of sentiment classification.

## Model Architecture

### Quantization:
The model uses **4-bit quantization** via **BitsAndBytes** for efficient memory usage, which enables it to run on lower-resource hardware.

### LoRA Configuration:
LoRA (Low-Rank Adaptation) was applied to specific transformer layers, allowing for parameter-efficient fine-tuning. The target modules include:
- `down_proj`, `gate_proj`, `q_proj`, `o_proj`, `up_proj`, `v_proj`, `k_proj`

LoRA parameters are:
- `r = 16`, `lora_alpha = 32`, `lora_dropout = 0.05`

### Custom Weighted Loss:
A custom weighted loss function was implemented to handle class imbalance, using the following weights:

\[
\text{weights} = [0.2032, 0.2704, 0.2529, 0.2735]
\]

These weights correspond to the classes: **무감정**, **슬픔**, **기쁨**, **분노**, respectively.

## Training Details

### Dataset:
The model was trained on a custom Korean sentiment analysis dataset. This dataset consists of text samples labeled with one of four sentiment classes: **무감정**, **슬픔**, **기쁨**, and **분노**.

- **Train Set Size**: Custom dataset
- **Test Set Size**: Custom dataset
- **Classes**: 4 (무감정, 슬픔, 기쁨, 분노)

### Preprocessing:
Data was tokenized using the `google/gemma-2-2b-it` tokenizer with a maximum sequence length of 128. The preprocessing steps included padding and truncation to ensure consistent input lengths.

### Hyperparameters:

- **Learning Rate**: 2e-4
- **Batch Size (train)**: 8
- **Batch Size (eval)**: 8
- **Epochs**: 4
- **Optimizer**: AdamW (with 8-bit optimization)
- **Weight Decay**: 0.01
- **Gradient Accumulation Steps**: 2
- **Evaluation Steps**: 500
- **Logging Steps**: 500
- **Metric for Best Model**: F1 (weighted)

## Evaluation

### Metrics:
The model was evaluated using the following metrics:
- **Accuracy**
- **F1 Score** (weighted)
- **Precision** (weighted)
- **Recall** (weighted)

The evaluation provides a detailed view of the model's performance across multiple metrics, which helps in understanding its strengths and areas for improvement.

### Code Example:

You can load the fine-tuned model and use it for inference on your own data as follows:

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification

# Load model and tokenizer
model = AutoModelForSequenceClassification.from_pretrained("your-model-directory")
tokenizer = AutoTokenizer.from_pretrained("your-model-directory")

# Tokenize input text
text = "이 영화는 정말 슬퍼요."
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)

# Get predictions
outputs = model(**inputs)
logits = outputs.logits
predicted_class = logits.argmax(-1).item()

# Map prediction to label
id2label = {0: "무감정", 1: "슬픔", 2: "기쁨", 3: "분노"}
print(f"Predicted sentiment: {id2label[predicted_class]}")