metadata
library_name: transformers
license: mit
language:
- en
metrics:
- accuracy
- f1
base_model:
- FacebookAI/roberta-base
pipeline_tag: text-classification
Model Card for Model ID
LoRA-based fine-tuned RoBERTa model for multi-intent classification in natural language utterances.
Model Details
Model Description
- Developed by: Taewook Kang (Github: @twkang43)
- Model type: Fine-tuned RoBERTa model with Low-Rank Adaptation (LoRA) for multiple intent classification.
- Language(s) (NLP): English
- License: MIT License
- Finetuned from model: RoBERTa-base
Uses
How to Get Started with the Model
Use the code below to download the tokenizer and fine-tuned model with LoRA.
from transformers import AutoTokenizer,AutoModelForSequenceClassification
from peft import PeftModel
tokenizer = AutoTokenizer.from_pretrained("twkang43/lora-roberta-cse4057")
base_model = AutoModelForSequenceClassification.from_pretrained(
MODEL_NAME,
problem_type="multi_label_classification",
num_labels=num_labels,
id2label=id2label,
label2id=label2id
).to(DEVICE)
model = PeftModel.from_pretrained(base_model, "twkang43/lora-roberta-cse4057")
Training Details
Training Data
Fine-tuned on the BlendX dataset ("train" dataset was split into training and validation datasets with a 9:1 ratio).
Training Procedure
This model is primarily fine-tuned with LoRA.
Training Hyperparameters
- Training regime: fp16 mixed precision
- Training epochs: 10
- Batch size: 16
- Learning rate: 1e-3
- Max gradient norm: 1.0
- Weight decay: 0.0
- Lr scheduler: cosine
- Warmup ratio: 0.1
Evaluation
Testing Data, Factors & Metrics
Testing Data
The model was tested on the BlendX dataset ("dev" data).
Metrics
- Accuracy with micro
- F1 score with micro
Results
- Accuracy = 0.87
- F1 score = 0.93
Summary
Compute Infrastructure
This model was primarily trained using Google Colab (free tier).
Hardware
- GPU: NVIDIA T4 with 16GB VRAM
- RAM: 16GB
- Processor: Shared virtual CPUs (details depend on Google Colab free tier allocations)
Software
- PyTorch
- Jupyter Notebook
- Tensorboard