---
library_name: transformers
license: mit
language:
- en
metrics:
- accuracy
- f1
base_model:
- FacebookAI/roberta-base
pipeline_tag: text-classification
---

# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

LoRA-based fine-tuned RoBERTa model for multi-intent classification in natural language utterances.

---

## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->

- **Developed by:** Taewook Kang (Github: [@twkang43](https://github.com/twkang43))
- **Model type:** Fine-tuned [RoBERTa](https://arxiv.org/abs/1907.11692) model with [Low-Rank Adaptation](https://arxiv.org/abs/2106.09685) (LoRA) for multiple intent classification.
- **Language(s) (NLP):** English
- **License:** MIT License
- **Finetuned from model:** [RoBERTa-base](https://huggingface.co/FacebookAI/roberta-base)

## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->


## How to Get Started with the Model

Use the code below to download the tokenizer and fine-tuned model with LoRA.

```python
from transformers import AutoTokenizer,AutoModelForSequenceClassification
from peft import PeftModel

tokenizer = AutoTokenizer.from_pretrained("twkang43/lora-roberta-cse4057")
base_model = AutoModelForSequenceClassification.from_pretrained(
    MODEL_NAME,
    problem_type="multi_label_classification",
    num_labels=num_labels,
    id2label=id2label,
    label2id=label2id
).to(DEVICE)
model = PeftModel.from_pretrained(base_model, "twkang43/lora-roberta-cse4057")
```

## Training Details

### Training Data

Fine-tuned on the [BlendX](https://arxiv.org/abs/2403.18277) dataset ("train" dataset was split into training and validation datasets with a 9:1 ratio).

### Training Procedure

This model is primarily fine-tuned with LoRA.

#### Training Hyperparameters

- **Training regime:** fp16 mixed precision
- **Training epochs:** 10
- **Batch size:** 16
- **Learning rate:** 1e-3
- **Max gradient norm:** 1.0
- **Weight decay:** 0.0
- **Lr scheduler:** cosine
- **Warmup ratio:** 0.1

## Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

### Testing Data, Factors & Metrics

#### Testing Data

The model was tested on the [BlendX](https://arxiv.org/abs/2403.18277) dataset ("dev" data).

#### Metrics

- Accuracy with micro
- F1 score with micro

### Results

- **Accuracy = 0.87**
- **F1 score = 0.93**

#### Summary




### Compute Infrastructure

This model was primarily trained using Google Colab (free tier).

#### Hardware

- GPU: NVIDIA T4 with 16GB VRAM
- RAM: 16GB
- Processor: Shared virtual CPUs (details depend on Google Colab free tier allocations)

#### Software

- PyTorch
- Jupyter Notebook
- Tensorboard

## Model Card Contact

- twkang43@hanyang.ac.kr