language: english
widget:
- text: Mi estas viro kej estas tago varma.
Welcome to RoBERTArg!
🤖 Model description
This model was trained on ~25k heterogeneous manually annotated sentences (📚 Stab et al. 2018) of controversial topics to classify text into one of two labels: 🏷 NON-ARGUMENT (0) and ARGUMENT (1).
🗃 Dataset
The dataset (📚 Stab et al. 2018) consists of ARGUMENTS (11k) that either support or oppose a topic if it includes a relevant reason for supporting or opposing the topic, or as a NON-ARGUMENT (14k) if it does not include reasons. The authors focus on controversial topics, i.e., topics that include an obvious polarity to the possible outcomes and compile a final set of eight controversial topics: abortion, school uniforms, death penalty, marijuana legalization, nuclear energy, cloning, gun control, and minimum wage.
| TOPIC | ARGUMENT | NON-ARGUMENT |
|---|---|---|
| abortion | 2213 | 2,427 |
| school uniforms | 325 | 1,734 |
| death penalty | 325 | 2,083 |
| marijuana legalization | 325 | 1,262 |
| nuclear energy | 325 | 2,118 |
| cloning | 325 | 1,494 |
| gun control | 325 | 1,889 |
| minimum wage | 325 | 1,346 |
🏃🏼♂️Model training
RoBERTArg was fine-tuned on a RoBERTA (base) pre-trained model from HuggingFace using the HuggingFace trainer with the following hyperparameters. The hyperparameters were determined using a hyperparameter search on a 20% validation set.
training_args = TrainingArguments(
num_train_epochs=2,
learning_rate=2.3102e-06,
seed=8,
per_device_train_batch_size=64,
per_device_eval_batch_size=64,
)
📊 Evaluation
The model was evaluated using 20% of the sentences (80-20 train-test split).
| Model | Acc | F1 | R arg | R non | P arg | P non |
|---|---|---|---|---|---|---|
| RoBERTArg | 0.8193 | 0.8021 | 0.8463 | 0.7986 | 0.7623 | 0.8719 |
Showing the confusion matrix using the 20% of the sentences as an evaluation set:
| ARGUMENT | NON-ARGUMENT | |
|---|---|---|
| ARGUMENT | 2213 | 558 |
| NON-ARGUMENT | 325 | 1790 |
⚠️ Intended Uses & Potential Limitations
The model can be a starting point to dive into the exciting area of argument mining. But be aware. An argument is a complex structure, topic-dependent, and often differs between different text types. Therefore, the model may perform less well on different topics and text types, which are not included in the training set.
Enjoy and stay tuned! 🚀
📚Stab et al. (2018): Cross-topic Argument Mining from Heterogeneous Sources. LINK.