|
--- |
|
license: mit |
|
language: |
|
- ro |
|
metrics: |
|
- accuracy |
|
base_model: google-bert/bert-base-multilingual-cased |
|
library_name: transformers.js |
|
tags: |
|
- romanian |
|
- fake news |
|
- BERT |
|
--- |
|
# Model Card for BERT |
|
|
|
This model classifies Romanian fake news in ["fake_news", "misinformation", "propaganda", "real_news", "satire"] |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
This model is a BERT-based model fine-tuned for the task of detecting fake news in Romanian text. It is designed to classify text into one of several predefined labels indicating the likelihood of the text being fake news. |
|
|
|
|
|
- **Developed by:** Bogdan Mihalca |
|
- **Model type:** BERT |
|
- **License:** MIT |
|
- **Finetuned from model:** bert-base-multilingual-cased |
|
|
|
### Model Sources [optional] |
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
- **Repository:** [More Information Needed] |
|
- **Paper [optional]:** [More Information Needed] |
|
- **Demo [optional]:** [More Information Needed] |
|
|
|
## Uses |
|
|
|
### Direct Use |
|
This model can be directly used to classify Romanian text into five categories: fake news, misinformation, propaganda, real news, and satire. |
|
It can be handy for media monitoring platforms, news verifiers, and research on misinformation. |
|
|
|
|
|
### Downstream Use |
|
|
|
Potential applications include integration into larger systems for real-time news validation or being part of a research toolkit for studying misinformation. |
|
|
|
### Out-of-Scope Use |
|
|
|
This model is not intended for classifying non-Romanian text, and its accuracy may diminish significantly if used on text outside of the Romanian language. |
|
It is also not suitable for detailed sentiment analysis or non-news related content. |
|
|
|
|
|
## Bias, Risks, and Limitations |
|
|
|
This model may exhibit bias due to the nature of the training data, which could lead to overfitting on certain types of news or propaganda specific to Romanian contexts. |
|
It might not generalize well to new, unseen forms of misinformation or satire. |
|
|
|
|
|
### Recommendations |
|
|
|
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. --> |
|
Users should evaluate the model's performance on their specific use case before deployment, particularly if applied in contexts different from those in which the training data was collected. |
|
|
|
|
|
## How to Get Started with the Model |
|
|
|
To get started with this model, load it using HuggingFace's Transformers library and ensure you have the appropriate tokenizer |
|
. Example code for loading and using the model is provided in the Python script. |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
The model was trained on a combined dataset of Romanian fake news, including data scraped from Veridica.ro and the Fakerom dataset. |
|
The dataset includes categories such as fake news, misinformation, propaganda, real news, and satire. |
|
### Training Procedure |
|
|
|
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. --> |
|
|
|
#### Preprocessing [optional] |
|
|
|
[More Information Needed] |
|
|
|
|
|
#### Training Hyperparameters |
|
|
|
- **Training regime:** The model was trained using fp32 precision. |
|
- **Number of epochs:** 3 |
|
- **Batch size:** 16 |
|
- **Warmup steps:** 500 |
|
- **Weight decay:** 0.01 |
|
- **Learning rate scheduler:** Warmup with cosine decay. |
|
|
|
|
|
## Evaluation |
|
|
|
|
|
### Testing Data, Factors & Metrics |
|
|
|
#### Testing Data |
|
|
|
The model was tested on a split of the combined dataset, with 20% of the data used for evaluation. |
|
|
|
#### Factors |
|
The evaluation was performed across all five classes of fake news categories, with results disaggregated by each category. |
|
|
|
|
|
#### Metrics |
|
|
|
The primary evaluation metrics were accuracy, precision, recall, F1 score, and log loss. ROC AUC scores were also computed for each class. |
|
|
|
|
|
### Results |
|
|
|
Results |
|
|
|
Accuracy: 93.51% |
|
Precision: 93.80% |
|
Recall: 93.51% |
|
F1 Score: 93.56% |
|
Log Loss: 0.225 |
|
ROC AUC Per Class: |
|
Fake News: 98.50% |
|
Misinformation: 98.89% |
|
Propaganda: 98.98% |
|
Real News: 99.55% |
|
Satire: 99.99% |
|
|
|
|
|
|
|
#### Summary |
|
The model demonstrates strong performance across all classes, with particularly high ROC AUC scores indicating good separability between classes. |
|
|
|
|
