metadata

license: mit
base_model: roberta-base
tags:
  - generated_from_trainer
metrics:
  - accuracy
  - recall
  - precision
  - f1
model-index:
  - name: roberta-base-suicide-prediction-phr-v2
    results:
      - task:
          type: text-classification
          name: Suicidal Tendency Prediction in text
        dataset:
          type: vibhorag101/phr_suicide_prediction_dataset_clean_light
          name: Suicide Prediction Dataset
          split: val
        metrics:
          - type: accuracy
            value: 0.9869
          - type: f1
            value: 0.9875
          - type: recall
            value: 0.9846
          - type: precision
            value: 0.9904
datasets:
  - vibhorag101/phr_suicide_prediction_dataset_clean_light
language:
  - en
library_name: transformers

vibhorag101/roberta-base-suicide-prediction-phr-v2

This model is a fine-tuned version of roberta-base on Suicide Prediction Dataset, sourced from Reddit. It achieves the following results on the evaluation set:

Loss: 0.0553
Accuracy: 0.9869
Recall: 0.9846
Precision: 0.9904
F1: 0.9875

Model description

This model is a finetune of roberta-base to detect suicidal tendencies in a given text.

Training and evaluation data

The dataset is sourced from Reddit and is available on Kaggle.
The dataset contains text with binary labels for suicide or non-suicide.
The dataset was cleaned minimally, as BERT depends on contextually sensitive information, which can worsely effect its performance.
- Removed numbers
- Removed URLs, Emojis, and accented characters.
- Remove any extra white spaces and any extra spaces after a single space.
- Removed any consecutive characters repeated more than 3 times.
- The rows with more than 512 BERT Tokens were removed, as they exceeded BERT's max token.
The cleaned dataset can be found here
The evaluation set had ~33k samples, while the training set had ~153k samples, i.e., a 70:15:15 (train:test:val) split.

Training procedure

The model was trained on an RTXA5000 GPU.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 16
eval_batch_size: 32
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
weight_decay=0.1
warmup_ratio: 0.06
num_epochs: 3
eval_steps: 500
save_steps: 500
Early Stopping:
- early_stopping_patience: 5
- early_stopping_threshold: 0.001
- parameter: F1 Score

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Recall	Precision	F1
0.1928	0.05	500	0.2289	0.9340	0.9062	0.9660	0.9352
0.0833	0.1	1000	0.1120	0.9752	0.9637	0.9888	0.9761
0.0366	0.16	1500	0.1165	0.9753	0.9613	0.9915	0.9762
0.071	0.21	2000	0.0973	0.9709	0.9502	0.9940	0.9716
0.0465	0.26	2500	0.0680	0.9829	0.9979	0.9703	0.9839
0.0387	0.31	3000	0.1583	0.9705	0.9490	0.9945	0.9712
0.1061	0.37	3500	0.0685	0.9848	0.9802	0.9907	0.9854
0.0593	0.42	4000	0.0550	0.9872	0.9947	0.9813	0.9879
0.0382	0.47	4500	0.0551	0.9871	0.9912	0.9842	0.9877
0.0831	0.52	5000	0.0502	0.9840	0.9768	0.9927	0.9847
0.0376	0.58	5500	0.0654	0.9865	0.9852	0.9889	0.9871
0.0634	0.63	6000	0.0422	0.9877	0.9897	0.9870	0.9883
0.0235	0.68	6500	0.0553	0.9869	0.9846	0.9904	0.9875

Framework versions

Transformers 4.38.2
Pytorch 2.1.0+cu121
Datasets 2.18.0
Tokenizers 0.15.0