base_model: google/pegasus-cnn_dailymail
tags:
- generated_from_trainer
model-index:
- name: pegasus-samsum-nlp-with-transformers-ch06
results: []
datasets:
- samsum
language:
- en
metrics:
- rouge
pipeline_tag: summarization
pegasus-samsum-nlp-with-transformers-ch06
This model is a fine-tuned version of google/pegasus-cnn_dailymail on the SAMSum dataset.
It achieves the following results on the evaluation set:
- Loss: 1.4839
It achieves the following ROUGE scores on the test set:
- rouge1: 0.555556
- rouge2: 0.230769
- rougeL: 0.518519
- rougeLsum: 0.518519
Quick human evaluation of summarization quality: the results are generally good, after visual inspection of the summaries generated on test set conversations. However it seems some entities/attributions are incorrect (saw an example where model confuses peoples' roles in multi-person chat)
Model description
PEGASUS doc can be found here: https://huggingface.co/docs/transformers/model_doc/pegasus
Intended uses & limitations
This model was trained while studying the NLP With Transformers book; it is not intended to be used for any real applications.
Training and evaluation data
The finetuning data is the SAMSum dataset only.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 1
- eval_batch_size: 1
- seed: 42
- gradient_accumulation_steps: 16
- total_train_batch_size: 16
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- num_epochs: 1
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 1.6592 | 0.54 | 500 | 1.4839 |
Framework versions
- Transformers 4.39.3
- Pytorch 2.1.2
- Datasets 2.18.0
- Tokenizers 0.15.2