knachinen/flan-t5-xl-finetuned-arxiv

flan-t5-xl-finetuned-arxiv

This model is a fine-tuned version of google/flan-t5-xl on the scientific_papers dataset. It achieves the following results on the evaluation set:

• Loss: 2.071128
• Rouge1: 12.600400
• Rouge2: 4.873200
• Rougel: 10.268200
• Rougelsum: 11.653600
• Gen Len: 19.000000

Training procedure

The following bitsandbytes quantization config was used during training:

• quant_method: bitsandbytes
• load_in_8bit: False
• load_in_4bit: True
• llm_int8_threshold: 6.0
• llm_int8_skip_modules: None
• llm_int8_enable_fp32_cpu_offload: False
• llm_int8_has_fp16_weight: False
• bnb_4bit_quant_type: nf4
• bnb_4bit_use_double_quant: True
• bnb_4bit_compute_dtype: bfloat16

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-4
• weight_decay: 0.01
• train_batch_size: 16
• optimizer: paged_adamw_8bit (8-bit adam optimization)
• num_epochs: 0.26 (datasets: 52,800 samples)
• fp16: False

Framework versions

• PEFT 0.5.0
• Transformers 4.35.0
• Pytorch 1.10.1+cu111
• Datasets 2.14.7
• Tokenizers 0.14.1
• bitsandbytes 0.41.2.post2
• accelerate 0.24.0
• evaluate 0.4.1
• rouge-score 0.1.2