Hugging Face's logo

kanishka
/
opt-babylm2-rewritten-clean-spacy-earlystop-bpe_seed-42_1e-3

Text Generation
Transformers
Safetensors
opt
Generated from Trainer
Eval Results
text-generation-inference
Inference Endpoints
Model card Files Community
opt-babylm2-rewritten-clean-spacy-earlystop-bpe_seed-42_1e-3 / README.md
kanishka's picture
kanishka
End of training
34e82f8 verified
preview code
|
raw
history blame contribute delete
3.17 kB
metadata
library_name: transformers
tags:
  - generated_from_trainer
datasets:
  - kanishka/babylm2-rewritten-clean-spacy
metrics:
  - accuracy
model-index:
  - name: opt-babylm2-rewritten-clean-spacy-earlystop-bpe_seed-42_1e-3
    results:
      - task:
          name: Causal Language Modeling
          type: text-generation
        dataset:
          name: kanishka/babylm2-rewritten-clean-spacy
          type: kanishka/babylm2-rewritten-clean-spacy
        metrics:
          - name: Accuracy
            type: accuracy
            value: 0.47868057440510814

opt-babylm2-rewritten-clean-spacy-earlystop-bpe_seed-42_1e-3

This model was trained from scratch on the kanishka/babylm2-rewritten-clean-spacy dataset. It achieves the following results on the evaluation set:

  • Loss: 2.6840
  • Accuracy: 0.4787

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.001
  • train_batch_size: 32
  • eval_batch_size: 64
  • seed: 42
  • gradient_accumulation_steps: 8
  • total_train_batch_size: 256
  • optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 32000
  • num_epochs: 20.0
  • mixed_precision_training: Native AMP

Training results

Training Loss Epoch Step Validation Loss Accuracy
4.1044 1.0 2256 3.8204 0.3604
3.4457 2.0 4512 3.3046 0.4093
3.13 3.0 6768 3.0945 0.4299
2.9219 4.0 9024 2.9890 0.4404
2.8444 5.0 11280 2.9282 0.4466
2.7883 6.0 13536 2.8910 0.4508
2.7434 7.0 15792 2.8579 0.4545
2.7158 8.0 18048 2.8428 0.4560
2.6905 9.0 20304 2.8298 0.4573
2.6697 10.0 22560 2.8169 0.4592
2.6509 11.0 24816 2.8080 0.4601
2.6494 12.0 27072 2.8020 0.4607
2.6384 13.0 29328 2.7958 0.4616
2.6297 14.0 31584 2.7939 0.4620
2.612 15.0 33840 2.7649 0.4653
2.5667 16.0 36096 2.7425 0.4686
2.5177 17.0 38352 2.7206 0.4714
2.4607 18.0 40608 2.6999 0.4746
2.397 19.0 42864 2.6865 0.4773
2.3241 19.9915 45100 2.6840 0.4787

Framework versions

  • Transformers 4.48.0
  • Pytorch 2.5.1
  • Datasets 3.2.0
  • Tokenizers 0.21.0