lapp0
/

distily_bench_gpt2_optim_extended2

+---
+base_model: gpt2
+library_name: distily
+license: mit
+tags:
+- generated_from_trainer
+model-index:
+- name: distily_bench_gpt2_optim_extended2
+  results: []
+---
+# distily_bench_gpt2_optim_extended2
+This student model is distilled from the teacher model [gpt2](https://huggingface.co/gpt2) using the dataset (unspecified).
+The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
+It achieves the following results on the evaluation set:
+- eval_enwikippl: 859.1765
+- eval_frwikippl: 4914.2627
+- eval_zhwikippl: 7375.7471
+- eval_loss: 6893.8027
+- eval_runtime: 65.3454
+- eval_samples_per_second: 45.91
+- eval_steps_per_second: 11.477
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment.
+## Model description
+More information needed
+## Intended uses & limitations
+More information needed
+## Training and evaluation data
+More information needed
+-->
+## Training procedure
+### Training hyperparameters
+The following hyperparameters were used during training:
+- distillation_objective: 'legacy'
+- loss_fn: kl
+- train_embeddings: True
+- learning_rate: 4e-05
+- train_batch_size: 8
+- eval_batch_size: 4
+- seed: 42
+- gradient_accumulation_steps: 2
+- total_train_batch_size: 16
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: constant
+- num_epochs: 1.0
+### Resource Usage
+Peak GPU Memory: 8.3331 GB
+### Eval-Phase Metrics
+| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| **teacher eval** |  | 30.2385 | 57.2728 |  |  |  |  | 18.1772 |
+| 0 | 0 | 59513.7266 | 59655.5039 | 330435.25 | 63.4949 | 47.248 | 11.812 | 53382.8789 |
+| 500 | 0.0269 | 2863.2634 | 11508.8330 | 12259.9258 | 63.8689 | 46.971 | 11.743 | 38858.9141 |
+| 1000 | 0.0539 | 2136.9172 | 7522.9854 | 10143.5098 | 63.9165 | 46.936 | 11.734 | 16435.1426 |
+| 1500 | 0.0808 | 1930.7159 | 7313.2690 | 9805.8242 | 64.0662 | 46.827 | 11.707 | 14467.3398 |
+| 2000 | 0.1077 | 1691.7502 | 6743.2197 | 9288.6826 | 64.7936 | 46.301 | 11.575 | 13923.3652 |
+| 2500 | 0.1347 | 1564.1060 | 6471.2065 | 8980.0957 | 65.967 | 45.477 | 11.369 | 13381.9658 |
+| 3000 | 0.1616 | 1452.3389 | 6264.2476 | 8701.7598 | 65.1396 | 46.055 | 11.514 | 14317.4297 |
+| 3500 | 0.1886 | 1346.0464 | 6201.8452 | 8515.4990 | 64.711 | 46.36 | 11.59 | 14058.8223 |
+| 4000 | 0.2155 | 1293.1278 | 5995.4678 | 8215.7871 | 64.881 | 46.238 | 11.56 | 14290.6924 |
+| 4500 | 0.2424 | 1188.8944 | 5653.4883 | 7998.4854 | 64.8042 | 46.293 | 11.573 | 12600.6602 |
+| 5000 | 0.2694 | 1150.8783 | 5734.7847 | 7816.5864 | 65.0575 | 46.113 | 11.528 | 11801.7627 |
+| 5500 | 0.2963 | 1106.1051 | 5579.8325 | 7761.1094 | 64.8301 | 46.275 | 11.569 | 10491.8916 |
+| 6000 | 0.3232 | 1062.4747 | 5325.9385 | 7568.8638 | 64.8331 | 46.273 | 11.568 | 9862.1807 |
+| 6500 | 0.3502 | 1016.8871 | 5219.4351 | 7424.0107 | 65.1855 | 46.023 | 11.506 | 8266.7178 |
+| 7000 | 0.3771 | 990.4482 | 5391.6797 | 7382.8267 | 64.9653 | 46.179 | 11.545 | 8352.1592 |
+| 7500 | 0.4040 | 957.9040 | 5171.6309 | 7296.6401 | 64.9498 | 46.19 | 11.547 | 7235.2671 |
+| 8000 | 0.4310 | 936.6096 | 5295.7979 | 7155.7334 | 64.9012 | 46.224 | 11.556 | 8302.1201 |
+| 8500 | 0.4579 | 893.5040 | 5091.3115 | 7057.3013 | 65.0288 | 46.133 | 11.533 | 7185.6812 |
+| 9000 | 0.4848 | 894.5970 | 5188.4321 | 6994.9229 | 65.1952 | 46.016 | 11.504 | 8320.9854 |
+| 9500 | 0.5118 | 859.1765 | 4914.2627 | 6893.8027 | 65.3454 | 45.91 | 11.477 | 7375.7471 |
+| 10000 | 0.5387 | 844.4594 | 5006.5923 | 6830.0801 | 64.9208 | 46.21 | 11.553 | 7772.1235 |
+| 10500 | 0.5657 | 825.0941 | 4731.5112 | 6761.4507 | 64.7216 | 46.352 | 11.588 | 6593.5493 |
+| 11000 | 0.5926 | 805.9855 | 4779.2881 | 6747.7651 | 65.2107 | 46.005 | 11.501 | 6463.2256 |
+| 11500 | 0.6195 | 798.3549 | 4823.9761 | 6676.7681 | 65.0141 | 46.144 | 11.536 | 9802.4414 |
+| 12000 | 0.6465 | 791.6104 | 4696.2798 | 6579.1147 | 64.8969 | 46.227 | 11.557 | 7084.2090 |
+| 12500 | 0.6734 | 761.6416 | 4785.1895 | 6548.4907 | 64.8647 | 46.25 | 11.563 | 9604.8457 |
+| 13000 | 0.7003 | 761.1536 | 4701.0835 | 6501.8882 | 65.0795 | 46.097 | 11.524 | 6464.0889 |
+| 13500 | 0.7273 | 734.9013 | 4585.2031 | 6514.5386 | 65.3072 | 45.937 | 11.484 | 6561.9272 |
+| 14000 | 0.7542 | 740.7595 | 4580.6797 | 6382.9121 | 65.4037 | 45.869 | 11.467 | 6729.2021 |
+| 14500 | 0.7811 | 711.6922 | 4399.3062 | 6421.3013 | 64.9755 | 46.171 | 11.543 | 10408.1582 |
+| 15000 | 0.8081 | 699.6238 | 4419.9839 | 6434.4214 | 64.9281 | 46.205 | 11.551 | 6650.1450 |
+| 15500 | 0.8350 | 689.9260 | 4160.6733 | 6344.1709 | 65.0911 | 46.089 | 11.522 | 5445.5518 |
+| 16000 | 0.8620 | 690.0198 | 4297.5234 | 6258.8799 | 64.8144 | 46.286 | 11.572 | 4706.0220 |
+| 16500 | 0.8889 | 691.7233 | 4264.9258 | 6328.2451 | 65.5974 | 45.734 | 11.433 | 5371.5254 |
+| 17000 | 0.9158 | 665.1479 | 4280.1362 | 6238.0371 | 64.909 | 46.219 | 11.555 | 7401.3955 |
+| 17500 | 0.9428 | 668.0211 | 4168.7480 | 6233.2695 | 64.7669 | 46.32 | 11.58 | 6142.2017 |
+| 18000 | 0.9697 | 653.9958 | 4099.5210 | 6188.6719 | 64.1993 | 46.729 | 11.682 | 5851.9600 |
+| 18500 | 0.9966 | 644.6927 | 3924.1768 | 6175.1572 | 64.4251 | 46.566 | 11.641 | 5162.6572 |
+| 18562 | 1.0000 | 649.2895 | 3895.5068 | 6124.0107 | 64.4839 | 46.523 | 11.631 | 4821.7959 |
+### Framework versions
+- Distily 0.2.0
+- Transformers 4.44.0
+- Pytorch 2.3.0
+- Datasets 2.20.0

generation_config.json ADDED Viewed

	@@ -0,0 +1,6 @@

+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.44.0"
+}