Training in progress, step 4000

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:4c2e980c4af16947f754546d66a0e8d4aaf433fb42674c6a3a2de16abffb6811
+oid sha256:b18a05fe3e872cd4dda6f2ab42d9fe2011ebfa27c9fd51548851eb83eeedbd84
 size 1262817457

wandb/run-20220129_131141-h6nhqm30/files/output.log

@@ -3703,3 +3703,533 @@ Configuration saved in ./checkpoint-3500/config.json
 {'eval_loss': 0.38118353486061096, 'eval_wer': 0.4635139179886262, 'eval_runtime': 65.5254, 'eval_samples_per_second': 28.065, 'eval_steps_per_second': 0.443, 'epoch': 0.81}
 Model weights saved in ./checkpoint-3500/pytorch_model.bin
 Configuration saved in ./checkpoint-3500/preprocessor_config.json
+Configuration saved in ./preprocessor_config.json
+Deleting older checkpoint [checkpoint-2000] due to args.save_total_limit
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 17%|██████████████████████████                                                                                                                                  | 3600/21520 [2:58:00<8:11:02,  1.64s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 17%|██████████████████████████▊                                                                                                                                 | 3699/21520 [3:02:22<8:11:14,  1.65s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 18%|███████████████████████████▌                                                                                                                                | 3800/21520 [3:06:46<7:57:07,  1.62s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 18%|████████████████████████████▎                                                                                                                               | 3898/21520 [3:11:08<8:29:00,  1.73s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 19%|████████████████████████████▉                                                                                                                               | 4000/21520 [3:15:36<7:50:45,  1.61s/it]The following columns in the evaluation set  don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.
+***** Running Evaluation *****
+  Num examples = 1839
+  Batch size = 64
+{'loss': 1.2001, 'learning_rate': 6.128321678321677e-05, 'epoch': 0.93}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 29/29 [00:55<00:00,  1.79s/it]
+
+Configuration saved in ./checkpoint-4000/config.json
+Model weights saved in ./checkpoint-4000/pytorch_model.bin
+Configuration saved in ./checkpoint-4000/preprocessor_config.json

wandb/run-20220129_131141-h6nhqm30/run-h6nhqm30.wandb

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ece8eef3c351fb3c87046f4d884182362e5fa3efb32610da431511ce696b015b
-size 23797300
+oid sha256:f9739eb30c365b55a391ae6ef9ae79354e19dafe75345239ba3cc9ab6c8ad81e
+size 27510055