Training in progress, step 6500

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:a4f0d0ca6d2aa65cbbdfd8b790b7678da2cdb3f8a819ed9d6d9bf22856681780
+oid sha256:0513c7a1b35399e6652884442e4f940cffb0c3bc1af3aeca10a7e91208aa67e3
 size 1262817457

wandb/run-20220129_131141-h6nhqm30/files/output.log

@@ -6356,3 +6356,531 @@ Configuration saved in ./checkpoint-6000/config.json
 {'eval_loss': 0.318962037563324, 'eval_wer': 0.4240646513020054, 'eval_runtime': 65.8584, 'eval_samples_per_second': 27.924, 'eval_steps_per_second': 0.44, 'epoch': 1.39}
 Model weights saved in ./checkpoint-6000/pytorch_model.bin
 Configuration saved in ./checkpoint-6000/preprocessor_config.json
+Configuration saved in ./preprocessor_config.json
+Deleting older checkpoint [checkpoint-4500] due to args.save_total_limit
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 28%|████████████████████████████████████████████▏                                                                                                               | 6099/21520 [5:02:30<8:27:43,  1.98s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 29%|████████████████████████████████████████████▉                                                                                                               | 6199/21520 [5:06:54<8:34:39,  2.02s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 29%|█████████████████████████████████████████████▋                                                                                                              | 6299/21520 [5:11:18<8:31:19,  2.02s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 30%|██████████████████████████████████████████████▍                                                                                                             | 6400/21520 [5:15:44<8:25:03,  2.00s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 30%|███████████████████████████████████████████████                                                                                                             | 6500/21520 [5:20:08<8:19:08,  1.99s/it]The following columns in the evaluation set  don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.
+***** Running Evaluation *****
+  Num examples = 1839
+  Batch size = 64
+{'loss': 1.1201, 'learning_rate': 5.2548951048951044e-05, 'epoch': 1.51}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 29/29 [00:57<00:00,  1.83s/it]
+
+Configuration saved in ./checkpoint-6500/config.json
+Model weights saved in ./checkpoint-6500/pytorch_model.bin
+Configuration saved in ./checkpoint-6500/preprocessor_config.json

wandb/run-20220129_131141-h6nhqm30/run-h6nhqm30.wandb

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:92f31e42a7660a3050880269092e88eebacb7a8c82b0357c649f71c39f9308dc
-size 42654383
+oid sha256:34f2bb24662d3fcb0bb0eff6e4d9ec40beceb97465ce839d824e1956703ac4f1
+size 46505748