Saving weights and log at step 1000000

README.md

@@ -27,13 +27,12 @@ Tokenizer:
 Training details:
 
 * Training started on step 360K (bs 16) ppl 21 of earlier model trained with Adam optimizer.
-* Training at step 880K of 2M (38%) ppl 15,3
+* Training at step 1000K of 2082K (48%) ppl 15,2
 * Block size: 512
 * Optimizer: adafactor
 * Learning rate: 3.3e-5
 * Batch size: 32
 * Warmup steps: 5000
-* Weight decay: 0.01
 
 Work in progress. Dec 2021-Jan2022
 

flax_model.msgpack

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:cfc08d5c4120eff8f546db9675affda9bdaba5b8dfe2e81f9257a499e1b3b53c
+oid sha256:2d77af2bdb5c6b7fbeae905633a8986e9c97140208268cd3a34909ebf30b1536
 size 3096134690

opt_state.msgpack

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:384b781970c22b5c572a23c3bf0f384c614dd13a0815f125b896de9735228377
+oid sha256:751daf3e7e3c5efa79ffa8a1cece06bd7cdb4fc23d141b13c64e75b61c0d4f48
 size 5611008

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6c85a006e388f0880cef392c8d434653cc69d3377ac9f44f6fa8ca6ba6bb2799
+oid sha256:8461ae6051ce46f98cdf810433154389d1871441325eda9f8d4380e0e688be54
 size 3134045897

runs/events.out.tfevents.1641055391.t1v-n-f9cfcc28-w-0.112189.0.v2

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8e01abfe0cb320dd193e448be51c5c1c7473b01caef47ea4246cf07d8b5e9f24
-size 131559285
+oid sha256:391e0abb9a67782191b2beb53ef7eef37caca761eec84e5f63d2000ee7a76404
+size 149388427

training_state.json

@@ -1 +1 @@
-{"step": 880001}
+{"step": 1000001}