v5

README.md

@@ -13,14 +13,14 @@ widget:
   - "Harry leapt forward, dodging Fingolfin's wildfire and reaching for the ring."
 ---
 
-GPT-2 (including tokenizer) trained **from scratch** on some of my favorite books (about 21M words in total).
+GPT-2 (including tokenizer) trained **from scratch** on some of my favorite books (about 31M words in total).
 
-It's only trained on an RTX 3090 for two hours, so don't take it seriously, just have fun!
+It's only trained on an RTX 3090 for three hours, so don't take it seriously, just have fun!
 
 - peak lr: 4e-4
 - global batch size: 32
 - weight decay: 0.01
-- training steps: 20k
+- training steps: 25k
 - warmup steps: 1k
 - lr decay: cosine
 
@@ -42,14 +42,20 @@ print(tokenizer.decode(output[0]))
 
 Sample output:
 
-> Daenerys kissed Gandalf, as the witcher hacked off Lord Voldemort's head with a brutal swing of Longclaw. The dragon’s eyes were blue, and red and green, were purple and yellow.
+> Daenerys kissed Gandalf, as the witcher hacked off Lord Voldemort's head with a brutal swing of Longclaw. “Do you know what a warrior like that will do?”
 >
-> ‘It is time,’ said Gandalf in a quiet voice. ‘For the sake of all the world we seek in peace and peace. For the lives of men of renown, for you should have. In these latter we are blessed. Yet you must come to your senses, which will have no way to seek. But now you will need the strength to become and the strength of our Enemy. When it is a signal that will not turn the blade of its foe and will it be enough, for our king will be merciful.’
+> “I am a knight of seven,” Geralt said. “And how is this knight?”
 >
-> The king laughed at that, and the others cried and screamed. ‘And the king?’ said Théoden.
+> “Prince, it is known,” replied the witcher, “I am a knight of Solamnia, not a Knight. A knight of Solamnia in the name of Reorx, with elven armies at the head of his knights and knights. You do not even remember my name, elf. It was a good call.”
 >
-> ‘I see.’ The Riders turned to look out over their shoulder.
+> “That name,” said the witcher, “what does it mean, elf?”
 >
-> It had begun. It seemed like nothing to Harry like the battle of Denethor. Now the silence within him was clear, as the men of the fortress could see in the battlements and walls below them and the gates. There is no way out and not one way out of the wall, and none comes of it.
+> “Some story,” said the old knight. “A good story from the Cataclysm.”
 >
-> In the midst of that great height of high valley on the summit, a tall man ran on. It was a wide
+> The witcher snorted and looked at the witcher. “That is how you feel, elf. You don't understand why the knights won't be executed for them.”
+>
+> “That's because the knight is a king,” Chireadan said finally, “the king's elder son. Because he is, the knights would also be honorable with the title of knighthood. But a warrior would be better suited to have the honor to win.”
+>
+> “But…”
+>
+> “It's true,” said the old knight, “that, isn't entirely what you want. The lance of your sword is one of the best

config.json

@@ -31,7 +31,7 @@
       "do_sample": true,
       "max_length": 256,
       "temperature": 1,
-      "top_p": 0.95
+      "top_p": 0.9
     }
   },
   "torch_dtype": "float32",

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:af0e9c35949ddd831d14ff8653be500e589351b8dbf31ae67d886d58f76b6419
+oid sha256:6c9a1f8ac848d03d09eca02882e8ef27faa7d7995260eafab1fe5bc94712b406
 size 510395581