Commit
•
d1c9a9c
1
Parent(s):
2eafd90
Update README.md
Browse files
README.md
CHANGED
|
@@ -24,6 +24,13 @@ license: apache-2.0
|
|
| 24 |
|
| 25 |
-----
|
| 26 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 27 |
<p align="center"><font size="5"> <b>Training Methodology</b> </font></p>
|
| 28 |
|
| 29 |
<p align="center">The model was trained on a variation of TheSkullery/NeMoria-21b, made by finetuning two NeMo models, one for each added “core” (set of repeated layers). One model was overfit to RP data, and the other was overfit to factual data and input analysis. Then the base NeMo was stitched together with the two models, so the repeated portion was one vanilla NeMo core, then the “Virgin” core, then the “Slut” core, a series of layers I like to call the “Whore/Madonna complex”. Now in place, the entire model was continually pretrained on ~1.5 GB private dataset of domain data mixed with stabilizing agents. The Virgin and Slut cores were then each instruct trained on their domains with all other layers frozen, one at a time. Finally, the entire model was SFT’d and DPO’d.
|
|
|
|
| 24 |
|
| 25 |
-----
|
| 26 |
|
| 27 |
+
<p align="center"><font size="5"> <b>Compantionship Chat Example - 8-bit GGUF</b> </font></p>
|
| 28 |
+
<p align="center"><font size="3"> <b>Using Goldie, one of the top characters on Chub.ai</b> </font></p>
|
| 29 |
+
<p align="center"><font size="3"> <b>(basic Ooba preset and system prompt)</b> </font></p>
|
| 30 |
+
<p align="center"><img src="https://cdn-uploads.huggingface.co/production/uploads/633a809fa4a8f33508dce32c/wNdL-rlbnwZmNBMoH9K8k.png"/></p>
|
| 31 |
+
|
| 32 |
+
-----
|
| 33 |
+
|
| 34 |
<p align="center"><font size="5"> <b>Training Methodology</b> </font></p>
|
| 35 |
|
| 36 |
<p align="center">The model was trained on a variation of TheSkullery/NeMoria-21b, made by finetuning two NeMo models, one for each added “core” (set of repeated layers). One model was overfit to RP data, and the other was overfit to factual data and input analysis. Then the base NeMo was stitched together with the two models, so the repeated portion was one vanilla NeMo core, then the “Virgin” core, then the “Slut” core, a series of layers I like to call the “Whore/Madonna complex”. Now in place, the entire model was continually pretrained on ~1.5 GB private dataset of domain data mixed with stabilizing agents. The Virgin and Slut cores were then each instruct trained on their domains with all other layers frozen, one at a time. Finally, the entire model was SFT’d and DPO’d.
|