Create README.md

README.md

@@ -0,0 +1,33 @@
+---
+license: apache-2.0
+datasets:
+- augmxnt/ultra-orca-boros-en-ja-v1
+language:
+- ja
+- en
+tags:
+- jamba
+- axolotl
+---
+Over the weekend after a failed initial run I got excited by Pete's success [Jamba Tuning](https://huggingface.co/lightblue/Jamba-v0.1-chat-multilingual) and decided to throw a little compute on a similar-sized dataset (the main [shisa-v1 bilingual tuning set](https://huggingface.co/datasets/augmxnt/ultra-orca-boros-en-ja-v1)).
+
+Like my [initial runs](https://api.wandb.ai/links/augmxnt/h4mc4dd5), [training graphs look fine](https://wandb.ai/augmxnt/shisa-v2/runs/o830e1kw), but the results were less than spectacular.
+
+Here are the JA MT-Bench evals for the 2416 checkpoint (eval/loss plateau) and the 4228 (3 epoch) tune:
+
+```
+shisa-jamba-v1-checkpoint-2416     2.491525
+shisa-jamba-v1-checkpoint-4228     2.508475
+```
+
+You can view the answers in the repo (lots of repetitions and nonsense) and compare to [proper JA MT-Bench scores](https://github.com/AUGMXNT/shisa/wiki/Evals-:-JA-MT%E2%80%90Bench) from my testing.
+
+While an "unsucessful" experiment, it was still worth the practice, although I got a little excited and should have gone w/ my more typical lighter testing obviously.
+
+This kicks off official `shisa-v2` base model evaluation. I was a bit hesitant about throwing this model out there (since it's useless as an artifact), but since I've actually made the in-process code available while working on it, I'll share this as well just in case (and to do this writeup).
+
+Here is the current full code/steps for Axolotl training and eval (modified llm-judge inferencing code):
+- https://github.com/shisa-ai/shisa-v2/tree/main/_base-evals/jamba/axolotl
+- https://github.com/shisa-ai/shisa-v2/tree/main/_base-evals/jamba/eval
+
+Thanks to Pete for the useful [initial report](https://wandb.ai/peterd/axolotl/reports/Jamba-chat-finetuning-Airoboros-and-ShareGPT---Vmlldzo3MzUwNTc1?accessToken=syabbmhblmwslnizuwws62tig3f0op75zh2jm2owbranaziq7thbgizl78nnowgc) and the [axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) team for their [fast integration of Jamba](https://github.com/OpenAccess-AI-Collective/axolotl/issues/1449) (way better than my [raw tune code](https://github.com/shisa-ai/shisa-v2/tree/main/_base-evals/jamba)).