Update README.md

README.md

@@ -1,58 +1,52 @@
 ---
-license: other
-base_model: qnguyen3/Mixtral-4x400M
-tags:
-- llama-factory
-- generated_from_trainer
-model-index:
-- name: mixtral-4x-400M-pt
-  results: []
+license: apache-2.0
+widget:
+- text: My name is El Microondas the Wise, and
+  example_title: El Microondas
+- text: Kennesaw State University is a public
+  example_title: Kennesaw State University
+- text: Bungie Studios is an American video game developer. They are most famous for
+    developing the award winning Halo series of video games. They also made Destiny.
+    The studio was founded
+  example_title: Bungie
+- text: The Mona Lisa is a world-renowned painting created by
+  example_title: Mona Lisa
+- text: The Harry Potter series, written by J.K. Rowling, begins with the book titled
+  example_title: Harry Potter Series
+- text: 'Question: I have cities, but no houses. I have mountains, but no trees. I
+    have water, but no fish. What am I?
+
+    Answer:'
+  example_title: Riddle
+- text: The process of photosynthesis involves the conversion of
+  example_title: Photosynthesis
+- text: Jane went to the store to buy some groceries. She picked up apples, oranges,
+    and a loaf of bread. When she got home, she realized she forgot
+  example_title: Story Continuation
+- text: 'Problem 2: If a train leaves Station A at 9:00 AM and travels at 60 mph,
+    and another train leaves Station B at 10:00 AM and travels at 80 mph, when will
+    they meet if the distance between the stations is 300 miles?
+
+    To determine'
+  example_title: Math Problem
+- text: In the context of computer programming, an algorithm is
+  example_title: Algorithm Definition
 ---
-
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
-# mixtral-4x-400M-pt
-
-This model is a fine-tuned version of [qnguyen3/Mixtral-4x400M](https://huggingface.co/qnguyen3/Mixtral-4x400M) on the thevault_function_xsmall, the redpajama_v2_small, the tiny_strange_textbooks, the tiny_textbooks, the code_textbook, the the_stack_smol_xl_cleaned, the refinedweb_1m_medium, the minipile, the goodwiki, the wikipedia_vi, the mathpile_arxiv_medium, the mathpile_stackexchange, the mathpile_proofpile, the mathpile_wikipedia, the thevault_class_xsmall, the tiny_stories_envi, the pretrain_instruct_1, the pretrain_instruct_2 and the pretrain_instruct_code datasets.
-
-## Model description
-
-More information needed
-
-## Intended uses & limitations
-
-More information needed
-
-## Training and evaluation data
-
-More information needed
-
-## Training procedure
-
-### Training hyperparameters
-
-The following hyperparameters were used during training:
-- learning_rate: 0.0003
-- train_batch_size: 64
-- eval_batch_size: 8
-- seed: 42
-- distributed_type: multi-GPU
-- num_devices: 4
-- gradient_accumulation_steps: 4
-- total_train_batch_size: 1024
-- total_eval_batch_size: 32
-- optimizer: Adam with betas=(0.9,0.95) and epsilon=1e-05
-- lr_scheduler_type: cosine
-- num_epochs: 3.0
-
-### Training results
-
-
-
-### Framework versions
-
-- Transformers 4.37.1
-- Pytorch 2.1.2+cu121
-- Datasets 2.16.1
-- Tokenizers 0.15.1
+# Mixsmol-4x400M-v0.1 by Ontocord
+This is the third checkpoint (Epoch 3) of Mixsmol-4x400M-v0.1
+Note that this is an experimental in data mixing. Therefore, we only trained the model on 50B tokens (95% English and 5% Vietnamese) to test the following:
+- Reasoining capabilities through high-quality synthetic textbooks data pretraining 
+- Crosslingual understanding through machine translation and multilingual + multiple tasks pretraining
+
+After verifying our hypothesis with this run, we will schedule a second run on bigger data and compute for it to achieve its maximum capability.
+
+## Data
+- Synthetic Textbooks: 8M samples
+- RefinedWeb: 1M samples
+- RedPajama-v2: 500K samples
+- MathPile: Everything
+- ThePile: MiniPile Subset
+- GoodWiki
+- The Stack Smol XL
+- The Vault: train_small split
+- Instruction Pretraining: 250k samples