---
license: llama3
---

This is an experimental 2x8B moe with random gates, using the following 2 models

- Hermes-2-Theta-l3-8B by Nous Research https://huggingface.co/NousResearch/Hermes-2-Theta-Llama-3-8B

- llama-3-cat-8B-instruct-V1 by TheSkullery https://huggingface.co/TheSkullery/llama-3-cat-8b-instruct-v1


***Important*** 

Make sure to add `</s>` a stop sequence as it uses llama-3-cat-8B-instruct-V1 as the base model.

Update:

Due to request i decided to add the rest of the quants. Enjoy


Mergekit recipe of the model if too lazy to check the files:

```
base_model: TheSkullery/llama-3-cat-8b-instruct-v1
gate_mode: random
dtype: bfloat16
experts_per_token: 2
experts:
 - source_model: TheSkullery/llama-3-cat-8b-instruct-v1
   positive_prompts:
    - " "
 - source_model: NousResearch/Hermes-2-Theta-Llama-3-8B
   positive_prompts:
    - " "
```