README.md

Merge of [SuperHOT-LoRA-prototype](https://huggingface.co/kaiokendev/SuperHOT-LoRA-prototype) and [llama-30b](https://huggingface.co/huggyllama/llama-30b)


Llama30B-SuperHOT-4bit-128g.safetensors Quantization:
```
CUDA_VISIBLE_DEVICES=0 python llama.py ausboss/Llama30B-SuperHOT c4 --wbits 4 --true-sequential --groupsize 128 --save_safetensors Llama30B-SuperHOT-4bit-128g.safetensors
```
Llama30B-SuperHOT-4bit.safetensors Quantization:
```
CUDA_VISIBLE_DEVICES=0 python llama.py ausboss/Llama30B-SuperHOT c4 --wbits 4 --true-sequential --save_safetensors Llama30B-SuperHOT-4bit.safetensors
```







# From the SuperHot Page:

## Prototypes for SuperHOT

No guarantees for output quality, simply uploading what I have so others can play around with it. Not even sure if the rank in cutoff-8192 is correct (think it should be 10 maybe.. can't remember)

All prototypes are extremely early epochs (sub 0.5)

## Model/Training
All trained with Flash Attention with conversation sequence lengths ranging from 8K to 16K tokens (No Alibi unless otherwise mentioned)

All trained on LLaMa 13B 4-bit (no groupsize)

(*Personally, I like the 8K cutoff version better, so I would say start with that one*)

## Data
A combination of various datasets and cleaned logs converted into datasets including but not limited to:
- Bluemoon Fanbased
- Roleplaying Guild
- Community-sourced outputs
- [Dan's PocketDoc/RUCAIBox-Story-Generation-Alpaca](https://huggingface.co/datasets/PocketDoc/RUCAIBox-Story-Generation-Alpaca)
- [IlyaGusev/gpt_roleplay_realm](https://huggingface.co/datasets/IlyaGusev/gpt_roleplay_realm)
- others

## Bias
SuperHOT is a fiction-focused model. No alignment has been performed on the training data. Be mindful that this model may output harmful, violent, or otherwise problematic content

## Format
Any format should work with such early checkpoints. However the training data is entirely in the following format:
```
---
mode: chat
characters:
    <char1 name>: <descriptive tags for char1>
    <char2 name>: <descriptive tags for char2>
summary: <summary of the story thus far or the purpose of the chat> (optional)
<any other miscellaneous data>
---
<chat history>
```

By "any other miscellaneous data", it means you should be able to put any additional metadata for the story or characters. I.e.,
```
...
locations:
    location1: <tags for location1>
inventory:
    item1: <tags for item1>
```

Again, format does not hold such a large weight on these early checkpoints. I have found success with the following setup for an RPG-like experience. Just play around with the format and see what works:
```
---
mode: rpg
characters:
     You: a new player
system: The system controls the RPG, handles character creation, world narration, and quest management. Also controls any NPCs and inventory tracking. Their first message provides a lengthy introduction for the player into the RPG world they are about to play in. After completing the character creation, the system will give a lengthy introduction into the world of ___. The first quest will follow right after
rpg setting: The world of ___
rpg rules: Any rules typical of RPG games, including typical items, battle stats, etc
---
```