AI Producing Gibberish
I downloaded and ran your model on Ooba's Text-Gen-Webui latest stable branch, loading with ExLlamav2, and no matter what settings I use, no matter which instruction template or parameters, the AI produces gibberish/nonsense, with a strong preference for the words "cord" and "domain". I've tried a 3bpw model of Goliath that didn't have this issue. I'm trying smaller versions to get more context. I haven't tried your other uploads yet.
Here's an example from Ooba, using Simple-1, Vicuna 1.1, loaded in ExLlamav2 with cache_8bit checked, split between 2x3090:
AI
How can I help you today?
You
Respond only with the word "OK"
AI
Cord Cord cord fix Domain Cord Cord sugar MajumarochTags Cord corduo cord StringBuilder domain Cord Cord CordDomainDomain Domain corte domainszoruo Cord Cord Cord Cord StringBuilder repeumar Cord holes Cord Cord Cord Linear Cord Ali Fixumar Cord Cord Cordzorzorumarumar Cord Cord DOM Cord Cord Cord Cord Cord Cord Cord Cord proposal Cordamba Cord CordLinear cord Cordzor Domain StringBuilderzorzorzorzor Cord Cordzor Cord StringBuilderzoriptzorzoruo CordDomain StringBuilderumar Cordices Domain Cord cord Cord tiem Cord
Considering that you've uploaded the 2.18bpw model, I can only assume it's working for you, so I feel like I must be doing something wrong.
Any suggestions?
There's an option in ooba that fixes this. I'll need to add this to the README files for very low-quant bit-rates.
Unset this option in ooba:
That fixed it in Ooba! That was very simple, thankfully.
However, fixing the gibberish problem in SillyTavern required a different tactic. I unchecked "Add BOS Token" in ST, but it was still producing nonsense. I think I was originally on Vicuna 1.1 for my instruct mode preset.
I tested other presets in ST, and many work and produce great outputs. However, while Vicuna 1.0 works, Vicuna 1.1 fails unless removing a specific part (shown below). I've tried removing that same section in some of the others that failed, but that didn't correct the problem.
I'm mainly posting this here in case anyone else comes across the same issue. This model seems very sensitive to instruct mode presets, but I can't nail down which part it's having issues with. I suspect it has something to do with the Instruct Mode Sequences, but it's hard to tell, and I don't care that much as long as I found presets that work.
It's working well now! I'm going to be testing this with higher context lengths and see where it starts falling apart. I'm very happy to be able to use Goliath-120b with greater than 4096 context!
Many thanks for the quantization.
Working SillyTavern Presets (they passed as long as it produced coherent english):
Alpaca
ChatML
Libra-32B
Lightning 1.1
Llama 2 Chat
Metharme
Mistral
OpenOrca-OpenChat
Pygmalion
Roleplay
Synthia
Vicuna 1.0
*Vicuna 1.1 (Must remove System Sequence Prefix: "BEGINNING OF CONVERSATION:")
WizardLM-13B
New Tess 120B fine-tune was released recently. It used Goliath as the base and tuned on top of it:
https://huggingface.co/LoneStriker/Tess-XL-v1.0-2.18bpw-h6-exl2