Technical issue/question
I'm running into an issue where the model will start spitting gibberish. The card in question is written to be a bit hyperactive, but there's something else at play given that it will write perfectly normal prose up until it just doesn't. It'll start what looks like a word at first then keep going pretty much for ever, i.e "goOOOooooEEEEEEGGGHHHHEEEEEE..." until I cut it off.
I'm using Q5 at 12k context and I've tried everything I can think of (which is admittedly not a lot as I'm fairly new to all this.) I did see something similar was listen in the known issues and I've fooled around with the rep pen and temp as well as tried using your Class 4 settings from the document you made but nothing seems to consistently nip this issue in the bud. Can anyone please offer any advice or assistance?
This model can go off the rails (side effect of pushing the creativity).
There are few ways to address this aside from Class 4 (and raising the specific settings noted for class 4 higher):
1 - Set a LIMIT on generation output. Then use continue for "next" chunk (s). This changes how the model behaves of subtle differences in this method, vs letting it go and go.
2 - Set a LIMIT, then see "Generational steering" -> this addresses both control and how to address "gibberish" issues and continue generation.
Other issues that can cause this:
- Flash attention on
- Context settings.
Also ; sometimes temp being TOO LOW can add to this. Using Dynamic temp (with at least 1 whole number of range) can also help here too.