Will there be quantized GGUF for the instruct model as well?

#1
by rsolva - opened

Having briefly tested the warm model, it is apparent that it is a bit all over the place (but very funny!). It would be great if the instruct model was released as GGUF as well.

Norwegian Large Language Models org

I believe Lucas is also working on that :) We found the default hyperparameters of llama-cpp a bit strange, it was better to turn off the repetition penalty (set it to 1.0) and to set the temperature to a lower value; the model behaved very chaotically otherwise. I'm sure there's more that can be done about these (and other) hyperparameters, they influence the outputs more than I'd like.

First: Thank you very much for providing gguf-files for the instruct model. It made the life for us amateurs a little more easy. 🤩

I am a total beginner and are experimenting a little with the instruct-model on Ollama. Does anyone have som tips for parameter setting that works well?

Currently my Ollama-modelfile looks like this:

TEMPLATE """
{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>{{ end }}
{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant
"""
SYSTEM """Du er en vennlig assistent som skal svare på spørsmål? Svar kort og konkret på spørsmål"""
PARAMETER num_ctx 4096
PARAMETER stop "<|im_end|>"
PARAMETER temperature 1

The responses are ok-ish, but I wonder if the settings/modelfile can be improved.

Here is my Modelfile using the suggestions from @davda54 and the README. I made the following changes to the Modelfile:

  • a space before system, user and assistant (as suggested in the README).
  • set the repeat penalty to 1.0
  • the temperature to 0.3
  • added extra stop parameters
TEMPLATE """
{{ if .System }}<|im_start|> system
{{ .System }}<|im_end|>{{ end }}
{{ if .Prompt }}<|im_start|> user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|> assistant
"""
SYSTEM """Du er Kari Nordmann, en jovial og hjelpsom assistent som svarer kort og konsist på spørsmål."""
PARAMETER num_ctx 4096
PARAMETER temperature 0.3
PARAMETER repeat_penalty 1.0
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|im_start|>"
PARAMETER stop " user"
PARAMETER stop " assistant"

It behaves quite well, but it is still a bit to verbose for my taste, so I will experiment further with different system messages.

Norwegian Large Language Models org

Thanks for the response @rsolva ! Verbosity is definitely a problem, we tried to augment the open instruction datasets by step-by-step reasoning, detailed descriptions, etc. and apparently we overdid it :) It's on our list of things to improve in the next release. By the way, if you noticed some other recurring unwanted patterns in the outputs, please let us know your feedback!

davda54 changed discussion status to closed

Sign up or log in to comment