|
--- |
|
license: mit |
|
tags: |
|
- gemma |
|
- gguf |
|
- llama.cpp |
|
- indic |
|
--- |
|
|
|
# GGUF for Indic-gemma-2b-finetuned-sft-Navarasa |
|
|
|
This model from [Telugu-LLM-Labs](https://huggingface.co/Telugu-LLM-Labs/Indic-gemma-2b-finetuned-sft-Navarasa) is based on google/gemma-2b and has been LoRA finetuned on 9 Indian languages and English instruction datasets |
|
|
|
|
|
```bash |
|
git clone https://huggingface.co/iAkashPaul/Indic-gemma-2b-finetuned-sft-Navarasa-GGUF # & cd into it, update paths accordingly |
|
|
|
# build llama.cpp for your hardware https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#build |
|
|
|
./main --file prompt.md --lora ./models/ggml-adapter-model.bin --lora-base ./models/indic-llm_Q8.gguf |
|
|
|
./main --file prompt.md -m ./models/merged_indic_llm_Q8.gguf -ngl 99 |
|
|
|
``` |
|
|
|
## Prompt template for Instruction adherence- |
|
|
|
Save this to a file(ex. prompt.md) & load it with the main executable. |
|
|
|
```markdown |
|
### Instruction: Translate following sentence to Kannada. |
|
|
|
### Input: This model is developed by Telugu LLM Labs |
|
|
|
## Response: |
|
``` |
|
|
|
## Performance |
|
|
|
* LORA+BASE (not merged) |
|
|
|
* ``` |
|
./server --lora ./models/ggml-adapter-model.bin --lora-base ./models/indic-llm_Q8.gguf -m ./models/indic-llm_Q8.gguf |
|
``` |
|
|
|
*  |
|
|
|
* Merged model |
|
|
|
* ``` |
|
./server -ngl 20 -m ./models/merged_indic_llm_Q8.gguf |
|
``` |
|
|
|
*  |
