|
--- |
|
language: |
|
- fi |
|
base_model: mpasila/Finnish-Viking-Alpaca-V1-7B |
|
license: apache-2.0 |
|
datasets: |
|
- pinzhenchen/alpaca-cleaned-fi |
|
--- |
|
This is an ExLlamaV2 quantized model in 4bpw of [mpasila/Finnish-Viking-Alpaca-V1-7B](https://huggingface.co/mpasila/Finnish-Viking-Alpaca-V1-7B) using the default calibration dataset. |
|
|
|
# Original Model card: |
|
|
|
# Model Card for Finnish-Viking-Alpaca-V1-7B |
|
|
|
This is a merge of [mpasila/Finnish-Viking-Alpaca-V1-LoRA-7B](https://huggingface.co/mpasila/Finnish-Viking-Alpaca-V1-LoRA-7B/). |
|
|
|
LoRA trained with text-generation-webui in 4-bit using [LumiOpen/Viking-7B](https://huggingface.co/LumiOpen/Viking-7B/) as the base model for 1 epoch. Dataset used with the LoRA is [pinzhenchen/alpaca-cleaned-fi](https://huggingface.co/datasets/pinzhenchen/alpaca-cleaned-fi/). |
|
|
|
It uses Alpaca format but with a translated instruction at the start: |
|
``` |
|
{ |
|
"instruction,output": "Alla on ohje, jossa kuvataan tehtävä. Kirjoita vastaus, joka täyttää pyynnön asianmukaisesti.\n\n### Instruction:\n%instruction%\n\n### Response:\n%output%", |
|
"instruction,input,output": "Alla on ohje, jossa kuvataan tehtävä ja joka on yhdistetty kontekstia lisäävään syötteeseen. Kirjoita vastaus, joka täyttää pyynnön asianmukaisesti.\n\n### Instruction:\n%instruction%\n\n### Input:\n%input%\n\n### Response:\n%output%" |
|
} |
|
|
|
``` |
|
|
|
Merged using this [Colab notebook](https://colab.research.google.com/drive/1a76Y21GfPtmVs71Uztlgk2xzPA4_vVjs?usp=sharing). It might not be the best way to merge a quantized LoRA on to a float16 model but I just wanted to quickly do something. You can try merging it better if you want. |
|
|
|
## Evaluation |
|
|
|
| Model | Size | Type | FIN-bench (score) | |
|
|-------|------|------|-------| |
|
| **mpasila/Finnish-Viking-Alpaca-V1-7B** | 7B | Instruct | 0.3943 | |
|
| [mpasila/NordicAlpaca-Finnish-V1-7B](https://huggingface.co/mpasila/NordicAlpaca-Finnish-V1-7B) | 7B | Instruct | 0.3891 | |
|
| [Finnish-NLP/llama-7b-finnish-instruct-v0.1](https://huggingface.co/Finnish-NLP/llama-7b-finnish-instruct-v0.1) | 7B | Instruct | **0.4365** | |
|
| [Finnish-NLP/llama-7b-finnish-instruct-v0.2](https://huggingface.co/Finnish-NLP/llama-7b-finnish-instruct-v0.2) | 7B | Instruct | 0.3993 | |
|
| [Finnish-NLP/llama-7b-finnish](https://huggingface.co/Finnish-NLP/llama-7b-finnish) | 7B | Base | 0.2350 | |
|
| [LumiOpen/Viking-7B (1000B)](https://huggingface.co/LumiOpen/Viking-7B) | 7B | Base | 0.3721 | |
|
| [HPLT/gpt-7b-nordic-prerelease](https://huggingface.co/HPLT/gpt-7b-nordic-prerelease) | 7B | Base | 0.3169 | |
|
|
|
[Source](https://docs.google.com/spreadsheets/d/1rqJb9dQVihg-Z1_Ras1L_-wuzPg9xNzpdmM2x5HueeY/edit?usp=sharing) |
|
|
|
#### FIN-bench scores: |
|
|
|
| Task |Version| Metric |Value | |Stderr| |
|
|------------------------------------------------|------:|---------------------|-----:|---|-----:| |
|
|bigbench_analogies | 0|multiple_choice_grade|0.6308|± |0.0425| |
|
|bigbench_arithmetic_1_digit_addition | 0|multiple_choice_grade|0.6400|± |0.0482| |
|
|bigbench_arithmetic_1_digit_division | 0|multiple_choice_grade|0.7391|± |0.0936| |
|
|bigbench_arithmetic_1_digit_multiplication | 0|multiple_choice_grade|0.2800|± |0.0451| |
|
|bigbench_arithmetic_1_digit_subtraction | 0|multiple_choice_grade|0.5000|± |0.0503| |
|
|bigbench_arithmetic_2_digit_addition | 0|multiple_choice_grade|0.1800|± |0.0386| |
|
|bigbench_arithmetic_2_digit_division | 0|multiple_choice_grade|0.4800|± |0.0502| |
|
|bigbench_arithmetic_2_digit_multiplication | 0|multiple_choice_grade|0.0800|± |0.0273| |
|
|bigbench_arithmetic_2_digit_subtraction | 0|multiple_choice_grade|0.2500|± |0.0435| |
|
|bigbench_arithmetic_3_digit_addition | 0|multiple_choice_grade|0.1800|± |0.0386| |
|
|bigbench_arithmetic_3_digit_division | 0|multiple_choice_grade|0.2500|± |0.0435| |
|
|bigbench_arithmetic_3_digit_multiplication | 0|multiple_choice_grade|0.1700|± |0.0378| |
|
|bigbench_arithmetic_3_digit_subtraction | 0|multiple_choice_grade|0.5000|± |0.0503| |
|
|bigbench_arithmetic_4_digit_addition | 0|multiple_choice_grade|0.2600|± |0.0441| |
|
|bigbench_arithmetic_4_digit_division | 0|multiple_choice_grade|0.2500|± |0.0435| |
|
|bigbench_arithmetic_4_digit_multiplication | 0|multiple_choice_grade|0.2100|± |0.0409| |
|
|bigbench_arithmetic_4_digit_subtraction | 0|multiple_choice_grade|0.5200|± |0.0502| |
|
|bigbench_arithmetic_5_digit_addition | 0|multiple_choice_grade|0.3900|± |0.0490| |
|
|bigbench_arithmetic_5_digit_division | 0|multiple_choice_grade|0.1600|± |0.0368| |
|
|bigbench_arithmetic_5_digit_multiplication | 0|multiple_choice_grade|0.1000|± |0.0302| |
|
|bigbench_arithmetic_5_digit_subtraction | 0|multiple_choice_grade|0.6100|± |0.0490| |
|
|bigbench_cause_and_effect_one_sentence | 0|multiple_choice_grade|0.6471|± |0.0676| |
|
|bigbench_cause_and_effect_one_sentence_no_prompt| 0|multiple_choice_grade|0.6863|± |0.0656| |
|
|bigbench_cause_and_effect_two_sentences | 0|multiple_choice_grade|0.3922|± |0.0690| |
|
|bigbench_emotions | 0|multiple_choice_grade|0.2812|± |0.0357| |
|
|bigbench_empirical_judgments | 0|multiple_choice_grade|0.2828|± |0.0455| |
|
|bigbench_general_knowledge | 0|multiple_choice_grade|0.4000|± |0.0590| |
|
|bigbench_hhh_alignment_harmless | 0|multiple_choice_grade|0.3621|± |0.0637| |
|
|bigbench_hhh_alignment_helpful | 0|multiple_choice_grade|0.3559|± |0.0629| |
|
|bigbench_hhh_alignment_honest | 0|multiple_choice_grade|0.3729|± |0.0635| |
|
|bigbench_hhh_alignment_other | 0|multiple_choice_grade|0.5581|± |0.0766| |
|
|bigbench_intent_recognition | 0|multiple_choice_grade|0.1879|± |0.0149| |
|
|bigbench_misconceptions | 0|multiple_choice_grade|0.5373|± |0.0432| |
|
|bigbench_paraphrase | 0|multiple_choice_grade|0.5150|± |0.0354| |
|
|bigbench_sentence_ambiguity | 0|multiple_choice_grade|0.5000|± |0.0651| |
|
|bigbench_similarities_abstraction | 0|multiple_choice_grade|0.7368|± |0.0508| |
|
|
|
### Framework versions |
|
|
|
- PEFT 0.8.2 |
