|
--- |
|
license: apache-2.0 |
|
language: |
|
- en |
|
- ru |
|
--- |
|
|
|
## Description |
|
This is an instruction following model (based on Mistral v0.1 Base) optimized for Russian language. |
|
It was trained using [kolibrify](https://github.com/oKatanaaa/kolibrify) on a multitude of instruction datasets. |
|
|
|
The model uses ChatML template. It was trained to be sensitive to the system prompt, experiment with it. |
|
|
|
Currently in pre-alpha, later releases will include more details regarding training procedure and data mix. |
|
|
|
> [!NOTE] |
|
> This model is an improved version of older kolibri-mistral-0427. |
|
|
|
## Instruction following evals |
|
The model was tested using the following benchmarks: |
|
- [ruIFEval](https://github.com/NLP-Core-Team/ruIFEval) |
|
- [ifeval](https://github.com/google-research/google-research/tree/master/instruction_following_eval) |
|
|
|
| Eval name |Strict Value| Loose Value |
|
|---------------------------------|----|----| |
|
|Avg. |*53.81*|*56.57*| |
|
|ifeval-prompt-level |52.68|56.19| |
|
|ifeval-instruction-level |62.82|66.18| |
|
|ru-ifeval-prompt-level |44.36|46.39| |
|
|ru-ifeval-instruction-level |55.39|57.55| |
|
|
