mistral-nemo-cc-12B
nbeerbower/mistral-nemo-gutenberg-12B-v3 finetuned on flammenai/casual-conversation-DPO.
This is an experimental finetune that formats the conversation data sequentially with ChatML.
Method
Finetuned using an A100 on Google Colab for 3 epochs.
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 17.08 |
| IFEval (0-Shot) | 14.35 |
| BBH (3-Shot) | 34.45 |
| MATH Lvl 5 (4-Shot) | 1.81 |
| GPQA (0-shot) | 8.72 |
| MuSR (0-shot) | 14.26 |
| MMLU-PRO (5-shot) | 28.87 |
- Downloads last month
- 2
Inference Providers
NEW
This model is not currently available via any of the supported third-party Inference Providers, and
the model is not deployed on the HF Inference API.
Model tree for nbeerbower/mistral-nemo-cc-12B
Base model
intervitens/mini-magnum-12b-v1.1
Finetuned
nbeerbower/mistral-nemo-gutenberg-12B-v3
Dataset used to train nbeerbower/mistral-nemo-cc-12B
Evaluation results
- strict accuracy on IFEval (0-Shot)Open LLM Leaderboard14.350
- normalized accuracy on BBH (3-Shot)Open LLM Leaderboard34.450
- exact match on MATH Lvl 5 (4-Shot)Open LLM Leaderboard1.810
- acc_norm on GPQA (0-shot)Open LLM Leaderboard8.720
- acc_norm on MuSR (0-shot)Open LLM Leaderboard14.260
- accuracy on MMLU-PRO (5-shot)test set Open LLM Leaderboard28.870