This is a distillation experiment with SmolLM2-1.7B as teacher and SmolLM2-360M as student model.

Eval results using SmolLM evaluation scripts (LightEval):

Eval results using SmolLM evaluation scripts show distilled model slightly gained over base, in a few tasks. Small margins.

Task	Version	Metric	aloobun/d-SmolLM2-360M Value	HuggingFaceTB/SmolLM2-360M Value
all		acc_norm	0.4653	0.4642
		qem	0.0961	0.1004
custom:arc:_average:0		acc_norm	0.5303	0.5305
custom:arc:challenge:0	0	acc_norm	0.3771	0.3797
custom:arc:easy:0	0	acc_norm	0.6835	0.6814
custom:commonsense_qa:0	0	acc_norm	0.3784	0.3759
custom:gsm8k:5	0	qem	0.0326	0.0334
custom:hellaswag:0	0	acc_norm	0.5418	0.5456
custom:mmlu_pro:0	0	acc_norm	0.1127	0.1130
custom:openbook_qa:0	0	acc_norm	0.3760	0.3720
custom:piqa:0	0	acc_norm	0.7214	0.7220
custom:trivia_qa:0	0	qem	0.1596	0.1675
custom:winogrande:0	0	acc_norm	0.5312	0.5241

Eval results using lm-eval evaluation scripts:

It slightly improves upon the performance of the basemodel on the following tasks:

Tasks	HuggingFaceTB/SmolLM2-360M Value	aloobun/d-SmolLM2-360M Value
- leaderboard_bbh_causal_judgement	0.4545	0.4652
- leaderboard_bbh_geometric_shapes	0.1680	0.2040
- leaderboard_bbh_movie_recommendation	0.2120	0.2440
- leaderboard_bbh_penguins_in_a_table	0.2055	0.2123
- leaderboard_bbh_reasoning_about_colored_objects	0.1160	0.1320
- leaderboard_bbh_ruin_names	0.2360	0.2480
- leaderboard_bbh_salient_translation_error_detection	0.1480	0.2120
- leaderboard_bbh_snarks	0.5169	0.5281
- leaderboard_bbh_temporal_sequences	0.2720	0.2800
- leaderboard_musr_murder_mysteries	0.5040	0.5160

Well, it didn’t work as well as I hoped, will try again.

Eval Results aloobun/d-SmolLM2-360M (WIP)

GPQA

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
leaderboard_gpqa	N/A
- leaderboard_gpqa_diamond	1	none	0	acc_norm	↑	0.2071	±	0.0289
- leaderboard_gpqa_extended	1	none	0	acc_norm	↑	0.2308	±	0.0180
- leaderboard_gpqa_main	1	none	0	acc_norm	↑	0.2679	±	0.0209

MUSR

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
leaderboard_musr	N/A
- leaderboard_musr_murder_mysteries	1	none	0	acc_norm	↑	0.5160	±	0.0317
- leaderboard_musr_object_placements	1	none	0	acc_norm	↑	0.2383	±	0.0267
- leaderboard_musr_team_allocation	1	none	0	acc_norm	↑	0.4400	±	0.0315

BBH

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
leaderboard_bbh	N/A
- leaderboard_bbh_boolean_expressions	1	none	3	acc_norm	↑	0.5480	±	0.0315
- leaderboard_bbh_causal_judgement	1	none	3	acc_norm	↑	0.4652	±	0.0366
- leaderboard_bbh_date_understanding	1	none	3	acc_norm	↑	0.1560	±	0.0230
- leaderboard_bbh_disambiguation_qa	1	none	3	acc_norm	↑	0.3120	±	0.0294
- leaderboard_bbh_formal_fallacies	1	none	3	acc_norm	↑	0.5240	±	0.0316
- leaderboard_bbh_geometric_shapes	1	none	3	acc_norm	↑	0.2040	±	0.0255
- leaderboard_bbh_hyperbaton	1	none	3	acc_norm	↑	0.5000	±	0.0317
- leaderboard_bbh_logical_deduction_five_objects	1	none	3	acc_norm	↑	0.2240	±	0.0264
- leaderboard_bbh_logical_deduction_seven_objects	1	none	3	acc_norm	↑	0.1440	±	0.0222
- leaderboard_bbh_logical_deduction_three_objects	1	none	3	acc_norm	↑	0.3320	±	0.0298
- leaderboard_bbh_movie_recommendation	1	none	3	acc_norm	↑	0.2440	±	0.0272
- leaderboard_bbh_navigate	1	none	3	acc_norm	↑	0.5800	±	0.0313
- leaderboard_bbh_object_counting	1	none	3	acc_norm	↑	0.2080	±	0.0257
- leaderboard_bbh_penguins_in_a_table	1	none	3	acc_norm	↑	0.2123	±	0.0340
- leaderboard_bbh_reasoning_about_colored_objects	1	none	3	acc_norm	↑	0.1320	±	0.0215
- leaderboard_bbh_ruin_names	1	none	3	acc_norm	↑	0.2480	±	0.0274
- leaderboard_bbh_salient_translation_error_detection	1	none	3	acc_norm	↑	0.2120	±	0.0259
- leaderboard_bbh_snarks	1	none	3	acc_norm	↑	0.5281	±	0.0375
- leaderboard_bbh_sports_understanding	1	none	3	acc_norm	↑	0.4600	±	0.0316
- leaderboard_bbh_temporal_sequences	1	none	3	acc_norm	↑	0.2800	±	0.0285
- leaderboard_bbh_tracking_shuffled_objects_five_objects	1	none	3	acc_norm	↑	0.1720	±	0.0239
- leaderboard_bbh_tracking_shuffled_objects_seven_objects	1	none	3	acc_norm	↑	0.1440	±	0.0222
- leaderboard_bbh_tracking_shuffled_objects_three_objects	1	none	3	acc_norm	↑	0.3000	±	0.0290
- leaderboard_bbh_web_of_lies	1	none	3	acc_norm	↑	0.5480	±	0.0315

MMLU_PRO

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
leaderboard_mmlu_pro	0.1	none	5	acc	↑	0.1173	±	0.0029

IFEVAL

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
leaderboard_ifeval	3	none	0	inst_level_loose_acc	↑	0.2866	±	N/A
		none	0	inst_level_strict_acc	↑	0.2770	±	N/A
		none	0	prompt_level_loose_acc	↑	0.1497	±	0.0154
		none	0	prompt_level_strict_acc	↑	0.1423	±	0.0150

MATH HARD

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
leaderboard_math_hard	N/A
- leaderboard_math_algebra_hard	2	none	4	exact_match	↑	0.0033	±	0.0033
- leaderboard_math_counting_and_prob_hard	2	none	4	exact_match	↑	0.0081	±	0.0081
- leaderboard_math_geometry_hard	2	none	4	exact_match	↑	0.0000	±	0.0000
- leaderboard_math_intermediate_algebra_hard	2	none	4	exact_match	↑	0.0000	±	0.0000
- leaderboard_math_num_theory_hard	2	none	4	exact_match	↑	0.0065	±	0.0065
- leaderboard_math_prealgebra_hard	2	none	4	exact_match	↑	0.0104	±	0.0073
- leaderboard_math_precalculus_hard	2	none	4	exact_match	↑	0.0000	±	0.0000

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	6.01
IFEval (0-Shot)	20.97
BBH (3-Shot)	4.76
MATH Lvl 5 (4-Shot)	0.23
GPQA (0-shot)	0.45
MuSR (0-shot)	7.76
MMLU-PRO (5-shot)	1.88