EEVE-Instruct-Math-10.8B

EEVE-Math 프로젝트는

• Orca-Math-200k 번역 (Orca-Math: Unlocking the potential of SLMs in Grade School Math)
• gsm8k 번역, lm_eval 활용
• Mergekit을 이용한 dare-ties 사용 (DARE)

에 대한 내용을 포괄하고 있습니다.

이 모델은 EEVE-Math와 EEVE-Instruct의 dare-ties로 병합한 병합 모델입니다. 이 프로젝트는 이런 과정을 통해 특화 모델의 EEVE-Math의 성능을 많이 잃지 않고 Instruct 모델의 사용성을 유지할 수 있음을 보여주는 Proof of concept의 성격을 가지고 있습니다.

Model	gsm8k-ko(pass@1)
EEVE(Base)	0.4049
EEVE-Math (epoch 1)	0.508
EEVE-Math (epoch 2)	0.539
EEVE-Instruct	0.4511
EEVE-Instruct + Math	0.4845

Merge Details

This model was merged using the DARE TIES merge method using yanolja/EEVE-Korean-Instruct-10.8B-v1.0 as a base.

Models Merged

The following models were included in the merge:

• kuotient/EEVE-Math-10.8B

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: yanolja/EEVE-Korean-10.8B-v1.0
    # no parameters necessary for base model
  - model: yanolja/EEVE-Korean-Instruct-10.8B-v1.0
    parameters:
      density: 0.53
      weight: 0.6
  - model: kuotient/EEVE-Math-10.8B
    parameters:
      density: 0.53
      weight: 0.4
merge_method: dare_ties
base_model: yanolja/EEVE-Korean-10.8B-v1.0
parameters:
  int8_mask: true
dtype: bfloat16

Evaluation

gsm8k-ko, kobest

git clone https://github.com/kuotient/lm-evaluation-harness.git
cd lm-evaluation-harness
pip install -e .

lm_eval --model hf \
    --model_args pretrained=yanolja/EEVE-Korean-Instruct-2.8B-v1.0 \
    --tasks gsm8k-ko \
    --device cuda:0 \
    --batch_size auto:4

Model	gsm8k(pass@1)	boolq(acc)	copa(acc)	hellaswag(acc)	Overall
yanolja/EEVE-Korean-10.8B-v1.0	0.4049	-	-	-	-
yanolja/EEVE-Korean-Instruct-10.8B-v1.0	0.4511	0.8668	0.7450	0.4940	0.6392
EEVE-Math-10.8B	0.5390	0.8027	0.7260	0.4760	0.6359
EEVE-Instruct-Math-10.8B	0.4845	0.8519	0.7410	0.4980	0.6439