README.md

metadata

base_model:
  - CultriX/Qwen2.5-14B-Wernickev5
  - CultriX/Qwen2.5-14B-Wernickev3
library_name: transformers
tags:
  - mergekit
  - merge

merge

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the SLERP merge method.

Models Merged

The following models were included in the merge:

• CultriX/Qwen2.5-14B-Wernickev5
• CultriX/Qwen2.5-14B-Wernickev3

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: CultriX/Qwen2.5-14B-Wernickev3
  - model: CultriX/Qwen2.5-14B-Wernickev5
merge_method: slerp
base_model: CultriX/Qwen2.5-14B-Wernickev3
dtype: bfloat16
parameters:
  t: [0, 0.5, 1, 0.5, 0]
dtype: bfloat16
adaptive_merge_parameters:
  task_weights:
    tinyArc: 1.2        # Emphasize Arc's logical and reasoning tasks
    tinyHellaswag: 1.1  # Maintain strong performance in contextual predictions
    tinyMMLU: 1.2       # Ensure domain knowledge is well-preserved
    tinyTruthfulQA: 1.3 # Leverage v3's top TruthfulQA score
    tinyTruthfulQA_mc1: 1.1 # Balance for both models' performance
    tinyWinogrande: 1.2 # Enhance contextual understanding
  smoothing_factor: 0.2  # Moderate blending for stable integration
gradient_clipping: 1.0   # Prevent over-contribution from any single model