metadata
base_model:
- unsloth/Qwen2.5-3B-Instruct
- unsloth/Qwen2.5-3B
library_name: transformers
tags:
- mergekit
- merge
merged_output_ties_1_4
This is a merge of pre-trained language models created using mergekit.
Merge Details
Merge Method
This model was merged using the TIES merge method using unsloth/Qwen2.5-3B as a base.
Models Merged
The following models were included in the merge:
- unsloth/Qwen2.5-3B-Instruct
- triples/merged_model
- genstruct/merged_model
- kg/merged_model
Configuration
The following YAML configuration was used to produce this model:
models:
# Base instructed model
- model: unsloth/Qwen2.5-3B-Instruct
parameters:
weight: 1
density: 1
# Merged LoRA models
- model: genstruct/merged_model
parameters:
weight: 1.0
density: 1.0
# - model: summary/merged_model
# parameters:
# weight: 1.0
# density: 1.0
- model: kg/merged_model
parameters:
weight: 1.0
density: 1.0
#### THIS BREAKS KG!!!
# - model: pII/merged_model
# parameters:
# weight: 1.0
# density: 1.0
# #### Breaks KG!
# - model: preference/merged_model
# parameters:
# weight: 1.0
# density: 1.0
- model: triples/merged_model
parameters:
weight: 1.0
density: 1.0
# - model: suitable/merged_model
# parameters:
# weight: 1.0
# density: 1.0
# - model: feedback/merged_model
# parameters:
# weight: 1.0
# density: 1.0
# Merge configuration
merge_method: ties
base_model: unsloth/Qwen2.5-3B
parameters:
normalize: true
int8_mask: true
dtype: bfloat16
# # Tokenizer configuration
# tokenizer_source: Qwen/Qwen1.5-14B-Chat
# tokenizer_parameters:
# trust_remote_code: true
# # Output configuration
# output:
# precision: bfloat16
# model_format: safetensors
# max_shard_size: "4GB"
# # Training configuration (for potential fine-tuning)
# training:
# learning_rate: 2e-5
# warmup_steps: 100
# gradient_checkpointing: true
# gradient_accumulation_steps: 4
# # Hardware optimization
# hardware:
# mixed_precision: true
# cuda_memory_fraction: 0.95
# optimize_model_memory: true