README.md

metadata

base_model: []
library_name: transformers
tags:
  - mergekit
  - merge

Gemma-The-Writer-Restless-F32

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the passthrough merge method.

Models Merged

The following models were included in the merge:

• E:/Gemma-The-Writer-f32
• G:/9B/Tiger-Gemma-9B-v2
• J:/Gemma-The-Writer-9B
• G:/9B/Ellaria-9B
• G:/9B/gemma2-gutenberg-9B

Configuration

The following YAML configuration was used to produce this model:

# Six splits plus "end game
# "D" starts at plus .1 VS D/O proj.
# 40 plus.

slices:
 - sources:
   - model: E:/Gemma-The-Writer-f32
     layer_range: [0, 41]

# O PROJ, DPROJ to .3333 / 
# end game

# .25/.25/.61
# .30/.30/.21
# .35/.35/.51
# .40/.50/.26
# .77/.77/.88

 - sources:
   - model:  G:/9B/gemma2-gutenberg-9B
     layer_range: [41, 42]
     parameters:
       scale:
         - filter: o_proj
           value: .25
         - filter: down_proj
           value: .25
         - value: .61
 - sources:
   - model:  J:/Gemma-The-Writer-9B
     layer_range: [41, 42]
     parameters:
       scale:
         - filter: o_proj
           value: .30
         - filter: down_proj
           value: .50
         - value: .21
 - sources:
   - model:  G:/9B/Tiger-Gemma-9B-v2
     layer_range: [41, 42]
     parameters:
       scale:
         - filter: o_proj
           value: 0.45
         - filter: down_proj
           value: 0.55
         - value: .51
 - sources:
   - model:  G:/9B/Ellaria-9B
     layer_range: [41, 42]
     parameters:
       scale:
         - filter: o_proj
           value: .40
         - filter: down_proj
           value: 0.50
         - value: 0.26
 - sources:
   - model:  G:/9B/gemma2-gutenberg-9B
     layer_range: [41, 42]
     parameters:
       scale:
         - filter: o_proj
           value: 0.9
         - filter: down_proj
           value: 0.97
         - value: 0.95
merge_method: passthrough
dtype: float32