|
--- |
|
base_model: |
|
- 152334H/miqu-1-70b-sf |
|
- meta-llama/Llama-2-70b-hf |
|
- sophosympatheia/Midnight-Rose-70B-v2.0.3 |
|
library_name: transformers |
|
tags: |
|
- mergekit |
|
- merge |
|
license: other |
|
--- |
|
|
|
A creative writing model with 32k context, created by adding the essence of "what makes [Midnight-Rose-70B-v2.0.3](https://huggingface.co/sophosympatheia/Midnight-Rose-70B-v2.0.3) different to [Llama-2-70b-hf](meta-llama/Llama-2-70b-hf)" onto [miqu-1-70b](https://huggingface.co/miqudev/miqu-1-70b). |
|
|
|
# Prompting format |
|
|
|
Vicuna format is preferred: |
|
|
|
``` |
|
USER: {prompt} ASSISTANT: |
|
``` |
|
|
|
Mistral and Alpaca formats are also supported: |
|
|
|
``` |
|
[INST] {prompt} [/INST] |
|
``` |
|
|
|
``` |
|
### Instruction: |
|
{prompt} |
|
|
|
### Response: |
|
``` |
|
|
|
# Licence and usage restrictions |
|
|
|
[miqu-1-70b-sf](https://huggingface.co/152334H/miqu-1-70b-sf) is a dequantized version of the [miqu-1-70b](https://huggingface.co/miqudev/miqu-1-70b) model leaked from MistralAI. All miqu-derived models, including this merge, are suitable for non-commercial, personal use only. |
|
|
|
# Mergekit configuration |
|
|
|
The following YAML configuration was used to produce this model: |
|
|
|
```yaml |
|
name: _miquplus-midnight-70b |
|
merge_method: task_arithmetic |
|
parameters: |
|
normalize : false |
|
weight: 1 |
|
models: |
|
- model: meta-llama/Llama-2-70b-hf |
|
- model: 152334H/miqu-1-70b-sf |
|
- model: sophosympatheia/Midnight-Rose-70B-v2.0.3 |
|
base_model: meta-llama/Llama-2-70b-hf |
|
dtype: float16 |
|
--- |
|
name: miquplus-midnight-70b |
|
merge_method: linear |
|
models: |
|
- model: 152334H/miqu-1-70b-sf |
|
parameters: |
|
weight: |
|
- filter: v_proj |
|
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1] |
|
- filter: o_proj |
|
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1] |
|
- filter: up_proj |
|
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1] |
|
- filter: gate_proj |
|
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1] |
|
- filter: down_proj |
|
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1] |
|
- value: 1 |
|
- model: _miquplus-midnight-70b |
|
parameters: |
|
weight: |
|
- filter: v_proj |
|
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0] |
|
- filter: o_proj |
|
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0] |
|
- filter: up_proj |
|
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0] |
|
- filter: gate_proj |
|
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0] |
|
- filter: down_proj |
|
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0] |
|
- value: 0 |
|
base_model: 152334H/miqu-1-70b-sf |
|
tokenizer_source: base |
|
dtype: float16 |
|
``` |
|
|
|
**NOTE**: Run with `mergekit-mega` rather than `mergekit` as there are 2 documents in this file. |
|
|
|
# Model background |
|
|
|
Created using [Mergekit](https://github.com/arcee-ai/mergekit) in two stages: |
|
|
|
- First, a (broken!) "donor" model was created by adding (`Midnight-Rose-70B-v2.0.3` - `Llama-2-70b-hf`) to `miqu-1-70b-sf` using `task_arithmetic`. |
|
- In the second stage, the `v_proj`, `o_proj`, `up_proj`, `gate_proj` and `down_proj` tensors were merged back into `miqu-1-70b-sf`. |
|
|
|
**NOTE**: After the second stage model is created the "donor" model can be deleted as it won't be needed again. |
|
|
|
<details><summary>Click to see more details on the reasoning behind the merge</summary> |
|
|
|
## A. There are 3 very compelling reasons to keep the `q_proj` and `k_proj` matrices intact: |
|
|
|
### 1. These are the only matrices that have the RoPE rotations applied: |
|
|
|
 |
|
see: https://adalkiran.github.io/llama-nuts-and-bolts/#model-diagram |
|
|
|
and trying to blend the the `q_proj` and `k_proj` matrices of models with 100x different base RoPE frequencies is just going to contract and dilate their perceived "distance" / "time" in the text they write (so both are wrong). If you did want to insist on blending them, then the base RoPE frequency would need to be set to something in-between that is the "least wrong" for both base RoPE frequencies and the maximum context reduced accordingly... |
|
|
|
### 2. They don't actually appear to be responsible for domain-specific adaptation anyway: |
|
 |
|
|
|
see: https://www.lesswrong.com/posts/j84JhErNezMxyK4dH/llm-modularity-the-separability-of-capabilities-in-large |
|
|
|
### 3: The top ~30% of embedding dimensions for the 10k base RoPE frequency models aren't being used properly: |
|
> For LLaMA2(Touvron et al., 2023b), the critical dimension dextra is 92. This implies that only the first 92 dimensions of the qt, ks vectors of LLaMA2 have seen the complete positional information during the pre-training phase and are adequately trained. In other words, the last 36 dimensions lack sufficient training, contributing to the extrapolation challenges seen in RoPE-based LLMs |
|
|
|
see: [Scaling Laws of RoPE-based Extrapolation](https://arxiv.org/abs/2310.05209) |
|
|
|
So even if we could try to map the 10k RoPE models embedding vectors to match the 1M RoPE models embedding vectors post-dot-product, the 1M RoPE models embedding vectors are likely fine-tuned to use these top 36 dimensions properly and have thus changed very significantly during continued pre-training. |
|
|
|
## B. The norm vectors from the "donor" model can't be merged: |
|
|
|
The `input_layernorm.weight`, `post_attention_layernorm.weight` and `norm.weight` vectors all use [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html), and: |
|
|
|
> **weight** – the learnable weights of the module of shape normalized_shapenormalized_shape when elementwise_affine is set to True. The values are initialized to 1. |
|
|
|
So to combine these we would need to use the [Geometric Mean](https://en.wikipedia.org/wiki/Geometric_mean) for the `task_arithmetic` operation (and hence why the "donor" model is broken). |
|
|
|
## C. The first and last 8 layers, the `embed_tokens.weight` and `lm_head.weight` tensors are not merged: |
|
|
|
The model still functions if these are merged into the second stage model, but empirically it seems to work better if we don't do this as keep the original `miqu-1-70b-sf` weights. |
|
|
|
</details> |
