README.md

---
base_model:
- 152334H/miqu-1-70b-sf
- meta-llama/Llama-2-70b-hf
- sophosympatheia/Midnight-Rose-70B-v2.0.3
library_name: transformers
tags:
- mergekit
- merge
license: other
---

A creative writing model with 32k context, created by adding the essence of "what makes [Midnight-Rose-70B-v2.0.3](https://huggingface.co/sophosympatheia/Midnight-Rose-70B-v2.0.3) different to [Llama-2-70b-hf](meta-llama/Llama-2-70b-hf)" onto [miqu-1-70b](https://huggingface.co/miqudev/miqu-1-70b).

# Prompting format

Vicuna format is preferred:

```
USER: {prompt} ASSISTANT:
```

Mistral and Alpaca formats are also supported:

```
[INST] {prompt} [/INST]
```

```
### Instruction:
{prompt}

### Response:
```

# Licence and usage restrictions

[miqu-1-70b-sf](https://huggingface.co/152334H/miqu-1-70b-sf) is a dequantized version of the [miqu-1-70b](https://huggingface.co/miqudev/miqu-1-70b) model leaked from MistralAI. All miqu-derived models, including this merge, are suitable for non-commercial, personal use only.

# Mergekit configuration

The following YAML configuration was used to produce this model:

```yaml
name: _miquplus-midnight-70b
merge_method: task_arithmetic
parameters:
  normalize : false
  weight: 1
models:
  - model: meta-llama/Llama-2-70b-hf
  - model: 152334H/miqu-1-70b-sf
  - model: sophosympatheia/Midnight-Rose-70B-v2.0.3
base_model: meta-llama/Llama-2-70b-hf
dtype: float16
---
name: miquplus-midnight-70b
merge_method: linear
models:
  - model: 152334H/miqu-1-70b-sf
    parameters:
      weight:
        - filter: v_proj
          value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
        - filter: o_proj
          value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
        - filter: up_proj
          value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
        - filter: gate_proj
          value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
        - filter: down_proj
          value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1]
        - value: 1
  - model: _miquplus-midnight-70b
    parameters:
      weight:
        - filter: v_proj
          value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
        - filter: o_proj
          value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
        - filter: up_proj
          value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
        - filter: gate_proj
          value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
        - filter: down_proj
          value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0]
        - value: 0
base_model: 152334H/miqu-1-70b-sf
tokenizer_source: base
dtype: float16
```

**NOTE**: Run with `mergekit-mega` rather than `mergekit` as there are 2 documents in this file.

# Model background

Created using [Mergekit](https://github.com/arcee-ai/mergekit) in two stages:

- First, a (broken!) "donor" model was created by adding (`Midnight-Rose-70B-v2.0.3` - `Llama-2-70b-hf`) to `miqu-1-70b-sf` using `task_arithmetic`.
- In the second stage, the `v_proj`, `o_proj`, `up_proj`, `gate_proj` and `down_proj` tensors were merged back into `miqu-1-70b-sf`.

**NOTE**: After the second stage model is created the "donor" model can be deleted as it won't be needed again.

<details><summary>Click to see more details on the reasoning behind the merge</summary>

## A. There are 3 very compelling reasons to keep the `q_proj` and `k_proj` matrices intact:

### 1. These are the only matrices that have the RoPE rotations applied:

![image.png](https://cdn-uploads.huggingface.co/production/uploads/65995c45539c808e84c38bf1/4j88A7gCkkifwxUX45gOQ.png)
see:  https://adalkiran.github.io/llama-nuts-and-bolts/#model-diagram

and trying to blend the the `q_proj` and `k_proj` matrices of models with 100x different base RoPE frequencies is just going to contract and dilate their perceived "distance" / "time" in the text they write (so both are wrong). If you did want to insist on blending them, then the base RoPE frequency would need to be set to something in-between that is the "least wrong" for both base RoPE frequencies and the maximum context reduced accordingly...

### 2. They don't actually appear to be responsible for domain-specific adaptation anyway:
![image.png](https://cdn-uploads.huggingface.co/production/uploads/65995c45539c808e84c38bf1/B4GL59ViEA66gMsYKvbZH.png)

see:  https://www.lesswrong.com/posts/j84JhErNezMxyK4dH/llm-modularity-the-separability-of-capabilities-in-large

### 3: The top ~30% of embedding dimensions for the 10k base RoPE frequency models aren't being used properly:
> For LLaMA2(Touvron et al., 2023b), the critical dimension dextra is 92. This implies that only the first 92 dimensions of the qt, ks vectors of LLaMA2 have seen the complete positional information during the pre-training phase and are adequately trained. In other words, the last 36 dimensions lack sufficient training, contributing to the extrapolation challenges seen in RoPE-based LLMs

see: [Scaling Laws of RoPE-based Extrapolation](https://arxiv.org/abs/2310.05209)

So even if we could try to map the 10k RoPE models embedding vectors to match the 1M RoPE models embedding vectors post-dot-product, the 1M RoPE models embedding vectors are likely fine-tuned to use these top 36 dimensions properly and have thus changed very significantly during continued pre-training.

## B. The norm vectors from the "donor" model can't be merged:

The `input_layernorm.weight`, `post_attention_layernorm.weight` and `norm.weight` vectors all use [LayerNorm](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html), and:

> **weight** – the learnable weights of the module of shape normalized_shapenormalized_shape when elementwise_affine is set to True. The values are initialized to 1.

So to combine these we would need to use the [Geometric Mean](https://en.wikipedia.org/wiki/Geometric_mean) for the `task_arithmetic` operation (and hence why the "donor" model is broken).

## C. The first and last 8 layers, the `embed_tokens.weight` and `lm_head.weight` tensors are not merged:

The model still functions if these are merged into the second stage model, but empirically it seems to work better if we don't do this as keep the original `miqu-1-70b-sf` weights.

</details>