README.md

metadata

license: llama3
library_name: transformers
tags:
  - mergekit
  - merge
  - not-for-all-audiences
base_model:
  - Hastagaras/anjrit
  - Hastagaras/anying
model-index:
  - name: Anjir-8B-L3
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: AI2 Reasoning Challenge (25-Shot)
          type: ai2_arc
          config: ARC-Challenge
          split: test
          args:
            num_few_shot: 25
        metrics:
          - type: acc_norm
            value: 63.57
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Hastagaras/Anjir-8B-L3
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: HellaSwag (10-Shot)
          type: hellaswag
          split: validation
          args:
            num_few_shot: 10
        metrics:
          - type: acc_norm
            value: 84.15
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Hastagaras/Anjir-8B-L3
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU (5-Shot)
          type: cais/mmlu
          config: all
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 67.67
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Hastagaras/Anjir-8B-L3
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: TruthfulQA (0-shot)
          type: truthful_qa
          config: multiple_choice
          split: validation
          args:
            num_few_shot: 0
        metrics:
          - type: mc2
            value: 52.67
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Hastagaras/Anjir-8B-L3
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: Winogrande (5-shot)
          type: winogrande
          config: winogrande_xl
          split: validation
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 78.61
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Hastagaras/Anjir-8B-L3
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: GSM8k (5-shot)
          type: gsm8k
          config: main
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 67.78
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Hastagaras/Anjir-8B-L3
          name: Open LLM Leaderboard

ANJIRRR

This model aims to achieve the human-like responses of the Halu Blackroot, the no refusal tendencies of the Halu OAS, and the smartness of the Standard Halu.

Model Details:

• Anjrit: This model is similar to my Halu Blackroot model, but instead of using the standard version, this model uses the OAS version.

• Anying: This model is also similar to the Halu Blackroot, but instead of using the model stock, I merged the Blackroot lora manually with a very low alpha.

Both models have downsides. The Anjrit model lacks coherency, while the Anying model lacks a human-like responses.

I decided to merge both models with the following method:

1. First, I compared the response from each layer of both models using the baukit notebook.

2. After comparing both, it seems that around the bottom layer, the Anjrit model is better, perhaps because it is unhinged.

3. From the bottom to the middle layer, the Anjrit is still better, but the Anying seems smarter.

4. At the middle layer, both seem equal, but again, the Anjrit is unhinged, so I prefer this one.

5. From the middle to the top layer, the Anying is better. It is smarter, and the response is more structured.

6. The top layer of the Anjrit model is better since the model itself is orthogonalized, so I prefer this one.

7. Then I performed slerp with the following configuration. I don't know if this is really how the slerp merge works, so let's just say this is an experimental merge.

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: Hastagaras/anjrit
  - model: Hastagaras/anying
merge_method: slerp
base_model: Hastagaras/anjrit
dtype: bfloat16
parameters:
  t: [0.12, 0.17, 0.29, 0.44, 0.26]

WARNING: This model has not been extensively tested or evaluated, and its performance characteristics are currently unknown. It may generate harmful, biased, or inappropriate content. Please exercise caution and use it at your own risk and discretion.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	69.07
AI2 Reasoning Challenge (25-Shot)	63.57
HellaSwag (10-Shot)	84.15
MMLU (5-Shot)	67.67
TruthfulQA (0-shot)	52.67
Winogrande (5-shot)	78.61
GSM8k (5-shot)	67.78