README.md

metadata

language:
  - en
license: apache-2.0
library_name: transformers
datasets:
  - Open-Orca/SlimOrca
  - ise-uiuc/Magicoder-OSS-Instruct-75K
  - ise-uiuc/Magicoder-Evol-Instruct-110K
  - meta-math/MetaMathQA
pipeline_tag: text-generation
arxiv: 2401.02731
model-index:
  - name: Camelidae-8x7B
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: AI2 Reasoning Challenge (25-Shot)
          type: ai2_arc
          config: ARC-Challenge
          split: test
          args:
            num_few_shot: 25
        metrics:
          - type: acc_norm
            value: 55.63
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=hywu/Camelidae-8x7B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: HellaSwag (10-Shot)
          type: hellaswag
          split: validation
          args:
            num_few_shot: 10
        metrics:
          - type: acc_norm
            value: 79.18
            name: normalized accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=hywu/Camelidae-8x7B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: MMLU (5-Shot)
          type: cais/mmlu
          config: all
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 50.1
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=hywu/Camelidae-8x7B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: TruthfulQA (0-shot)
          type: truthful_qa
          config: multiple_choice
          split: validation
          args:
            num_few_shot: 0
        metrics:
          - type: mc2
            value: 42.86
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=hywu/Camelidae-8x7B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: Winogrande (5-shot)
          type: winogrande
          config: winogrande_xl
          split: validation
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 76.24
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=hywu/Camelidae-8x7B
          name: Open LLM Leaderboard
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: GSM8k (5-shot)
          type: gsm8k
          config: main
          split: test
          args:
            num_few_shot: 5
        metrics:
          - type: acc
            value: 22.82
            name: accuracy
        source:
          url: >-
            https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=hywu/Camelidae-8x7B
          name: Open LLM Leaderboard

Parameter-Efficient Sparsity Crafting From Dense to Mixture-of-Experts for Instruction Tuning on General Tasks

News

• 1/10/2024 - Camelidae models are now available on 🤗HuggingFace.
• 1/4/2024 - We released the paper, Parameter-Efficient Sparsity Crafting From Dense to Mixture-of-Experts for Instruction Tuning on General Tasks.
• 12/22/2023 - We released the training repo that craft the dense model with LLaMA architecture to the MoE model.

Introduction

Camelidae models are trained utilizing Parameter-Efficient Sparsity Crafting techniques

Parameter-Efficient Sparsity Crafting can help dense models learn knowledge from different fields (including code and math). This appraoch perfrom instruction tuning and utilize MoE structure in an efficient way.

Specifically, Parameter-Efficient Sparsity Crafting utilizes parameter efficient techiniques including QLoRA and Adapter to perfrom Efficient Sparse Upcycling.

Model Lists

Model	Download
Camelidae-8x7B	🤗HuggingFace
Camelidae-8x13B	🤗HuggingFace
Camelidae-8x34B	🤗HuggingFace

Performance

Model	MMLU (5shot)	GSM8k (5shot)	MATH (4shot)	HumanEval (0shot)	MBPP (4shot)	HellaSwag (10shot)	TriviaQA (0shot)
GPT3.5	70.0%	57.1%	34.1%	48.1%	-	85.5%	-
Camelidae-8x34B	75.6%	78.3%	22.6%	43.9%	41.4%	85.3%	63.4%
SUSChat-34B	76.4%	72.3%	22.0%	11.6%	40.2%	83.9%	56.1%
Mixtral-8x7B-instruct	68.7%	71.7%	22.1%	25.6%	40.6%	86.5%	57.7%
LLaMA2-70B-chat	63.8%	59.3%	10.4%	32.3%	35.6%	84.8%	63.0%
Camelidae-8x13B	54.4%	52.6%	9.8%	30.6%	30.4%	82.5%	59.4%
LLaMA2-13B-chat	53.9%	37.1%	5.2%	18.9%	27.2%	81.9%	55.0%
Camelidae-8x7B	48.3%	44.0%	5.8%	18.3%	23.4%	79.2%	51.0%
LLaMA2-7B-chat	47.2%	26.3%	3.9%	12.2%	17.6%	78.6%	46.4%

We bold the highest scores for open-source models and all models separately.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

# tokenizer = AutoTokenizer.from_pretrained("hywu/Camelidae-8x7B", trust_remote_code=True)
# tokenizer = AutoTokenizer.from_pretrained("hywu/Camelidae-8x13B", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("hywu/Camelidae-8x34B", trust_remote_code=True)

# model = AutoModelForCausalLM.from_pretrained("hywu/Camelidae-8x7B", device_map="auto", trust_remote_code=True).eval()
# model = AutoModelForCausalLM.from_pretrained("hywu/Camelidae-8x13B", device_map="auto", trust_remote_code=True).eval()
model = AutoModelForCausalLM.from_pretrained("hywu/Camelidae-8x34B", device_map="auto", trust_remote_code=True).eval()

inputs = tokenizer('### Human:\nHow are you?\n### Assistant:\n', return_tensors='pt')
inputs = inputs.to(model.device)
pred = model.generate(**inputs)
print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))
# I am doing well, thank you.

Citation

@article{wu2024parameter,
  title={Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks},
  author={Wu, Haoyuan and Zheng, Haisheng and Yu, Bei},
  journal={arXiv preprint arXiv:2401.02731},
  year={2024}
}

License

The source code in this repo is licensed under the Apache 2.0 License. Camelidae models are developed for academic research and free commercial use, all usage must adhere to the license from facebookresearch and 01-ai.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	54.47
AI2 Reasoning Challenge (25-Shot)	55.63
HellaSwag (10-Shot)	79.18
MMLU (5-Shot)	50.10
TruthfulQA (0-shot)	42.86
Winogrande (5-shot)	76.24
GSM8k (5-shot)	22.82