File size: 8,172 Bytes

---
language:
- en
license: apache-2.0
library_name: transformers
tags:
- transformers
datasets:
- mwitiderrick/OpenPlatypus
base_model: openlm-research/open_llama_3b
inference: true
model_type: llama
prompt_template: '### Instruction:\n

  {prompt}

  ### Response:

  '
created_by: mwitiderrick
pipeline_tag: text-generation
model-index:
- name: mwitiderrick/open_llama_3b_instruct_v_0.2
  results:
  - task:
      type: text-generation
    dataset:
      name: hellaswag
      type: hellaswag
    metrics:
    - type: hellaswag (0-Shot)
      value: 0.4882
      name: hellaswag(0-Shot)
  - task:
      type: text-generation
    dataset:
      name: winogrande
      type: winogrande
    metrics:
    - type: winogrande (0-Shot)
      value: 0.6133
      name: winogrande(0-Shot)
  - task:
      type: text-generation
    dataset:
      name: arc_challenge
      type: arc_challenge
    metrics:
    - type: arc_challenge (0-Shot)
      value: 0.3362
      name: arc_challenge(0-Shot)
    source:
      url: https://huggingface.co/mwitiderrick/open_llama_3b_instruct_v_0.2
      name: open_llama_3b_instruct_v_0.2 model card
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: AI2 Reasoning Challenge (25-Shot)
      type: ai2_arc
      config: ARC-Challenge
      split: test
      args:
        num_few_shot: 25
    metrics:
    - type: acc_norm
      value: 38.48
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_instruct_v_0.2
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: HellaSwag (10-Shot)
      type: hellaswag
      split: validation
      args:
        num_few_shot: 10
    metrics:
    - type: acc_norm
      value: 66.77
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_instruct_v_0.2
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MMLU (5-Shot)
      type: cais/mmlu
      config: all
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 25.34
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_instruct_v_0.2
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: TruthfulQA (0-shot)
      type: truthful_qa
      config: multiple_choice
      split: validation
      args:
        num_few_shot: 0
    metrics:
    - type: mc2
      value: 38.16
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_instruct_v_0.2
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: Winogrande (5-shot)
      type: winogrande
      config: winogrande_xl
      split: validation
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 63.46
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_instruct_v_0.2
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: GSM8k (5-shot)
      type: gsm8k
      config: main
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 1.59
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=mwitiderrick/open_llama_3b_instruct_v_0.2
      name: Open LLM Leaderboard
---
# OpenLLaMA Instruct: An Open Reproduction of LLaMA

This is an [OpenLlama model](https://huggingface.co/openlm-research/open_llama_3b) that has been fine-tuned on 1 epoch of the
[Open-Platypus](https://huggingface.co/datasets/garage-bAInd/Open-Platypus) dataset.

The modified version of the dataset can be found [here](mwitiderrick/Open-Platypus)
## Prompt Template
```
### Instruction:

{query}

### Response:
<Leave new line for model to respond> 
```
## Usage 
```python
from transformers import AutoTokenizer, AutoModelForCausalLM,pipeline

tokenizer = AutoTokenizer.from_pretrained("mwitiderrick/open_llama_3b_instruct_v_0.2")
model = AutoModelForCausalLM.from_pretrained("mwitiderrick/open_llama_3b_instruct_v_0.2")
query = "Provide step-by-step instructions for making a sweet chicken bugger"
text_gen = pipeline(task="text-generation", model=model, tokenizer=tokenizer, max_length=500)
output = text_gen(f"### Instruction:\n{query}\n### Response:\n")
print(output[0]['generated_text'])
"""
### Instruction:
Provide step-by-step instructions for making a sweet chicken bugger
### Response:
Step 1: Gather your ingredients
1. 1/2 cup of sugar
2. 1/2 cup of corn syrup
3. 1/2 cup of water
4. 1/2 cup of vegetable oil
5. 1/2 cup of vanilla extract
6. 1/2 cup of baking soda
7. 1/2 cup of salt
8. 1/2 cup of flour
9. 1/2 cup of milk
10. 1/2 cup of egg whites

Step 2: Mix the ingredients together
1. Combine the sugar, corn syrup, water, vegetable oil, vanilla extract, baking soda, and salt in a large bowl.
2. Whisk together until smooth.
3. Add the flour and mix until combined.
4. Add the milk and egg whites and mix until combined.
5. Pour the mixture into a greased 9x13 inch baking pan.
6. Bake for 30 minutes or until a toothpick inserted into the center comes out clean.

Step 3: Make the chicken bugger
1. Preheat the oven to 350 degrees Fahrenheit.
2. In a large bowl, combine the corn syrup, sugar, and cornstarch.
3. Add the chicken and mix well.
4. Divide the mixture into 12 equal portions and shape each portion into a chicken shape.
5. Place the chicken shapes on a baking sheet lined with parchment paper.
6. Bake for 15 minutes or until the chicken is cooked through.
7. Remove the chicken from the oven and allow to cool for 5 minutes.
8. Using a fork, carefully remove the chicken from the shells and place on a serving platter.
9. Serve with a side of gravy.

Step 4: Make the gravy
1. In a small saucepan, combine the cornstarch and water.
2. Stir until the mixture is smooth and begins to thicken.
3. Add the chicken broth and bring to a boil.
4. Reduce the heat to low and simmer for 10 minutes or until the gravy is
"""
```
## TruthfulQA metrics
```


|  Groups  |Version|Filter|n-shot|  Metric   | Value  |   |Stderr|
|----------|-------|------|-----:|-----------|-------:|---|-----:|
|truthfulqa|N/A    |none  |     0|acc        |  0.3166|±  |0.0012|
|          |       |none  |     0|bleu_max   | 23.7766|±  |0.7660|
|          |       |none  |     0|bleu_acc   |  0.3207|±  |0.0163|
|          |       |none  |     0|bleu_diff  | -7.1853|±  |0.7396|
|          |       |none  |     0|rouge1_max | 48.6534|±  |0.8706|
|          |       |none  |     0|rouge1_acc |  0.2766|±  |0.0157|
|          |       |none  |     0|rouge1_diff| -9.8011|±  |0.7883|
|          |       |none  |     0|rouge2_max | 31.9289|±  |0.9637|
|          |       |none  |     0|rouge2_acc |  0.2399|±  |0.0149|
|          |       |none  |     0|rouge2_diff|-11.3958|±  |0.9220|
|          |       |none  |     0|rougeL_max | 45.4592|±  |0.8754|
|          |       |none  |     0|rougeL_acc |  0.2754|±  |0.0156|
|          |       |none  |     0|rougeL_diff|-10.0740|±  |0.7807|
```
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_mwitiderrick__open_llama_3b_instruct_v_0.2)

|             Metric              |Value|
|---------------------------------|----:|
|Avg.                             |38.97|
|AI2 Reasoning Challenge (25-Shot)|38.48|
|HellaSwag (10-Shot)              |66.77|
|MMLU (5-Shot)                    |25.34|
|TruthfulQA (0-shot)              |38.16|
|Winogrande (5-shot)              |63.46|
|GSM8k (5-shot)                   | 1.59|