File size: 2,939 Bytes
b0cbfd3 3509b5c 8de887d b0cbfd3 3509b5c b0cbfd3 6bae5f8 64b2704 2f49197 64b2704 3509b5c e06a18a ee98521 e06a18a 3509b5c 1ff7374 3509b5c e06a18a 3509b5c e06a18a ee98521 5bc7832 3f20f63 5bc7832 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 |
---
base_model: aiplanet/effi-7b
inference: false
license: apache-2.0
language:
- en
library_name: transformers
model_type: llama
pipeline_tag: text-generation
tags:
- awq
- llama-2
- text-generation-inference
---
effi 7b AWQ is a quantized version of effi 7b whiich is a 7 billion parameter model built by AI Planet. We have used Auto-AWQ for quantising the model
## Model Details
### Model Description
This original model (Effi-7B) has been fine-tuned on Chain of Thought datasets, which has context from mixed sources with corresponding rationale. The final finetuned Large Language Model(LLM) have shown enhanced capabilities of solving novel tasks by providing a reasoning.And the final model was quantized into AWQ format
- **Developed by:** AI Planet
- **Model type:** Casual Decoder only
- **Language(s) (NLP):** English
- **Quantisation type:** AWQ(4-bit)
- **License:** Apache 2.0
- **Quantized from model:** Effi-7b
### Qunatization Configuration
- **zero_point:** true
- **q_group_size:** 128
- **w_bit:** 4
- **version:** "GEMM"
- **modules_to_not_convert:** null
### Example of usage
```py
import torch
from transformers import AutoTokenizer , AutoModelForCausalLM
quant_path = "aiplanet/effi-7b-awq"
model = AutoModelForCausalLM.from_pretrained(quant_path , device_map='cuda')
tokenizer = AutoTokenizer.from_pretrained(quant_path, trust_remote_code=True , safetensors=True , fuse_layers=True)
tst = """
### INSTRUCTION:
Virgin Australia, the trading name of Virgin Australia Airlines Pty Ltd, is an Australian-based airline. It is the largest airline by fleet size to use the Virgin brand. It commenced services on 31 August 2000 as Virgin Blue, with two aircraft on a single route. It suddenly found itself as a major airline in Australia's domestic market after the collapse of Ansett Australia in September 2001. The airline has since grown to directly serve 32 cities in Australia, from hubs in Brisbane, Melbourne and Sydney.Is Virgin Australia and Virgin Blue the same airlines?
"""
system_message = "Given your chain of thought reasoning, provide a rationale for the context in the source."
template=f"""
Context: {system_message}
Human: {tst}
"""
# Tokenize the input
input_ids = tokenizer(template, return_tensors="pt", truncation=True).input_ids.cuda()
# Run the model to infere an output
outputs = model.generate(input_ids=input_ids, max_new_tokens=512, top_p=0.9,temperature=0.1 , top_k=1, repetition_penalty=1.1)
# Print the result
print(f"{tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0][len(template):]}")
```
### Framework versions
- **Transformers** 4.37.2
- **Autoawq** 0.1.8
### Citation
```
@misc {bhavyaaiplanet,
author = { {Bhavya Bhola} },
title = { Quantized version of effi-7b by AI Planet},
year = 2024,
url = { https://huggingface.co/aiplanet/effi-7b-awq },
publisher = { Hugging Face }
}
``` |