| license: apache-2.0 | |
| tags: | |
| - finetuned | |
| - quantized | |
| - 4-bit | |
| - gptq | |
| - transformers | |
| - safetensors | |
| - mistral | |
| - text-generation | |
| - mlabonne/NeuralMarcoro14-7B | |
| - dpo | |
| - 7B | |
| - winograd | |
| - mmlu_abstract_algebra | |
| - dataset:hromi/winograd_dpo_basic | |
| - base_model:mlabonne/NeuralMarcoro14-7B | |
| - doi:10.57967/hf/1611 | |
| - license:apache-2.0 | |
| - autotrain_compatible | |
| - endpoints_compatible | |
| - text-generation-inference | |
| - region:us | |
| - has_space | |
| model_name: Turdus-GPTQ | |
| base_model: udkai/Turdus | |
| inference: false | |
| model_creator: udkai | |
| pipeline_tag: text-generation | |
| quantized_by: MaziyarPanahi | |
| # Description | |
| [MaziyarPanahi/Turdus-GPTQ](https://huggingface.co/MaziyarPanahi/Turdus-GPTQ) is a quantized (GPTQ) version of [udkai/Turdus](https://huggingface.co/udkai/Turdus) | |
| ## How to use | |
| ### Install the necessary packages | |
| ``` | |
| pip install --upgrade accelerate auto-gptq transformers | |
| ``` | |
| ### Example Python code | |
| ```python | |
| from transformers import AutoTokenizer, pipeline | |
| from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig | |
| import torch | |
| model_id = "MaziyarPanahi/Turdus-GPTQ" | |
| quantize_config = BaseQuantizeConfig( | |
| bits=4, | |
| group_size=128, | |
| desc_act=False | |
| ) | |
| model = AutoGPTQForCausalLM.from_quantized( | |
| model_id, | |
| use_safetensors=True, | |
| device="cuda:0", | |
| quantize_config=quantize_config) | |
| tokenizer = AutoTokenizer.from_pretrained(model_id) | |
| pipe = pipeline( | |
| "text-generation", | |
| model=model, | |
| tokenizer=tokenizer, | |
| max_new_tokens=512, | |
| temperature=0.7, | |
| top_p=0.95, | |
| repetition_penalty=1.1 | |
| ) | |
| outputs = pipe("What is a large language model?") | |
| print(outputs[0]["generated_text"]) | |
| ``` |