|
--- |
|
license: mit |
|
language: |
|
- zh |
|
--- |
|
|
|
|
|
--- |
|
基于baichuan-inc/Baichuan-13B-Chat 做的GPTQ的量化，可直接加载，占用GPU约12G左右，用起来效果不错 |
|
--- |
|
|
|
|
|
调用代码： |
|
|
|
---------------- |
|
import torch |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
from transformers.generation.utils import GenerationConfig |
|
|
|
model_dir = 'yinfupai/Baichuan-13B-Chat-GPTQ' |
|
|
|
tokenizer = AutoTokenizer.from_pretrained(model_dir,trust_remote_code=True) |
|
model = AutoModelForCausalLM.from_pretrained(model_dir,device_map="auto",torch_dtype=torch.float16,trust_remote_code=True) |
|
|
|
model.generation_config = GenerationConfig.from_pretrained(model_dir) |
|
model.eval() |
|
|
|
messages = [] |
|
|
|
#按baichuan要求的格式 |
|
messages.append({"role": "user", "content": "列举一下先天八卦的卦象"}) |
|
|
|
response = model.chat(tokenizer, messages) |
|
|
|
print(response) |
|
|
|
----------------------- |
|
|
|
请注意模型的商用授权，请遵照baichuan-inc/Baichuan-13B-Chat的页面中的声明 |
