中文  |  English 
欢迎访问360智脑官网 https://ai.360.com 体验更多更强大的功能。
# 项目介绍 🎉🎉🎉我们开源了360智脑大模型的系列工作,本次开源了以下模型: - **360Zhinao-7B-Base** - **360Zhinao-7B-Chat-4K** - **360Zhinao-7B-Chat-32K** - **360Zhinao-7B-Chat-360K** 360智脑大模型特点如下: - **基础模型**:采用 3.4 万亿 Tokens 的高质量语料库训练,以中文、英文、代码为主,在相关基准评测中,同尺寸有竞争力。 - **对话模型**:具有强大的对话能力,开放4k、32k、360k三种不同窗口长度。据了解,360k(约50万字)在国内目前开源的长文本能力中最长。 # 更新信息 - [2024.04.10] 我们发布了360Zhinao-7B 1.0版本,同时开放Base模型和4k、32k、360k三种文本长度的Chat模型。 # 目录 - [下载地址](#下载地址) - [模型评估](#模型评估) - [快速开始](#快速开始) - [模型推理](#模型推理) - [模型微调](#模型微调) - [许可证](#许可证) # 下载地址 本次发布版本和下载链接见下表: | | Zhinao-Base | Zhinao-Chat | Zhinao-Chat(Int8) | Zhinao-Chat(Int4) | |-|-|-|-|-| | 1.8B | 🤖 🤗 | 🤖 🤗 | 🤖 🤗 | 🤖 🤗 | | 7B | 🤖 🤗 | 🤖 🤗 | 🤖 🤗 | 🤖 🤗 | # 模型评估 我们在OpenCompass的主流评测数据集上验证了我们的模型性能,包括C-Eval、AGIEval、MMLU、CMMLU、HellaSwag、MATH、GSM8K、HumanEval、MBPP、BBH、LAMBADA,考察的能力包括自然语言理解、知识、数学计算和推理、代码生成、逻辑推理等。 ## 基础模型 | Model | C-Eval | AGIEval | MMLU | CMMLU | HellaSwag | MATH | GSM8K | HumanEval | MBPP | BBH | LAMBADA | | - | - | - | - | - | - | - | - | - | - | - | - | | Phi-1.5-1.3B | 27.8 | 23.4 | 44.3 | 26 | 57.1 | 2.6 | 32.5 | 25 | 33 | 29.6 | 54.6 | | Qwen-1.8B | 53.3 | 36.5 | 46.4 | 51.9 | 58.7 | 2.4 | 10.2 | 7.3 | 14 | 22.6 | 54.3 | | Qwen-1.5-1.8B | 59.48 | 38.76 | 47.14 | 57.08 | 56.02 | 9.66 | 34.87 | 23.17 | 17.6 | 27.02 | 56.49 | | Baichuan2-7B-Base | 56.3 | 34.6 | 54.7 | 57 | 67 | 5.4 | 24.6 | 17.7 | 24 | 41.8 | 73.3 | | ChatGLM3-6B-Base | 67 | 47.4 | 62.8 | 66.5 | 76.5 | 19.2 | 61 | 44.5 | 57.2 | 66.2 | 77.1 | | DeepSeek-7B-Base | 45 | 24 | 49.3 | 46.8 | 73.4 | 4.2 | 18.3 | 25 | 36.4 | 42.8 | 72.6 | | InternLM2-7B | 65.7 | 50.2 | 65.5 | 66.2 | 79.6 | 19.9 | 70.6 | 41.5 | 42.4 | 64.4 | 72.1 | | InternLM-7B | 53.4 | 36.9 | 51 | 51.8 | 70.6 | 6.3 | 31.2 | 13.4 | 14 | 37 | 67 | | LLaMA-2-7B | 32.5 | 21.8 | 46.8 | 31.8 | 74 | 3.3 | 16.7 | 12.8 | 14.8 | 38.2 | 73.3 | | LLaMA-7B | 27.3 | 20.6 | 35.6 | 26.8 | 74.3 | 2.9 | 10 | 12.8 | 16.8 | 33.5 | 73.3 | | Mistral-7B-v0.1 | 47.4 | 32.8 | 64.1 | 44.7 | 78.9 | 11.3 | 47.5 | 27.4 | 38.6 | 56.7 | 75 | | MPT-7B | 23.5 | 21.3 | 27.5 | 25.9 | 75 | 2.9 | 9.1 | 17.1 | 22.8 | 35.6 | 70 | | Qwen-7B | 63.4 | 45.3 | 59.7 | 62.5 | 75 | 13.3 | 54.1 | 27.4 | 31.4 | 45.2 | 67.5 | | XVERSE-7B | 61.1 | 39 | 58.4 | 60.8 | 73.7 | 2.2 | 11.7 | 4.9 | 10.2 | 31 | 24 | | Yi-6B | 73 | 44.3 | 64 | 73.5 | 73.1 | 6.3 | 39.9 | 15.2 | 23.6 | 44.9 | 68 | | Zhinao-1.8B-Base | 49.78 | 31.87 | 50.05 | 52.58 | 57.31 | 4.82 | 15.01 | 14.02 | 19.4 | 29.76 | 69.77 | | 360Zhinao-7B-Base | 74.11 | 49.49 | 67.44 | 72.38 | 83.05 | 16.38 | 53.83 | 35.98 | 42.4 | 43.95 | 78.59 | 以上结果,在官方[Opencompass](https://rank.opencompass.org.cn/leaderboard-llm)上可查询或可复现。 # 快速开始 简单的示例来说明如何利用🤖 ModelScope和🤗 Transformers快速使用360Zhinao-7B-Base和360Zhinao-7B-Chat ## 依赖安装 - python 3.8 and above - pytorch 2.0 and above - transformers 4.37.2 and above - CUDA 11.4 and above are recommended. ```shell pip install -r requirements.txt ``` 我们推荐安装flash-attention(当前已支持flash attention 2)来提高你的运行效率以及降低显存占用。(flash-attention只是可选项,不安装也可正常运行该项目) >flash-attn >= 2.3.6 ```shell FLASH_ATTENTION_FORCE_BUILD=TRUE pip install flash-attn==2.3.6 ``` ## 🤗 Transformers ### Base模型推理 此代码演示使用transformers快速使用360Zhinao-7B-Base模型进行推理 ```python from transformers import AutoTokenizer, AutoModelForCausalLM from transformers.generation import GenerationConfig MODEL_NAME_OR_PATH = "qihoo360/360Zhinao-7B-Base" tokenizer = AutoTokenizer.from_pretrained( MODEL_NAME_OR_PATH, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( MODEL_NAME_OR_PATH, device_map="auto", trust_remote_code=True) generation_config = GenerationConfig.from_pretrained( MODEL_NAME_OR_PATH, trust_remote_code=True) inputs = tokenizer('中国二十四节气\n1. 立春\n2. 雨水\n3. 惊蛰\n4. 春分\n5. 清明\n', return_tensors='pt') inputs = inputs.to(model.device) pred = model.generate(input_ids=inputs["input_ids"], generation_config=generation_config) print("outputs:\n", tokenizer.decode(pred.cpu()[0], skip_special_tokens=True)) ``` ### Chat模型推理 此代码演示使用transformers快速使用360Zhinao-7B-Chat-4K模型进行推理 ```python from transformers import AutoTokenizer, AutoModelForCausalLM from transformers.generation import GenerationConfig MODEL_NAME_OR_PATH = "qihoo360/360Zhinao-7B-Chat-4K" tokenizer = AutoTokenizer.from_pretrained( MODEL_NAME_OR_PATH, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( MODEL_NAME_OR_PATH, device_map="auto", trust_remote_code=True) generation_config = GenerationConfig.from_pretrained( MODEL_NAME_OR_PATH, trust_remote_code=True) messages = [] #round-1 messages.append({"role": "user", "content": "介绍一下刘德华"}) response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config) messages.append({"role": "assistant", "content": response}) print(messages) #round-2 messages.append({"role": "user", "content": "他有什么代表作?"}) response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config) messages.append({"role": "assistant", "content": response}) print(messages) ``` ## 🤖 ModelScope ### Base模型推理 此代码演示使用ModelScope快速使用360Zhinao-7B-Base模型进行推理 ```python from modelscope import AutoModelForCausalLM, AutoTokenizer from modelscope import GenerationConfig MODEL_NAME_OR_PATH = "qihoo360/360Zhinao-7B-Base" tokenizer = AutoTokenizer.from_pretrained( MODEL_NAME_OR_PATH, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( MODEL_NAME_OR_PATH, device_map="auto", trust_remote_code=True) generation_config = GenerationConfig.from_pretrained( MODEL_NAME_OR_PATH, trust_remote_code=True) inputs = tokenizer('中国二十四节气\n1. 立春\n2. 雨水\n3. 惊蛰\n4. 春分\n5. 清明\n', return_tensors='pt') inputs = inputs.to(model.device) pred = model.generate(input_ids=inputs["input_ids"], generation_config=generation_config) print("outputs:\n", tokenizer.decode(pred.cpu()[0], skip_special_tokens=True)) ``` ### Chat模型推理 此代码演示使用ModelScope快速使用360Zhinao-7B-Chat-4K模型进行推理 ```python from modelscope import AutoModelForCausalLM, AutoTokenizer from modelscope import GenerationConfig MODEL_NAME_OR_PATH = "qihoo360/360Zhinao-7B-Chat-4K" tokenizer = AutoTokenizer.from_pretrained( MODEL_NAME_OR_PATH, trust_remote_code=True) model = AutoModelForCausalLM.from_pretrained( MODEL_NAME_OR_PATH, device_map="auto", trust_remote_code=True) generation_config = GenerationConfig.from_pretrained( MODEL_NAME_OR_PATH, trust_remote_code=True) messages = [] #round-1 messages.append({"role": "user", "content": "介绍一下刘德华"}) response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config) messages.append({"role": "assistant", "content": response}) print(messages) #round-2 messages.append({"role": "user", "content": "他有什么代表作?"}) response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config) messages.append({"role": "assistant", "content": response}) print(messages) ``` ## 终端 Demo 可使用终端交互实现快速体验 ```shell python cli_demo.py ```
## 网页 Demo 也可使用网页交互实现快速体验 ```shell streamlit run web_demo.py ```
## API Demo
启动命令
```shell
python openai_api.py
```
请求参数
```shell
curl --location --request POST 'http://localhost:8360/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data-raw '{
"max_new_tokens": 200,
"do_sample": true,
"top_k": 0,
"top_p": 0.8,
"temperature": 1.0,
"repetition_penalty": 1.0,
"messages": [
{
"role": "user",
"content": "你叫什么名字"
}
]
}'
```
# 模型推理
## 模型量化
我们提供了基于AutoGPTQ的量化方案,并开源了Int4量化模型。模型的效果损失很小,但能显著降低显存占用并提升推理速度。
对BF16,Int8和Int4模型在基准评测上做了测试,结果如下所示:
| Quantization | MMLU | CEval (val) | GSM8K | Humaneval |
|-|-|-|-|-|
| 360Zhinao-7B-Chat-4K (BF16) |-|-|-|-|
| 360Zhinao-7B-Chat-4K (Int8) |-|-|-|-|
| 360Zhinao-7B-Chat-4K (Int4) |-|-|-|-|
## 模型部署
### vLLM安装环境
如希望部署及加速推理,我们建议你使用 `vLLM==0.3.3`。
如果你使用**CUDA 12.1和PyTorch 2.1**,可以直接使用以下命令安装vLLM。
```shell
pip install vllm==0.3.3
```
否则请参考vLLM官方的[安装说明](https://docs.vllm.ai/en/latest/getting_started/installation.html)。
>安装完成后,还需要以下操作~
1. 把vllm/zhinao.py文件复制到env环境对应的vllm/model_executor/models目录下。
2. 然后在vllm/model_executor/models/\_\_init\_\_.py文件增加一行代码
```shell
"ZhinaoForCausalLM": ("zhinao", "ZhinaoForCausalLM"),
```
### vLLM服务启动
启动服务
```shell
python -m vllm.entrypoints.openai.api_server \
--served-model-name 360Zhinao-7B-Chat-4K \
--model qihoo360/360Zhinao-7B-Chat-4K \
--trust-remote-code \
--tensor-parallel-size 1
--max-model-len 18000 \
--host 0.0.0.0 \
--port 8360
```
使用curl请求服务
```shell
curl http://localhost:8360/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "360Zhinao-7B-Chat-4K",
"max_tokens": 200,
"top_k": 0,
"top_p": 0.8,
"temperature": 1.0,
"presence_penalty": 0.0,
"frequency_penalty": 0.0,
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "你好"}
],
"stop": [
"