中文｜ English

360智脑

🤗 Hugging Face | 🤖 ModelScope ｜ 💬 WeChat (微信)

Welcome to visit 360Zhinao official website https://ai.360.com to experience more powerful functions.

Models Introduction

🎉🎉🎉We opensource our 360Zhinao series，The following models are open sourced：

360Zhinao-7B-Base
360Zhinao-7B-Chat-4K
360Zhinao-7B-Chat-32K
360Zhinao-7B-Chat-360K

The characteristics of the 360Zhinao open-source project are：

Base Model: Leveraging a high-quality corpus of 3.4 trillion Tokens, primarily in Chinese, English, and code, we achieved competitive performance in relevant benchmark evaluations of the same scale.
Chat Model: Powerful chat capabilities and three different sequence lengths of 4k, 32k, and 360k. It is understood that 360k (about 500,000 words) is the longest sequcence length among the current chinese open source Large language model.

News and Updates

2024.04.10 We release 360Zhinao-7B 1.0 version, include the base model and three chat model with sequence length of 4k, 32k, 360k.

Download URL
Model Evaluation
Quickstart
Model Inference
Model Finetune
License

Download URL

See the following table for this release and download links:

	Zhinao-Base	Zhinao-Chat	Zhinao-Chat(Int8)	Zhinao-Chat(Int4)
1.8B	🤖 🤗	🤖 🤗	🤖 🤗	🤖 🤗
7B	🤖 🤗	🤖 🤗	🤖 🤗	🤖 🤗

Model Evaluation

We validate the performance of our model on the mainstream OpenCompass evaluation datasets, including C-Eval, AGIEval, MMLU, CMMLU, HellaSwag, MATH, GSM8K, HumanEval, MBPP, BBH, LAMBADA. The competencies examined include natural language understanding, knowledge, mathematical computation and reasoning, code generation, logical reasoning, etc.

Base Models

Model	C-Eval	AGIEval	MMLU	CMMLU	HellaSwag	MATH	GSM8K	HumanEval	MBPP	BBH	LAMBADA
Phi-1.5-1.3B	27.8	23.4	44.3	26	57.1	2.6	32.5	25	33	29.6	54.6
Qwen-1.8B	53.3	36.5	46.4	51.9	58.7	2.4	10.2	7.3	14	22.6	54.3
Qwen-1.5-1.8B	59.48	38.76	47.14	57.08	56.02	9.66	34.87	23.17	17.6	27.02	56.49
Baichuan2-7B-Base	56.3	34.6	54.7	57	67	5.4	24.6	17.7	24	41.8	73.3
ChatGLM3-6B-Base	67	47.4	62.8	66.5	76.5	19.2	61	44.5	57.2	66.2	77.1
DeepSeek-7B-Base	45	24	49.3	46.8	73.4	4.2	18.3	25	36.4	42.8	72.6
InternLM2-7B	65.7	50.2	65.5	66.2	79.6	19.9	70.6	41.5	42.4	64.4	72.1
InternLM-7B	53.4	36.9	51	51.8	70.6	6.3	31.2	13.4	14	37	67
LLaMA-2-7B	32.5	21.8	46.8	31.8	74	3.3	16.7	12.8	14.8	38.2	73.3
LLaMA-7B	27.3	20.6	35.6	26.8	74.3	2.9	10	12.8	16.8	33.5	73.3
Mistral-7B-v0.1	47.4	32.8	64.1	44.7	78.9	11.3	47.5	27.4	38.6	56.7	75
MPT-7B	23.5	21.3	27.5	25.9	75	2.9	9.1	17.1	22.8	35.6	70
Qwen-7B	63.4	45.3	59.7	62.5	75	13.3	54.1	27.4	31.4	45.2	67.5
XVERSE-7B	61.1	39	58.4	60.8	73.7	2.2	11.7	4.9	10.2	31	24
Yi-6B	73	44.3	64	73.5	73.1	6.3	39.9	15.2	23.6	44.9	68
Zhinao-1.8B-Base	49.78	31.87	50.05	52.58	57.31	4.82	15.01	14.02	19.4	29.76	69.77
360Zhinao-7B-Base	74.11	49.49	67.44	72.38	83.05	16.38	53.83	35.98	42.4	43.95	78.59

The above results, the official Opencompass can query or can emersion.

Quickstart

Simple examples to illustrate how to use 360Zhinao-7B-Base and 360Zhinao-7B-Chat quickly using 🤖 ModelScope and 🤗 Transformers

Dependency Installation

python 3.8 and above
pytorch 2.0 and above
transformers 4.37.2 and above
CUDA 11.4 and above are recommended.

pip install -r requirements.txt

We recommend installing Flash-Attention (which currently supports flash attention 2) to increase your performance and reduce your memory footprint. (flash-attention is optional and will work without installation)

flash-attn >= 2.3.6

FLASH_ATTENTION_FORCE_BUILD=TRUE pip install flash-attn==2.3.6

🤗 Transformers

Demonstration of Base Model Inference

This code demonstrates fast inference with 360Zhinao-7B-Base models using transformers.

from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers.generation import GenerationConfig

MODEL_NAME_OR_PATH = "qihoo360/360Zhinao-7B-Base"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME_OR_PATH, 
    trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME_OR_PATH,
    device_map="auto",
    trust_remote_code=True)

generation_config = GenerationConfig.from_pretrained(
    MODEL_NAME_OR_PATH,
    trust_remote_code=True)

inputs = tokenizer('中国二十四节气\n1. 立春\n2. 雨水\n3. 惊蛰\n4. 春分\n5. 清明\n', return_tensors='pt')
inputs = inputs.to(model.device)

pred = model.generate(input_ids=inputs["input_ids"], generation_config=generation_config)
print("outputs:\n", tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))

Demonstration of Chat Model Inference

This code demo uses transformers to quickly use the 360Zhinao-7B-Chat-4K model for inference.

from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers.generation import GenerationConfig

MODEL_NAME_OR_PATH = "qihoo360/360Zhinao-7B-Chat-4K"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME_OR_PATH, 
    trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME_OR_PATH,
    device_map="auto",
    trust_remote_code=True)

generation_config = GenerationConfig.from_pretrained(
    MODEL_NAME_OR_PATH,
    trust_remote_code=True)

messages = []
#round-1
messages.append({"role": "user", "content": "介绍一下刘德华"})
response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)
messages.append({"role": "assistant", "content": response})
print(messages)

#round-2
messages.append({"role": "user", "content": "他有什么代表作？"})
response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)
messages.append({"role": "assistant", "content": response})
print(messages)

🤖 ModelScope

Demonstration of Base Model Inference

This code demonstrates using ModelScope to quickly use the 360Zhinao-7B-Base model for inference.

from modelscope import AutoModelForCausalLM, AutoTokenizer
from modelscope import GenerationConfig

MODEL_NAME_OR_PATH = "qihoo360/360Zhinao-7B-Base"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME_OR_PATH, 
    trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME_OR_PATH,
    device_map="auto",
    trust_remote_code=True)

generation_config = GenerationConfig.from_pretrained(
    MODEL_NAME_OR_PATH,
    trust_remote_code=True)

inputs = tokenizer('中国二十四节气\n1. 立春\n2. 雨水\n3. 惊蛰\n4. 春分\n5. 清明\n', return_tensors='pt')
inputs = inputs.to(model.device)

pred = model.generate(input_ids=inputs["input_ids"], generation_config=generation_config)
print("outputs:\n", tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))

Demonstration of Base Model Inference

This code demonstrates using ModelScope to quickly use the 360Zhinao-7B-Chat-4K model for inference.

from modelscope import AutoModelForCausalLM, AutoTokenizer
from modelscope import GenerationConfig

MODEL_NAME_OR_PATH = "qihoo360/360Zhinao-7B-Chat-4K"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_NAME_OR_PATH, 
    trust_remote_code=True)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_NAME_OR_PATH,
    device_map="auto",
    trust_remote_code=True)

generation_config = GenerationConfig.from_pretrained(
    MODEL_NAME_OR_PATH,
    trust_remote_code=True)

messages = []
#round-1
messages.append({"role": "user", "content": "介绍一下刘德华"})
response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)
messages.append({"role": "assistant", "content": response})
print(messages)

#round-2
messages.append({"role": "user", "content": "他有什么代表作？"})
response = model.chat(tokenizer=tokenizer, messages=messages, generation_config=generation_config)
messages.append({"role": "assistant", "content": response})
print(messages)

CLI Demo

Use terminal interaction for a fast experience

python cli_demo.py

Web Demo

You can also use web interaction for a quick experience

streamlit run web_demo.py

API Demo

Start command

python openai_api.py

Request parameter

curl --location --request POST 'http://localhost:8360/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data-raw '{
    "max_new_tokens": 200,
    "do_sample": true,
    "top_k": 0,
    "top_p": 0.8,
    "temperature": 1.0,
    "repetition_penalty": 1.0,
    "messages": [
        {
            "role": "user",
            "content": "你叫什么名字？"
        }
    ]
}'

Model Inference

Quantization

We provide quantization schemes based on AutoGPTQ and open source the Int4 quantization models. The quantization model has little effect loss, but it can significantly reduce the video memory occupation and improve the inference speed.

The BF16, Int8, and Int4 models are tested on the benchmarks, and the results are as follows:

Quantization	MMLU	CEval (val)	GSM8K	Humaneval
360Zhinao-7B-Chat-4K (BF16)	-	-	-	-
360Zhinao-7B-Chat-4K (Int8)	-	-	-	-
360Zhinao-7B-Chat-4K (Int4)	-	-	-	-

Deployment

vLLM Installation

If you want to deploy and accelerate inference, we recommend using vLLM==0.3.3。

If you are using CUDA 12.1 and PyTorch 2.1, you can install vLLM directly with the following command.

pip install vllm==0.3.3

Otherwise, please refer to the official vLLM Installation Instructions。

Once the installation is complete, you will need to do the following

Copy the vllm/zhinao.py file to the vllm/model_executor/models directory corresponding to your env environment.
Then add a line to vllm/model_executor/models/__init__.py
```
"ZhinaoForCausalLM": ("zhinao", "ZhinaoForCausalLM"),
```

vLLM Service Start

Starting the service

python -m vllm.entrypoints.openai.api_server \
    --served-model-name 360Zhinao-7B-Chat-4K \
    --model qihoo360/360Zhinao-7B-Chat-4k \
    --trust-remote-code \
    --tensor-parallel-size 1
    --max-model-len 18000 \
    --host 0.0.0.0 \
    --port 8360

Use curl to request the service

curl http://localhost:8360/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
    "model": "360Zhinao-7B-Chat-4K",
    "max_tokens": 200,
    "top_k": 0,
    "top_p": 0.8,
    "temperature": 1.0,
    "presence_penalty": 0.0,
    "frequency_penalty": 0.0,
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "你好"}
    ],
    "stop": [
        "<eod>",
        "<|im_end|>",
        "<|im_start|>"
    ]
}'

Use python to request the service

from openai import OpenAI
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

chat_response = client.chat.completions.create(
    model="360Zhinao-7B-Chat-4K",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "你好"},
    ],
    stop=[
        "<eod>",
        "<|im_end|>",
        "<|im_start|>"
    ],
    presence_penalty=0.0,
    frequency_penalty=0.0
)
print("Chat response:", chat_response)

Notice: If you need to enable repetition penalty, recommended to use presence_penalty and frequency_penalty parameters.

Model Finetune

Training data

Training Data: data/training_data_sample.json. The sample data is 10,000 pieces sampled from multiturn_chat_0.8M and format converted.

Data Format:

[
  {
    "id": 1,
    "conversations": [
        {
            "from": "system",
            "value": "You are a helpful assistant."
        },
        {
            "from": "user",
            "value": "您好啊"
        },
        {
            "from": "assistant",
            "value": "你好！我今天能为您做些什么？有什么问题或需要帮助吗? 我在这里为您提供服务。"
        }
    ]
  }
]

Fine-tuning scripts

set -x

HOSTFILE=hostfile
DS_CONFIG=./finetune/ds_config_zero2.json

# PARAMS
LR=5e-6
EPOCHS=3
MAX_LEN=4096
BATCH_SIZE=4
NUM_NODES=1
NUM_GPUS=8
MASTER_PORT=29500

IS_CONCAT=False # Whether to concatenate to maximum length (MAX_LEN)

DATA_PATH="./data/training_data_sample.json"
MODEL_PATH="qihoo360/360Zhinao-7B-Base"
OUTPUT_DIR="./outputs/"

deepspeed --hostfile ${HOSTFILE} \
        --master_port ${MASTER_PORT} \
        --num_nodes ${NUM_NODES} \
        --num_gpus ${NUM_GPUS} \
        finetune.py \
        --report_to "tensorboard" \
        --data_path ${DATA_PATH} \
        --model_name_or_path ${MODEL_PATH} \
        --output_dir ${OUTPUT_DIR} \
        --model_max_length ${MAX_LEN} \
        --num_train_epochs ${EPOCHS} \
        --per_device_train_batch_size ${BATCH_SIZE} \
        --gradient_accumulation_steps 1 \
        --save_strategy steps \
        --save_steps 200 \
        --learning_rate ${LR} \
        --lr_scheduler_type cosine \
        --adam_beta1 0.9 \
        --adam_beta2 0.95 \
        --adam_epsilon 1e-8 \
        --max_grad_norm 1.0 \
        --weight_decay 0.1 \
        --warmup_ratio 0.01 \
        --gradient_checkpointing True \
        --bf16 True \
        --tf32 True \
        --deepspeed ${DS_CONFIG} \
        --is_concat ${IS_CONCAT} \
        --logging_steps 1 \
        --log_on_each_node False

bash finetune/ds_finetune.sh

By configuring the hostfile, single-machine and multi-machine training can be realized.
By configuring ds_config, realize zero2 and zero3 training
By configuring the fp16、bf16 realize mixed precision training, bf16 is recommended to be consistent with the pre-trained model.
By configuring is_concat, Whether the training data is concatenated or not is controlled. When the magnitude of the training data is large, the training efficiency can be improved by concatenation.

License

The source code of this warehouse follows the open source license Apache 2.0.

The 360 Zhinao open source model supports commercial use. If you need to use this model and its derivative models for commercial purposes, please contact us via email ([email protected]) to apply. For the specific license agreement, please see 《360 Zhinao Open Source Model License》.

qihoo360
/

360Zhinao-7B-Chat-360K

360智脑

Models Introduction

News and Updates

Table of contents

Download URL

Model Evaluation

Base Models

Quickstart

Dependency Installation

🤗 Transformers

Demonstration of Base Model Inference

Demonstration of Chat Model Inference

🤖 ModelScope

Demonstration of Base Model Inference

Demonstration of Base Model Inference

CLI Demo

Web Demo

API Demo

Model Inference

Quantization

Deployment

vLLM Installation

vLLM Service Start

Model Finetune

Training data

Fine-tuning scripts

License