IDEA-CCNL/Ziya-BLIP2-14B-Visual-v1 · 按照指引测试了一下，但是生成出来的内容都无意义

Sep 20, 2023

试了多个图片+相同的问题，输出的内容都一样：flags崂awранasiedor胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽胥炽

代码如下：
from transformers import LlamaForCausalLM, LlamaTokenizer, BlipImageProcessor
from modeling_ziya_blip2 import ZiyaBlip2ForCausalLM
from PIL import Image
import torch

LM_MODEL_PATH = "/data/model/models--IDEA-CCNL--Ziya-LLaMA-13B-v1/snapshots/64d931f346e1a49ea3bbca07a831370
75bab1c66"
lm_model = LlamaForCausalLM.from_pretrained(LM_MODEL_PATH,
torch_dtype=torch.float16)
tokenizer = LlamaTokenizer.from_pretrained(LM_MODEL_PATH)

OPENAI_CLIP_MEAN = [0.48145466, 0.4578275, 0.40821073]
OPENAI_CLIP_STD = [0.26862954, 0.26130258, 0.27577711]

model = ZiyaBlip2ForCausalLM.from_pretrained(".", language_model=lm_model,
torch_dtype=torch.float16)
image_size = model.config.vision_config.image_size
image_processor = BlipImageProcessor(
size={"height": image_size, "width": image_size},
image_mean=OPENAI_CLIP_MEAN,
image_std=OPENAI_CLIP_STD,
)
model.cuda()

generate_config = {
"max_new_tokens": 128,
"top_p": 0.1,
"temperature": 0.7
}

pmt = "简单描述下这张图片"
output = model.chat(
tokenizer=tokenizer,
pixel_values=image_processor(Image.open("/data/imgs/test01.png"), return_tensors="pt").pixel_values.to(model.device),
query=pmt,
previous_querys=[],
previous_outputs=[],
**generate_config,
)
print(output)

wuxiaojun

Fengshenbang-LM org Sep 20, 2023

你的ziya-llama-v1是完整权重吗

jamestang0219

Sep 20, 2023

你的ziya-llama-v1是完整权重吗

应该是的，使用transformers直接下载的ziya-llama-v1的权重

wuxiaojun

Fengshenbang-LM org Sep 20, 2023

jamestang0219

Sep 20, 2023

fyi

wuxiaojun

Fengshenbang-LM org Sep 20, 2023

你下载的是delta权重，需要合并后才能获得完整的ziya-llama-v1权重

jamestang219

Sep 20, 2023

你下载的是delta权重，需要合并后才能获得完整的ziya-llama-v1权重

感谢，了解了～请问原始llama权重是13b还是13b-chat？

wuxiaojun

Fengshenbang-LM org Sep 20, 2023

https://huggingface.co/IDEA-CCNL/Ziya-LLaMA-13B-v1 参考使用说明进行合并

jamestang219

Sep 20, 2023

https://huggingface.co/IDEA-CCNL/Ziya-LLaMA-13B-v1 参考使用说明进行合并

再次感谢，看到了是llama-13b，我试一下合并之后看看行不行

jamestang219

Sep 20, 2023

https://huggingface.co/IDEA-CCNL/Ziya-LLaMA-13B-v1 参考使用说明进行合并

参考了这个说明进行合并，使用了原始Llama-13b（非chat）的模型参数，以及差值模型IDEA-CCNL--Ziya-LLaMA-13B-v1，经过上述步骤合并之后得到的文本态模型做推理仍然是乱码，转换过程中并未报错，所以猜测是需要使用Llama-13b-chat模型吗？或者差值模型需要使用Ziya-v1.1？

推理代码：
from transformers import AutoTokenizer
from transformers import LlamaForCausalLM
import torch

device = torch.device("cuda")
ckpt = './Ziya-LLaMA-13B'

query="帮我写一份去西安的旅游计划"
model = LlamaForCausalLM.from_pretrained(ckpt, torch_dtype=torch.float16, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(ckpt, use_fast=False)
inputs = ':' + query.strip() + '\n:'

input_ids = tokenizer(inputs, return_tensors="pt").input_ids.to(device)
generate_ids = model.generate(
input_ids,
max_new_tokens=1024,
do_sample = True,
top_p = 0.85,
temperature = 1.0,
repetition_penalty=1.,
eos_token_id=2,
bos_token_id=1,
pad_token_id=0)
output = tokenizer.batch_decode(generate_ids)[0]
print(output)

输出结果： :帮我写一份去西安的旅游计划 :culo tea Dresendar Wiesmode hall Ende refreshmode Nuraihonneurscr sensscr nuidi Rectculoidi서profilepushREF Wies WiesHEADHEAD
riz势attice nu兴culo hall hall Wiesbraries Wies Wies Nu舛Podpushodes ISSNodes fol hallitionHOamma吐modeettes Wiesbraries MotHO literaturelane� hallonneIEsoap铌 hallmouthleb
NF Columbia Brow teaсліurlsDATAụ Brow polygon coffeetoolemployee椽 DresCPRemote DresmodeNF све椽 polygonlgPushierz Employeeessaisiemouth Schlurlskęlaneildarnскаtex Po aller
Dres converlanegroELDurls烛 polygon ban Wiesplanavaneso萝ause ESRemoteiesz

wuxiaojun

Fengshenbang-LM org Sep 20, 2023

看起来是没合并对，因为看起来使用ziya-llama-v1推理的结果也有问题，合并过程中有warning什么的吗，可以贴出来

jamestang219

Sep 21, 2023

看起来是没合并对，因为看起来使用ziya-llama-v1推理的结果也有问题，合并过程中有warning什么的吗，可以贴出来

Llama原始参数通过md5check是一致的：
/Llama-13b]# cat checklist.chk
76087fcc6c5269ddea74c4f0f6144e4b consolidated.00.pth
3f893d85e21541021aea58a3c4162e4b consolidated.01.pth
b54ad5614e2a66c1e6eb3d4d25aa8d94 params.json
/Llama-13b]# md5sum consolidated.00.pth
76087fcc6c5269ddea74c4f0f6144e4b consolidated.00.pth
/Llama-13b]# md5sum consolidated.01.pth
3f893d85e21541021aea58a3c4162e4b consolidated.01.pth
/Llama-13b]# md5sum params.json
b54ad5614e2a66c1e6eb3d4d25aa8d94 params.json

Ziya-LLaMA-13B-v1差值模型通过hf的from_pretrained进行下载：
/models--IDEA-CCNL--Ziya-LLaMA-13B-v1]# ll snapshots/64d931f346e1a49ea3bbca07a83137075bab1c66/
total 0
lrwxrwxrwx 1 501 games 52 Sep 19 15:14 added_tokens.json -> ../../blobs/ac47931a423c09e6907dc1ce771b0d2e5d9baac3
lrwxrwxrwx 1 501 games 52 Sep 19 15:15 config.json -> ../../blobs/8789a9d9bf830dc2aaa304fa7eb0d0a1482d7b65
lrwxrwxrwx 1 501 games 52 Sep 19 16:45 generation_config.json -> ../../blobs/2c057487fd479614df7ac437b5ae2faedb245063
lrwxrwxrwx 1 501 games 76 Sep 19 15:18 pytorch_model-00001-of-00028.bin -> ../../blobs/23fc9735c66896bc90316f916da4575c1c845e35c2987f961148d13af5f058cd
lrwxrwxrwx 1 501 games 76 Sep 19 15:21 pytorch_model-00002-of-00028.bin -> ../../blobs/81eb8e86db149746e75a283774dc39e95c6d21deae25c75de8b2dc454048d3b5
lrwxrwxrwx 1 501 games 76 Sep 19 15:24 pytorch_model-00003-of-00028.bin -> ../../blobs/e23f4df4d8d1a443e9439d8c341f86bc5338ad2b3def759bf80256d7fed0d4dd
lrwxrwxrwx 1 501 games 76 Sep 19 15:28 pytorch_model-00004-of-00028.bin -> ../../blobs/9b61d1672b5571ad44a7a763017ee2f92464dfc9f7a3e6d5b56ec01165ad1a0e
lrwxrwxrwx 1 501 games 76 Sep 19 15:31 pytorch_model-00005-of-00028.bin -> ../../blobs/06c8806bdb6775f91d4adb32a36114267dd6f7a93a80f717a739d879a86e91ce
lrwxrwxrwx 1 501 games 76 Sep 19 15:34 pytorch_model-00006-of-00028.bin -> ../../blobs/514462397cc59434916e3071de7c89a192e81df4c378a5d299f5bc7a508c0bf7
lrwxrwxrwx 1 501 games 76 Sep 19 15:37 pytorch_model-00007-of-00028.bin -> ../../blobs/7bbb816d7ceed28b34aa2107296b13b2031568b7a9e8b844d40a92e76ad78fc8
lrwxrwxrwx 1 501 games 76 Sep 19 15:41 pytorch_model-00008-of-00028.bin -> ../../blobs/96af0f47f9c6da189110ea243f24141a1d7ff7eab5c5b2e1961ef1c1cba20dbf
lrwxrwxrwx 1 501 games 76 Sep 19 15:44 pytorch_model-00009-of-00028.bin -> ../../blobs/1463d7d3c6b9afd9513cd8bb2c4dda2e6d5667ebc1a69e81944a7f34ad458eaa
lrwxrwxrwx 1 501 games 76 Sep 19 15:47 pytorch_model-00010-of-00028.bin -> ../../blobs/e5af1f1d941286cd7d9d7bb4fbd6d041555b0bbc40403879518bcbd532b9af27
lrwxrwxrwx 1 501 games 76 Sep 19 15:50 pytorch_model-00011-of-00028.bin -> ../../blobs/cf096fda017dd723bc545e432b52a0b6aaf456fc0a73b6d983c42abd5c1c64b0
lrwxrwxrwx 1 501 games 76 Sep 19 15:53 pytorch_model-00012-of-00028.bin -> ../../blobs/b92925f0b58f4c7a2f1c91e17e7f9835070279ef912e61147dba45a22ef34b5b
lrwxrwxrwx 1 501 games 76 Sep 19 15:56 pytorch_model-00013-of-00028.bin -> ../../blobs/2927d69751954ed780094cd23c7342c188de33dc62297c6510448283de603aaa
lrwxrwxrwx 1 501 games 76 Sep 19 16:00 pytorch_model-00014-of-00028.bin -> ../../blobs/53b65b4a0a2b4a3d6187174adea447da88c090ee95081230f183e90fc7aa6ecb
lrwxrwxrwx 1 501 games 76 Sep 19 16:03 pytorch_model-00015-of-00028.bin -> ../../blobs/64f25743c52d3aaacd1de45bcc23e367df48e21c0099a52181497394c069c09c
lrwxrwxrwx 1 501 games 76 Sep 19 16:06 pytorch_model-00016-of-00028.bin -> ../../blobs/35773f9415d00511d7981224e08ef73b0b4b252dab0cc9a1fa5c56d5a671bb3b
lrwxrwxrwx 1 501 games 76 Sep 19 16:09 pytorch_model-00017-of-00028.bin -> ../../blobs/7bbd3b172fe86306d0338824cf2cec330d93ab6fbac5a60ef1b4f4d60bbdf7af
lrwxrwxrwx 1 501 games 76 Sep 19 16:12 pytorch_model-00018-of-00028.bin -> ../../blobs/b26c3627cec92f3eeb876aa6610760d8f0440df0db0cef2cacecb9cc97fce74e
lrwxrwxrwx 1 501 games 76 Sep 19 16:15 pytorch_model-00019-of-00028.bin -> ../../blobs/490ddcb4b3d6ca514559eb01820843dc516c0379f7a9b2473ff947074d0cf035
lrwxrwxrwx 1 501 games 76 Sep 19 16:19 pytorch_model-00020-of-00028.bin -> ../../blobs/8694ef104ecde2e006388ff2c39a0a8115ae6f8ffb65a7efe7cf843d81802977
lrwxrwxrwx 1 501 games 76 Sep 19 16:22 pytorch_model-00021-of-00028.bin -> ../../blobs/a0357d3a092a7d2706dbf55370e3348dcaa3f9b4488ccb6ba5aee5f4dffcb09c
lrwxrwxrwx 1 501 games 76 Sep 19 16:25 pytorch_model-00022-of-00028.bin -> ../../blobs/0606ff197a8d2bd6856deaec399b54b8cdec76871e05f6fc2f58dd33dee5eda3
lrwxrwxrwx 1 501 games 76 Sep 19 16:28 pytorch_model-00023-of-00028.bin -> ../../blobs/a1f93ea617fe335ee65683837ea6b832064be574a52b136d2f4bb51401a1c2cf
lrwxrwxrwx 1 501 games 76 Sep 19 16:32 pytorch_model-00024-of-00028.bin -> ../../blobs/f66af7e885b2bd3652aaa0a133de73ada060f6070a5ccf7fd92d8838c061be91
lrwxrwxrwx 1 501 games 76 Sep 19 16:35 pytorch_model-00025-of-00028.bin -> ../../blobs/9b815e8e8a3b5a3c58cd9552f5147186abf7994ce20b1375d0b5f6fc0423dd17
lrwxrwxrwx 1 501 games 76 Sep 19 16:39 pytorch_model-00026-of-00028.bin -> ../../blobs/8295c1a27243c062ef8d5eeb523852d24624734ee4a62e93ab2441f136752bcd
lrwxrwxrwx 1 501 games 76 Sep 19 16:42 pytorch_model-00027-of-00028.bin -> ../../blobs/70b687a3df6cfa98293491aff473f8a007f9a2294cc21c249e0c14299a0c501c
lrwxrwxrwx 1 501 games 76 Sep 19 16:44 pytorch_model-00028-of-00028.bin -> ../../blobs/a1dae44eef4f578cafda912abc63a332a073eb2f9680c35e84b547ef02adf9db
lrwxrwxrwx 1 501 games 52 Sep 19 15:15 pytorch_model.bin.index.json -> ../../blobs/989e0277ef1827d92a36a406f7a2e609778fd833
lrwxrwxrwx 1 501 games 52 Sep 19 15:14 special_tokens_map.json -> ../../blobs/c0fc1d580b4781190e696deec8c8c6834186f7ce
lrwxrwxrwx 1 501 games 52 Sep 19 15:14 tokenizer_config.json -> ../../blobs/400e3de6ffc3884ec3c158a046f6a04da00ef3ca
lrwxrwxrwx 1 501 games 52 Sep 19 15:14 tokenizer.json -> ../../blobs/acadf9c42fc05e32d5d5e81578a7257e08678d32
lrwxrwxrwx 1 501 games 76 Sep 19 15:14 tokenizer.model -> ../../blobs/6fd7e445833dd0889206aba242c2a51ecbae2437fd328d1759a35475fd8c0423

第一步转换原始Llama参数为hf格式：
python convert_llama_weights_to_hf.py --input_dir ./Llama-13b/ --model_size 13B --output_dir ./llama-13b-hf
[2023-09-21 10:15:26,139] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. If you see this, DO NOT PANIC! This is expected, and simply means that the legacy (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set legacy=False. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
Fetching all parameters from the checkpoint at ./Llama-13b/.
Loading the checkpoint in a Llama model.
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 41/41 [00:19<00:00, 2.07it/s]
Saving in the Transformers format.

第二步合并差值模型：
python3 -m apply_delta --base-model-path ./llama-13b-hf/ --target-model-path ./merged_ziya_13b --delta-path models--IDEA-CCNL--Ziya-LLaMA-13B-v1/snapshots/64d931f346e1a49ea3bbca07a83137075bab1c66/
[2023-09-21 10:28:43,876] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Loading the delta weights from models--IDEA-CCNL--Ziya-LLaMA-13B-v1/snapshots/64d931f346e1a49ea3bbca07a83137075bab1c66/
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. If you see this, DO NOT PANIC! This is expected, and simply means that the legacy (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set legacy=False. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 28/28 [00:33<00:00, 1.20s/it]
Loading the base model from ./llama-13b-hf/
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:17<00:00, 5.76s/it]
Applying the delta
Applying delta: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 363/363 [00:02<00:00, 126.78it/s]
Saving the target model to ./merged_ziya_13b

wuxiaojun

Fengshenbang-LM org Sep 21, 2023

首先得确保ziya-llama-v1的推理没问题，才能确保ziya-blip-v1的推理没问题（因为是基于ziya-llama-v1训练的），你现在ziya-llama-v1的推理正常吗？

jamestang2190

Sep 21, 2023

首先得确保ziya-llama-v1的推理没问题，才能确保ziya-blip-v1的推理没问题（因为是基于ziya-llama-v1训练的），你现在ziya-llama-v1的推理正常吗？

就是ziya-llama-v1推理不正常，使用参数转换后的原始llama模型，推理是正常的，所以我怀疑要么是huggingface上下载的ziya-llama-v1差值模型不对，要么是基座模型需要llama-13b-chat那个模型，可以麻烦作者确认下基座模型是哪一个吗？感谢

jamestang2190

Sep 21, 2023

首先得确保ziya-llama-v1的推理没问题，才能确保ziya-blip-v1的推理没问题（因为是基于ziya-llama-v1训练的），你现在ziya-llama-v1的推理正常吗？

又下载了llama-13b-chat模型，进行参数合并之后，效果还是不对。可否指导下是哪里出了问题，是huggingface下载的差值模型有问题吗？

:帮我写一份去西安的旅游计划 :肤 Nemisieisie艟tw
WiesIPtw Wiesface Hoffkup Werner earthrigSR彟 Clear Search菠 earthzia faciernoeaudp mach sc拮舛眶ierno floatrig tun
fac tuEventListener float纸 _纸 __
podrig server清馨rig nu Clear tun
isie
滑rig 圣 nu Clear 圣曾 Tu曾EventListener圣 Profile
isierig熨菠做rig nu爱dy tun
rig dr纸rig纸rig tu曾una爱rig滑priv nu dr爱 Profile滑柔垢菠

wuxiaojun

Fengshenbang-LM org Sep 21, 2023

不是就是用的ziya-llama-v1模型，因为ziya-llama-v1这个模型不是我训练的，所以我也无法确定你在合并中出现了什么问题，你可以去ziya-llama-v1那边提个issue问一下

jamestang2190

Sep 21, 2023

不是就是用的ziya-llama-v1模型，因为ziya-llama-v1这个模型不是我训练的，所以我也无法确定你在合并中出现了什么问题，你可以去ziya-llama-v1那边提个issue问一下

好的，谢谢