Hugging Face's logo

mlc-ai
/
Qwen2-1.5B-Instruct-q4f16_1-MLC

MLC-LLM
web-llm
Model card Files Community
Qwen2-1.5B-Instruct-q4f16_1-MLC / logs.txt
riczhou's picture
riczhou
Initial commit
5ffb52e verified
raw
history blame contribute delete
103 kB
/opt/conda/envs/py310/bin/python -m mlc_llm gen_config /models/Qwen2-1.5B-Instruct --quantization q4f16_1 --conv-template chatml --output /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC
[2024-06-06 23:39:14] INFO auto_config.py:116: Found model configuration: /models/Qwen2-1.5B-Instruct/config.json
[2024-06-06 23:39:14] INFO auto_config.py:154: Found model type: qwen2. Use `--model-type` to override.
[2024-06-06 23:39:14] INFO qwen2_model.py:49: context_window_size not found in config.json. Falling back to max_position_embeddings (32768)
[2024-06-06 23:39:14] INFO qwen2_model.py:66: prefill_chunk_size defaults to 2048
[2024-06-06 23:39:14] INFO config.py:107: Overriding max_batch_size from 1 to 80
[2024-06-06 23:39:14] INFO gen_config.py:143: [generation_config.json] Setting bos_token_id: 151643
[2024-06-06 23:39:14] INFO gen_config.py:143: [generation_config.json] Setting pad_token_id: 151643
[2024-06-06 23:39:14] INFO gen_config.py:143: [generation_config.json] Setting eos_token_id: [151645, 151643]
[2024-06-06 23:39:14] INFO gen_config.py:143: [generation_config.json] Setting repetition_penalty: 1.1
[2024-06-06 23:39:14] INFO gen_config.py:143: [generation_config.json] Setting temperature: 0.7
[2024-06-06 23:39:14] INFO gen_config.py:143: [generation_config.json] Setting top_p: 0.8
[2024-06-06 23:39:14] INFO gen_config.py:157: Not found tokenizer config: /models/Qwen2-1.5B-Instruct/tokenizer.model
[2024-06-06 23:39:14] INFO gen_config.py:155: Found tokenizer config: /models/Qwen2-1.5B-Instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC/tokenizer.json
[2024-06-06 23:39:14] INFO gen_config.py:155: Found tokenizer config: /models/Qwen2-1.5B-Instruct/vocab.json. Copying to /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC/vocab.json
[2024-06-06 23:39:14] INFO gen_config.py:155: Found tokenizer config: /models/Qwen2-1.5B-Instruct/merges.txt. Copying to /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC/merges.txt
[2024-06-06 23:39:14] INFO gen_config.py:157: Not found tokenizer config: /models/Qwen2-1.5B-Instruct/added_tokens.json
[2024-06-06 23:39:14] INFO gen_config.py:155: Found tokenizer config: /models/Qwen2-1.5B-Instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC/tokenizer_config.json
[2024-06-06 23:39:14] INFO gen_config.py:216: Detected tokenizer info: {'token_postproc_method': 'byte_level', 'prepend_space_in_encode': False, 'strip_space_in_decode': False}
[2024-06-06 23:39:14] INFO gen_config.py:32: [System default] Setting presence_penalty: 0.0
[2024-06-06 23:39:14] INFO gen_config.py:32: [System default] Setting frequency_penalty: 0.0
[2024-06-06 23:39:14] INFO gen_config.py:223: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC/mlc-chat-config.json
/opt/conda/envs/py310/bin/python -m mlc_llm convert_weight /models/Qwen2-1.5B-Instruct --quantization q4f16_1 --output /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC
[2024-06-06 23:39:15] INFO auto_config.py:116: Found model configuration: /models/Qwen2-1.5B-Instruct/config.json
[2024-06-06 23:39:17] INFO auto_device.py:79: Found device: cuda:0
[2024-06-06 23:39:18] INFO auto_device.py:88: Not found device: rocm:0
[2024-06-06 23:39:20] INFO auto_device.py:88: Not found device: metal:0
[2024-06-06 23:39:22] INFO auto_device.py:79: Found device: vulkan:0
[2024-06-06 23:39:22] INFO auto_device.py:79: Found device: vulkan:1
[2024-06-06 23:39:22] INFO auto_device.py:79: Found device: vulkan:2
[2024-06-06 23:39:22] INFO auto_device.py:79: Found device: vulkan:3
[2024-06-06 23:39:23] INFO auto_device.py:88: Not found device: opencl:0
[2024-06-06 23:39:23] INFO auto_device.py:35: Using device: cuda:0
[2024-06-06 23:39:23] INFO auto_weight.py:71: Finding weights in: /models/Qwen2-1.5B-Instruct
[2024-06-06 23:39:23] INFO auto_weight.py:137: Not found Huggingface PyTorch
[2024-06-06 23:39:24] INFO auto_weight.py:161: Found source weight format: huggingface-safetensor. Source configuration: /models/Qwen2-1.5B-Instruct/model.safetensors.index.json
[2024-06-06 23:39:24] INFO auto_weight.py:107: Using source weight configuration: /models/Qwen2-1.5B-Instruct/model.safetensors.index.json. Use `--source` to override.
[2024-06-06 23:39:24] INFO auto_weight.py:111: Using source weight format: huggingface-safetensor. Use `--source-format` to override.
[2024-06-06 23:39:24] INFO auto_config.py:154: Found model type: qwen2. Use `--model-type` to override.
[2024-06-06 23:39:24] INFO qwen2_model.py:49: context_window_size not found in config.json. Falling back to max_position_embeddings (32768)
[2024-06-06 23:39:24] INFO qwen2_model.py:66: prefill_chunk_size defaults to 2048
Traceback (most recent call last):
File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/opt/conda/envs/py310/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 64, in <module>
main()
File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/__main__.py", line 37, in main
cli.main(sys.argv[2:])
File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/cli/convert_weight.py", line 88, in main
convert_weight(
File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 181, in convert_weight
_convert_args(args)
File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 145, in _convert_args
tvmjs.dump_ndarray_cache(
File "/opt/conda/envs/py310/lib/python3.10/site-packages/tvm/contrib/tvmjs.py", line 272, in dump_ndarray_cache
for k, origin_v in param_generator:
File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/interface/convert_weight.py", line 122, in _param_generator
loader = LOADER[args.source_format](
File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/loader/huggingface_loader.py", line 99, in __init__
check_parameter_usage(extern_param_map, set(self.torch_to_path.keys()))
File "/opt/conda/envs/py310/lib/python3.10/site-packages/mlc_llm/loader/utils.py", line 33, in check_parameter_usage
raise ValueError(
ValueError: The following extern parameters do not exist in the weight files:
lm_head.weight
Weight conversion with arguments:
--config /models/Qwen2-1.5B-Instruct/config.json
--quantization GroupQuantize(name='q4f16_1', kind='group-quant', group_size=32, quantize_dtype='int4', storage_dtype='uint32', model_dtype='float16', linear_weight_layout='NK', quantize_embedding=True, quantize_final_fc=True, num_elem_per_storage=8, num_storage_per_group=4, max_int_value=7)
--model-type qwen2
--device cuda:0
--source /models/Qwen2-1.5B-Instruct/model.safetensors.index.json
--source-format huggingface-safetensor
--output /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC
Start storing to cache /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC
/home/rickzhou/miniconda3/envs/mlc/bin/python -m mlc_llm gen_config /ssd2/models/Qwen2-1.5B-Instruct --quantization q4f16_1 --conv-template chatml --output /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC
[2024-06-06 22:37:11] INFO auto_config.py:116: Found model configuration: /ssd2/models/Qwen2-1.5B-Instruct/config.json
[2024-06-06 22:37:11] INFO auto_config.py:154: Found model type: qwen2. Use `--model-type` to override.
[2024-06-06 22:37:11] INFO qwen2_model.py:50: context_window_size not found in config.json. Falling back to max_position_embeddings (32768)
[2024-06-06 22:37:11] INFO qwen2_model.py:67: prefill_chunk_size defaults to 2048
[2024-06-06 22:37:11] INFO config.py:107: Overriding max_batch_size from 1 to 80
[2024-06-06 22:37:11] INFO gen_config.py:143: [generation_config.json] Setting bos_token_id: 151643
[2024-06-06 22:37:11] INFO gen_config.py:143: [generation_config.json] Setting pad_token_id: 151643
[2024-06-06 22:37:11] INFO gen_config.py:143: [generation_config.json] Setting eos_token_id: [151645, 151643]
[2024-06-06 22:37:11] INFO gen_config.py:143: [generation_config.json] Setting repetition_penalty: 1.1
[2024-06-06 22:37:11] INFO gen_config.py:143: [generation_config.json] Setting temperature: 0.7
[2024-06-06 22:37:11] INFO gen_config.py:143: [generation_config.json] Setting top_p: 0.8
[2024-06-06 22:37:11] INFO gen_config.py:157: Not found tokenizer config: /ssd2/models/Qwen2-1.5B-Instruct/tokenizer.model
[2024-06-06 22:37:11] INFO gen_config.py:155: Found tokenizer config: /ssd2/models/Qwen2-1.5B-Instruct/tokenizer.json. Copying to /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC/tokenizer.json
[2024-06-06 22:37:11] INFO gen_config.py:155: Found tokenizer config: /ssd2/models/Qwen2-1.5B-Instruct/vocab.json. Copying to /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC/vocab.json
[2024-06-06 22:37:11] INFO gen_config.py:155: Found tokenizer config: /ssd2/models/Qwen2-1.5B-Instruct/merges.txt. Copying to /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC/merges.txt
[2024-06-06 22:37:11] INFO gen_config.py:157: Not found tokenizer config: /ssd2/models/Qwen2-1.5B-Instruct/added_tokens.json
[2024-06-06 22:37:11] INFO gen_config.py:155: Found tokenizer config: /ssd2/models/Qwen2-1.5B-Instruct/tokenizer_config.json. Copying to /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC/tokenizer_config.json
[2024-06-06 22:37:11] INFO gen_config.py:216: Detected tokenizer info: {'token_postproc_method': 'byte_level', 'prepend_space_in_encode': False, 'strip_space_in_decode': False}
[2024-06-06 22:37:11] INFO gen_config.py:32: [System default] Setting presence_penalty: 0.0
[2024-06-06 22:37:11] INFO gen_config.py:32: [System default] Setting frequency_penalty: 0.0
[2024-06-06 22:37:11] INFO gen_config.py:223: Dumping configuration file to: /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC/mlc-chat-config.json
/home/rickzhou/miniconda3/envs/mlc/bin/python -m mlc_llm convert_weight /ssd2/models/Qwen2-1.5B-Instruct --quantization q4f16_1 --output /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC
[2024-06-06 22:37:12] INFO auto_config.py:116: Found model configuration: /ssd2/models/Qwen2-1.5B-Instruct/config.json
[2024-06-06 22:37:13] INFO auto_device.py:79: Found device: cuda:0
[2024-06-06 22:37:13] INFO auto_device.py:79: Found device: cuda:1
[2024-06-06 22:37:14] INFO auto_device.py:88: Not found device: rocm:0
[2024-06-06 22:37:15] INFO auto_device.py:88: Not found device: metal:0
[2024-06-06 22:37:17] INFO auto_device.py:79: Found device: vulkan:0
[2024-06-06 22:37:17] INFO auto_device.py:79: Found device: vulkan:1
[2024-06-06 22:37:17] INFO auto_device.py:79: Found device: vulkan:2
[2024-06-06 22:37:18] INFO auto_device.py:88: Not found device: opencl:0
[2024-06-06 22:37:18] INFO auto_device.py:35: Using device: cuda:0
[2024-06-06 22:37:18] INFO auto_weight.py:71: Finding weights in: /ssd2/models/Qwen2-1.5B-Instruct
[2024-06-06 22:37:18] INFO auto_weight.py:137: Not found Huggingface PyTorch
[2024-06-06 22:37:18] INFO auto_weight.py:144: Found source weight format: huggingface-safetensor. Source configuration: /ssd2/models/Qwen2-1.5B-Instruct/model.safetensors.index.json
[2024-06-06 22:37:18] INFO auto_weight.py:107: Using source weight configuration: /ssd2/models/Qwen2-1.5B-Instruct/model.safetensors.index.json. Use `--source` to override.
[2024-06-06 22:37:18] INFO auto_weight.py:111: Using source weight format: huggingface-safetensor. Use `--source-format` to override.
[2024-06-06 22:37:18] INFO auto_config.py:154: Found model type: qwen2. Use `--model-type` to override.
[2024-06-06 22:37:18] INFO qwen2_model.py:50: context_window_size not found in config.json. Falling back to max_position_embeddings (32768)
[2024-06-06 22:37:18] INFO qwen2_model.py:67: prefill_chunk_size defaults to 2048
Weight conversion with arguments:
--config /ssd2/models/Qwen2-1.5B-Instruct/config.json
--quantization GroupQuantize(name='q4f16_1', kind='group-quant', group_size=32, quantize_dtype='int4', storage_dtype='uint32', model_dtype='float16', linear_weight_layout='NK', quantize_embedding=True, quantize_final_fc=True, num_elem_per_storage=8, num_storage_per_group=4, max_int_value=7)
--model-type qwen2
--device cuda:0
--source /ssd2/models/Qwen2-1.5B-Instruct/model.safetensors.index.json
--source-format huggingface-safetensor
--output /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC
Start storing to cache /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC
0%| | 0/198 [00:00<?, ?it/s] [2024-06-06 22:37:21] INFO huggingface_loader.py:185: Loading HF parameters from: /ssd2/models/Qwen2-1.5B-Instruct/model.safetensors
0%| | 0/198 [00:00<?, ?it/s] [2024-06-06 22:37:24] INFO group_quantization.py:217: Compiling quantize function for key: ((151936, 1536), float16, cuda, axis=1, output_transpose=False)
0%| | 0/198 [00:02<?, ?it/s] [2024-06-06 22:37:25] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.embed_tokens.q_weight", shape: (151936, 192), dtype: uint32
0%| | 0/198 [00:03<?, ?it/s] [2024-06-06 22:37:25] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.embed_tokens.q_scale", shape: (151936, 48), dtype: float16
0%| | 0/198 [00:04<?, ?it/s] 1%| | 1/198 [00:04<13:43, 4.18s/it] [2024-06-06 22:37:25] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.0.input_layernorm.weight", shape: (1536,), dtype: float16
1%| | 1/198 [00:04<13:43, 4.18s/it] [2024-06-06 22:37:25] INFO group_quantization.py:217: Compiling quantize function for key: ((1536, 8960), float16, cuda, axis=1, output_transpose=False)
1%| | 1/198 [00:04<13:43, 4.18s/it] [2024-06-06 22:37:26] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.0.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
1%| | 1/198 [00:04<13:43, 4.18s/it] [2024-06-06 22:37:26] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.0.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
1%| | 1/198 [00:04<13:43, 4.18s/it] 2%|▏ | 3/198 [00:04<04:04, 1.25s/it] [2024-06-06 22:37:26] INFO group_quantization.py:217: Compiling quantize function for key: ((17920, 1536), float16, cuda, axis=1, output_transpose=False)
2%|▏ | 3/198 [00:04<04:04, 1.25s/it] [2024-06-06 22:37:26] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.0.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
2%|▏ | 3/198 [00:05<04:04, 1.25s/it] [2024-06-06 22:37:26] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.0.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
2%|▏ | 3/198 [00:05<04:04, 1.25s/it] 2%|▏ | 4/198 [00:05<03:17, 1.02s/it] [2024-06-06 22:37:26] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.0.post_attention_layernorm.weight", shape: (1536,), dtype: float16
2%|▏ | 4/198 [00:05<03:17, 1.02s/it] [2024-06-06 22:37:26] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.0.self_attn.c_attn.bias", shape: (2048,), dtype: float16
2%|▏ | 4/198 [00:05<03:17, 1.02s/it] [2024-06-06 22:37:26] INFO group_quantization.py:217: Compiling quantize function for key: ((2048, 1536), float16, cuda, axis=1, output_transpose=False)
2%|▏ | 4/198 [00:05<03:17, 1.02s/it] [2024-06-06 22:37:27] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.0.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
2%|▏ | 4/198 [00:05<03:17, 1.02s/it] [2024-06-06 22:37:27] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.0.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
2%|▏ | 4/198 [00:05<03:17, 1.02s/it] 4%|▎ | 7/198 [00:05<01:35, 2.01it/s] [2024-06-06 22:37:27] INFO group_quantization.py:217: Compiling quantize function for key: ((1536, 1536), float16, cuda, axis=1, output_transpose=False)
4%|▎ | 7/198 [00:05<01:35, 2.01it/s] [2024-06-06 22:37:27] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.0.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
4%|▎ | 7/198 [00:06<01:35, 2.01it/s] [2024-06-06 22:37:27] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.0.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
4%|▎ | 7/198 [00:06<01:35, 2.01it/s] 4%|▍ | 8/198 [00:06<01:30, 2.11it/s] [2024-06-06 22:37:27] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.1.input_layernorm.weight", shape: (1536,), dtype: float16
4%|▍ | 8/198 [00:06<01:30, 2.11it/s] [2024-06-06 22:37:27] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.1.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
4%|▍ | 8/198 [00:06<01:30, 2.11it/s] [2024-06-06 22:37:27] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.1.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
4%|▍ | 8/198 [00:06<01:30, 2.11it/s] [2024-06-06 22:37:27] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.1.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
4%|▍ | 8/198 [00:06<01:30, 2.11it/s] [2024-06-06 22:37:27] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.1.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
4%|▍ | 8/198 [00:06<01:30, 2.11it/s] 6%|▌ | 11/198 [00:06<00:52, 3.57it/s] [2024-06-06 22:37:27] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.1.post_attention_layernorm.weight", shape: (1536,), dtype: float16
6%|▌ | 11/198 [00:06<00:52, 3.57it/s] [2024-06-06 22:37:27] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.1.self_attn.c_attn.bias", shape: (2048,), dtype: float16
6%|▌ | 11/198 [00:06<00:52, 3.57it/s] [2024-06-06 22:37:27] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.1.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
6%|▌ | 11/198 [00:06<00:52, 3.57it/s] [2024-06-06 22:37:27] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.1.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
6%|▌ | 11/198 [00:06<00:52, 3.57it/s] [2024-06-06 22:37:27] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.1.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
6%|▌ | 11/198 [00:06<00:52, 3.57it/s] [2024-06-06 22:37:27] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.1.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
6%|▌ | 11/198 [00:06<00:52, 3.57it/s] [2024-06-06 22:37:27] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.10.input_layernorm.weight", shape: (1536,), dtype: float16
6%|▌ | 11/198 [00:06<00:52, 3.57it/s] [2024-06-06 22:37:27] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.10.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
6%|▌ | 11/198 [00:06<00:52, 3.57it/s] [2024-06-06 22:37:27] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.10.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
6%|▌ | 11/198 [00:06<00:52, 3.57it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.10.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
6%|▌ | 11/198 [00:06<00:52, 3.57it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.10.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
6%|▌ | 11/198 [00:06<00:52, 3.57it/s] 9%|▉ | 18/198 [00:06<00:23, 7.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.10.post_attention_layernorm.weight", shape: (1536,), dtype: float16
9%|▉ | 18/198 [00:06<00:23, 7.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.10.self_attn.c_attn.bias", shape: (2048,), dtype: float16
9%|▉ | 18/198 [00:06<00:23, 7.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.10.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
9%|▉ | 18/198 [00:06<00:23, 7.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.10.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
9%|▉ | 18/198 [00:06<00:23, 7.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.10.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
9%|▉ | 18/198 [00:06<00:23, 7.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.10.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
9%|▉ | 18/198 [00:06<00:23, 7.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.11.input_layernorm.weight", shape: (1536,), dtype: float16
9%|▉ | 18/198 [00:06<00:23, 7.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.11.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
9%|▉ | 18/198 [00:06<00:23, 7.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.11.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
9%|▉ | 18/198 [00:06<00:23, 7.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.11.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
9%|▉ | 18/198 [00:06<00:23, 7.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.11.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
9%|▉ | 18/198 [00:06<00:23, 7.68it/s] 13%|█▎ | 25/198 [00:06<00:14, 11.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.11.post_attention_layernorm.weight", shape: (1536,), dtype: float16
13%|█▎ | 25/198 [00:06<00:14, 11.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.11.self_attn.c_attn.bias", shape: (2048,), dtype: float16
13%|█▎ | 25/198 [00:06<00:14, 11.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.11.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
13%|█▎ | 25/198 [00:06<00:14, 11.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.11.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
13%|█▎ | 25/198 [00:06<00:14, 11.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.11.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
13%|█▎ | 25/198 [00:06<00:14, 11.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.11.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
13%|█▎ | 25/198 [00:06<00:14, 11.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.12.input_layernorm.weight", shape: (1536,), dtype: float16
13%|█▎ | 25/198 [00:06<00:14, 11.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.12.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
13%|█▎ | 25/198 [00:06<00:14, 11.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.12.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
13%|█▎ | 25/198 [00:06<00:14, 11.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.12.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
13%|█▎ | 25/198 [00:06<00:14, 11.68it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.12.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
13%|█▎ | 25/198 [00:07<00:14, 11.68it/s] 16%|█▌ | 32/198 [00:07<00:10, 15.35it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.12.post_attention_layernorm.weight", shape: (1536,), dtype: float16
16%|█▌ | 32/198 [00:07<00:10, 15.35it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.12.self_attn.c_attn.bias", shape: (2048,), dtype: float16
16%|█▌ | 32/198 [00:07<00:10, 15.35it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.12.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
16%|█▌ | 32/198 [00:07<00:10, 15.35it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.12.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
16%|█▌ | 32/198 [00:07<00:10, 15.35it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.12.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
16%|█▌ | 32/198 [00:07<00:10, 15.35it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.12.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
16%|█▌ | 32/198 [00:07<00:10, 15.35it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.13.input_layernorm.weight", shape: (1536,), dtype: float16
16%|█▌ | 32/198 [00:07<00:10, 15.35it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.13.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
16%|█▌ | 32/198 [00:07<00:10, 15.35it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.13.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
16%|█▌ | 32/198 [00:07<00:10, 15.35it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.13.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
16%|█▌ | 32/198 [00:07<00:10, 15.35it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.13.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
16%|█▌ | 32/198 [00:07<00:10, 15.35it/s] 20%|█▉ | 39/198 [00:07<00:08, 18.57it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.13.post_attention_layernorm.weight", shape: (1536,), dtype: float16
20%|█▉ | 39/198 [00:07<00:08, 18.57it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.13.self_attn.c_attn.bias", shape: (2048,), dtype: float16
20%|█▉ | 39/198 [00:07<00:08, 18.57it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.13.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
20%|█▉ | 39/198 [00:07<00:08, 18.57it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.13.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
20%|█▉ | 39/198 [00:07<00:08, 18.57it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.13.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
20%|█▉ | 39/198 [00:07<00:08, 18.57it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.13.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
20%|█▉ | 39/198 [00:07<00:08, 18.57it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.14.input_layernorm.weight", shape: (1536,), dtype: float16
20%|█▉ | 39/198 [00:07<00:08, 18.57it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.14.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
20%|█▉ | 39/198 [00:07<00:08, 18.57it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.14.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
20%|█▉ | 39/198 [00:07<00:08, 18.57it/s] [2024-06-06 22:37:28] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.14.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
20%|█▉ | 39/198 [00:07<00:08, 18.57it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.14.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
20%|█▉ | 39/198 [00:07<00:08, 18.57it/s] 23%|██▎ | 46/198 [00:07<00:07, 21.23it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.14.post_attention_layernorm.weight", shape: (1536,), dtype: float16
23%|██▎ | 46/198 [00:07<00:07, 21.23it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.14.self_attn.c_attn.bias", shape: (2048,), dtype: float16
23%|██▎ | 46/198 [00:07<00:07, 21.23it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.14.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
23%|██▎ | 46/198 [00:07<00:07, 21.23it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.14.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
23%|██▎ | 46/198 [00:07<00:07, 21.23it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.14.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
23%|██▎ | 46/198 [00:07<00:07, 21.23it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.14.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
23%|██▎ | 46/198 [00:07<00:07, 21.23it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.15.input_layernorm.weight", shape: (1536,), dtype: float16
23%|██▎ | 46/198 [00:07<00:07, 21.23it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.15.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
23%|██▎ | 46/198 [00:07<00:07, 21.23it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.15.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
23%|██▎ | 46/198 [00:07<00:07, 21.23it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.15.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
23%|██▎ | 46/198 [00:07<00:07, 21.23it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.15.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
23%|██▎ | 46/198 [00:07<00:07, 21.23it/s] 27%|██▋ | 53/198 [00:07<00:06, 23.32it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.15.post_attention_layernorm.weight", shape: (1536,), dtype: float16
27%|██▋ | 53/198 [00:07<00:06, 23.32it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.15.self_attn.c_attn.bias", shape: (2048,), dtype: float16
27%|██▋ | 53/198 [00:07<00:06, 23.32it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.15.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
27%|██▋ | 53/198 [00:07<00:06, 23.32it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.15.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
27%|██▋ | 53/198 [00:07<00:06, 23.32it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.15.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
27%|██▋ | 53/198 [00:07<00:06, 23.32it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.15.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
27%|██▋ | 53/198 [00:07<00:06, 23.32it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.16.input_layernorm.weight", shape: (1536,), dtype: float16
27%|██▋ | 53/198 [00:07<00:06, 23.32it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.16.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
27%|██▋ | 53/198 [00:07<00:06, 23.32it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.16.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
27%|██▋ | 53/198 [00:07<00:06, 23.32it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.16.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
27%|██▋ | 53/198 [00:07<00:06, 23.32it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.16.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
27%|██▋ | 53/198 [00:07<00:06, 23.32it/s] 30%|███ | 60/198 [00:07<00:05, 24.53it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.16.post_attention_layernorm.weight", shape: (1536,), dtype: float16
30%|███ | 60/198 [00:07<00:05, 24.53it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.16.self_attn.c_attn.bias", shape: (2048,), dtype: float16
30%|███ | 60/198 [00:07<00:05, 24.53it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.16.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
30%|███ | 60/198 [00:08<00:05, 24.53it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.16.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
30%|███ | 60/198 [00:08<00:05, 24.53it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.16.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
30%|███ | 60/198 [00:08<00:05, 24.53it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.16.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
30%|███ | 60/198 [00:08<00:05, 24.53it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.17.input_layernorm.weight", shape: (1536,), dtype: float16
30%|███ | 60/198 [00:08<00:05, 24.53it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.17.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
30%|███ | 60/198 [00:08<00:05, 24.53it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.17.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
30%|███ | 60/198 [00:08<00:05, 24.53it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.17.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
30%|███ | 60/198 [00:08<00:05, 24.53it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.17.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
30%|███ | 60/198 [00:08<00:05, 24.53it/s] 34%|███▍ | 67/198 [00:08<00:05, 25.67it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.17.post_attention_layernorm.weight", shape: (1536,), dtype: float16
34%|███▍ | 67/198 [00:08<00:05, 25.67it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.17.self_attn.c_attn.bias", shape: (2048,), dtype: float16
34%|███▍ | 67/198 [00:08<00:05, 25.67it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.17.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
34%|███▍ | 67/198 [00:08<00:05, 25.67it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.17.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
34%|███▍ | 67/198 [00:08<00:05, 25.67it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.17.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
34%|███▍ | 67/198 [00:08<00:05, 25.67it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.17.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
34%|███▍ | 67/198 [00:08<00:05, 25.67it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.18.input_layernorm.weight", shape: (1536,), dtype: float16
34%|███▍ | 67/198 [00:08<00:05, 25.67it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.18.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
34%|███▍ | 67/198 [00:08<00:05, 25.67it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.18.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
34%|███▍ | 67/198 [00:08<00:05, 25.67it/s] [2024-06-06 22:37:29] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.18.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
34%|███▍ | 67/198 [00:08<00:05, 25.67it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.18.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
34%|███▍ | 67/198 [00:08<00:05, 25.67it/s] 37%|███▋ | 74/198 [00:08<00:04, 26.34it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.18.post_attention_layernorm.weight", shape: (1536,), dtype: float16
37%|███▋ | 74/198 [00:08<00:04, 26.34it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.18.self_attn.c_attn.bias", shape: (2048,), dtype: float16
37%|███▋ | 74/198 [00:08<00:04, 26.34it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.18.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
37%|███▋ | 74/198 [00:08<00:04, 26.34it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.18.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
37%|███▋ | 74/198 [00:08<00:04, 26.34it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.18.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
37%|███▋ | 74/198 [00:08<00:04, 26.34it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.18.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
37%|███▋ | 74/198 [00:08<00:04, 26.34it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.19.input_layernorm.weight", shape: (1536,), dtype: float16
37%|███▋ | 74/198 [00:08<00:04, 26.34it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.19.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
37%|███▋ | 74/198 [00:08<00:04, 26.34it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.19.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
37%|███▋ | 74/198 [00:08<00:04, 26.34it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.19.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
37%|███▋ | 74/198 [00:08<00:04, 26.34it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.19.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
37%|███▋ | 74/198 [00:08<00:04, 26.34it/s] 41%|████ | 81/198 [00:08<00:04, 27.09it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.19.post_attention_layernorm.weight", shape: (1536,), dtype: float16
41%|████ | 81/198 [00:08<00:04, 27.09it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.19.self_attn.c_attn.bias", shape: (2048,), dtype: float16
41%|████ | 81/198 [00:08<00:04, 27.09it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.19.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
41%|████ | 81/198 [00:08<00:04, 27.09it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.19.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
41%|████ | 81/198 [00:08<00:04, 27.09it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.19.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
41%|████ | 81/198 [00:08<00:04, 27.09it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.19.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
41%|████ | 81/198 [00:08<00:04, 27.09it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.2.input_layernorm.weight", shape: (1536,), dtype: float16
41%|████ | 81/198 [00:08<00:04, 27.09it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.2.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
41%|████ | 81/198 [00:08<00:04, 27.09it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.2.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
41%|████ | 81/198 [00:08<00:04, 27.09it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.2.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
41%|████ | 81/198 [00:08<00:04, 27.09it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.2.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
41%|████ | 81/198 [00:08<00:04, 27.09it/s] 44%|████▍ | 88/198 [00:08<00:03, 27.59it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.2.post_attention_layernorm.weight", shape: (1536,), dtype: float16
44%|████▍ | 88/198 [00:08<00:03, 27.59it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.2.self_attn.c_attn.bias", shape: (2048,), dtype: float16
44%|████▍ | 88/198 [00:08<00:03, 27.59it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.2.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
44%|████▍ | 88/198 [00:08<00:03, 27.59it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.2.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
44%|████▍ | 88/198 [00:08<00:03, 27.59it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.2.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
44%|████▍ | 88/198 [00:09<00:03, 27.59it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.2.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
44%|████▍ | 88/198 [00:09<00:03, 27.59it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.20.input_layernorm.weight", shape: (1536,), dtype: float16
44%|████▍ | 88/198 [00:09<00:03, 27.59it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.20.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
44%|████▍ | 88/198 [00:09<00:03, 27.59it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.20.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
44%|████▍ | 88/198 [00:09<00:03, 27.59it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.20.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
44%|████▍ | 88/198 [00:09<00:03, 27.59it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.20.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
44%|████▍ | 88/198 [00:09<00:03, 27.59it/s] 48%|████▊ | 95/198 [00:09<00:03, 27.98it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.20.post_attention_layernorm.weight", shape: (1536,), dtype: float16
48%|████▊ | 95/198 [00:09<00:03, 27.98it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.20.self_attn.c_attn.bias", shape: (2048,), dtype: float16
48%|████▊ | 95/198 [00:09<00:03, 27.98it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.20.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
48%|████▊ | 95/198 [00:09<00:03, 27.98it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.20.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
48%|████▊ | 95/198 [00:09<00:03, 27.98it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.20.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
48%|████▊ | 95/198 [00:09<00:03, 27.98it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.20.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
48%|████▊ | 95/198 [00:09<00:03, 27.98it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.21.input_layernorm.weight", shape: (1536,), dtype: float16
48%|████▊ | 95/198 [00:09<00:03, 27.98it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.21.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
48%|████▊ | 95/198 [00:09<00:03, 27.98it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.21.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
48%|████▊ | 95/198 [00:09<00:03, 27.98it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.21.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
48%|████▊ | 95/198 [00:09<00:03, 27.98it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.21.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
48%|████▊ | 95/198 [00:09<00:03, 27.98it/s] 52%|█████▏ | 102/198 [00:09<00:03, 28.30it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.21.post_attention_layernorm.weight", shape: (1536,), dtype: float16
52%|█████▏ | 102/198 [00:09<00:03, 28.30it/s] [2024-06-06 22:37:30] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.21.self_attn.c_attn.bias", shape: (2048,), dtype: float16
52%|█████▏ | 102/198 [00:09<00:03, 28.30it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.21.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
52%|█████▏ | 102/198 [00:09<00:03, 28.30it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.21.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
52%|█████▏ | 102/198 [00:09<00:03, 28.30it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.21.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
52%|█████▏ | 102/198 [00:09<00:03, 28.30it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.21.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
52%|█████▏ | 102/198 [00:09<00:03, 28.30it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.22.input_layernorm.weight", shape: (1536,), dtype: float16
52%|█████▏ | 102/198 [00:09<00:03, 28.30it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.22.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
52%|█████▏ | 102/198 [00:09<00:03, 28.30it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.22.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
52%|█████▏ | 102/198 [00:09<00:03, 28.30it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.22.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
52%|█████▏ | 102/198 [00:09<00:03, 28.30it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.22.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
52%|█████▏ | 102/198 [00:09<00:03, 28.30it/s] 55%|█████▌ | 109/198 [00:09<00:03, 28.55it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.22.post_attention_layernorm.weight", shape: (1536,), dtype: float16
55%|█████▌ | 109/198 [00:09<00:03, 28.55it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.22.self_attn.c_attn.bias", shape: (2048,), dtype: float16
55%|█████▌ | 109/198 [00:09<00:03, 28.55it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.22.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
55%|█████▌ | 109/198 [00:09<00:03, 28.55it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.22.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
55%|█████▌ | 109/198 [00:09<00:03, 28.55it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.22.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
55%|█████▌ | 109/198 [00:09<00:03, 28.55it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.22.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
55%|█████▌ | 109/198 [00:09<00:03, 28.55it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.23.input_layernorm.weight", shape: (1536,), dtype: float16
55%|█████▌ | 109/198 [00:09<00:03, 28.55it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.23.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
55%|█████▌ | 109/198 [00:09<00:03, 28.55it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.23.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
55%|█████▌ | 109/198 [00:09<00:03, 28.55it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.23.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
55%|█████▌ | 109/198 [00:09<00:03, 28.55it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.23.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
55%|█████▌ | 109/198 [00:09<00:03, 28.55it/s] 59%|█████▊ | 116/198 [00:09<00:02, 28.81it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.23.post_attention_layernorm.weight", shape: (1536,), dtype: float16
59%|█████▊ | 116/198 [00:09<00:02, 28.81it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.23.self_attn.c_attn.bias", shape: (2048,), dtype: float16
59%|█████▊ | 116/198 [00:09<00:02, 28.81it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.23.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
59%|█████▊ | 116/198 [00:09<00:02, 28.81it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.23.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
59%|█████▊ | 116/198 [00:09<00:02, 28.81it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.23.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
59%|█████▊ | 116/198 [00:09<00:02, 28.81it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.23.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
59%|█████▊ | 116/198 [00:09<00:02, 28.81it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.24.input_layernorm.weight", shape: (1536,), dtype: float16
59%|█████▊ | 116/198 [00:09<00:02, 28.81it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.24.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
59%|█████▊ | 116/198 [00:10<00:02, 28.81it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.24.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
59%|█████▊ | 116/198 [00:10<00:02, 28.81it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.24.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
59%|█████▊ | 116/198 [00:10<00:02, 28.81it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.24.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
59%|█████▊ | 116/198 [00:10<00:02, 28.81it/s] 62%|██████▏ | 123/198 [00:10<00:02, 28.96it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.24.post_attention_layernorm.weight", shape: (1536,), dtype: float16
62%|██████▏ | 123/198 [00:10<00:02, 28.96it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.24.self_attn.c_attn.bias", shape: (2048,), dtype: float16
62%|██████▏ | 123/198 [00:10<00:02, 28.96it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.24.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
62%|██████▏ | 123/198 [00:10<00:02, 28.96it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.24.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
62%|██████▏ | 123/198 [00:10<00:02, 28.96it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.24.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
62%|██████▏ | 123/198 [00:10<00:02, 28.96it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.24.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
62%|██████▏ | 123/198 [00:10<00:02, 28.96it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.25.input_layernorm.weight", shape: (1536,), dtype: float16
62%|██████▏ | 123/198 [00:10<00:02, 28.96it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.25.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
62%|██████▏ | 123/198 [00:10<00:02, 28.96it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.25.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
62%|██████▏ | 123/198 [00:10<00:02, 28.96it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.25.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
62%|██████▏ | 123/198 [00:10<00:02, 28.96it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.25.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
62%|██████▏ | 123/198 [00:10<00:02, 28.96it/s] 66%|██████▌ | 130/198 [00:10<00:02, 29.03it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.25.post_attention_layernorm.weight", shape: (1536,), dtype: float16
66%|██████▌ | 130/198 [00:10<00:02, 29.03it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.25.self_attn.c_attn.bias", shape: (2048,), dtype: float16
66%|██████▌ | 130/198 [00:10<00:02, 29.03it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.25.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
66%|██████▌ | 130/198 [00:10<00:02, 29.03it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.25.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
66%|██████▌ | 130/198 [00:10<00:02, 29.03it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.25.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
66%|██████▌ | 130/198 [00:10<00:02, 29.03it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.25.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
66%|██████▌ | 130/198 [00:10<00:02, 29.03it/s] [2024-06-06 22:37:31] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.26.input_layernorm.weight", shape: (1536,), dtype: float16
66%|██████▌ | 130/198 [00:10<00:02, 29.03it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.26.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
66%|██████▌ | 130/198 [00:10<00:02, 29.03it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.26.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
66%|██████▌ | 130/198 [00:10<00:02, 29.03it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.26.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
66%|██████▌ | 130/198 [00:10<00:02, 29.03it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.26.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
66%|██████▌ | 130/198 [00:10<00:02, 29.03it/s] 69%|██████▉ | 137/198 [00:10<00:02, 28.99it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.26.post_attention_layernorm.weight", shape: (1536,), dtype: float16
69%|██████▉ | 137/198 [00:10<00:02, 28.99it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.26.self_attn.c_attn.bias", shape: (2048,), dtype: float16
69%|██████▉ | 137/198 [00:10<00:02, 28.99it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.26.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
69%|██████▉ | 137/198 [00:10<00:02, 28.99it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.26.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
69%|██████▉ | 137/198 [00:10<00:02, 28.99it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.26.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
69%|██████▉ | 137/198 [00:10<00:02, 28.99it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.26.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
69%|██████▉ | 137/198 [00:10<00:02, 28.99it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.27.input_layernorm.weight", shape: (1536,), dtype: float16
69%|██████▉ | 137/198 [00:10<00:02, 28.99it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.27.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
69%|██████▉ | 137/198 [00:10<00:02, 28.99it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.27.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
69%|██████▉ | 137/198 [00:10<00:02, 28.99it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.27.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
69%|██████▉ | 137/198 [00:10<00:02, 28.99it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.27.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
69%|██████▉ | 137/198 [00:10<00:02, 28.99it/s] 73%|███████▎ | 144/198 [00:10<00:01, 28.95it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.27.post_attention_layernorm.weight", shape: (1536,), dtype: float16
73%|███████▎ | 144/198 [00:10<00:01, 28.95it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.27.self_attn.c_attn.bias", shape: (2048,), dtype: float16
73%|███████▎ | 144/198 [00:10<00:01, 28.95it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.27.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
73%|███████▎ | 144/198 [00:10<00:01, 28.95it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.27.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
73%|███████▎ | 144/198 [00:10<00:01, 28.95it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.27.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
73%|███████▎ | 144/198 [00:10<00:01, 28.95it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.27.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
73%|███████▎ | 144/198 [00:10<00:01, 28.95it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.3.input_layernorm.weight", shape: (1536,), dtype: float16
73%|███████▎ | 144/198 [00:10<00:01, 28.95it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.3.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
73%|███████▎ | 144/198 [00:10<00:01, 28.95it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.3.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
73%|███████▎ | 144/198 [00:10<00:01, 28.95it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.3.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
73%|███████▎ | 144/198 [00:11<00:01, 28.95it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.3.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
73%|███████▎ | 144/198 [00:11<00:01, 28.95it/s] 76%|███████▋ | 151/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.3.post_attention_layernorm.weight", shape: (1536,), dtype: float16
76%|███████▋ | 151/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.3.self_attn.c_attn.bias", shape: (2048,), dtype: float16
76%|███████▋ | 151/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.3.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
76%|███████▋ | 151/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.3.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
76%|███████▋ | 151/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.3.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
76%|███████▋ | 151/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.3.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
76%|███████▋ | 151/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.4.input_layernorm.weight", shape: (1536,), dtype: float16
76%|███████▋ | 151/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.4.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
76%|███████▋ | 151/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.4.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
76%|███████▋ | 151/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.4.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
76%|███████▋ | 151/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.4.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
76%|███████▋ | 151/198 [00:11<00:01, 28.93it/s] 80%|███████▉ | 158/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.4.post_attention_layernorm.weight", shape: (1536,), dtype: float16
80%|███████▉ | 158/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.4.self_attn.c_attn.bias", shape: (2048,), dtype: float16
80%|███████▉ | 158/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.4.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
80%|███████▉ | 158/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.4.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
80%|███████▉ | 158/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.4.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
80%|███████▉ | 158/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.4.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
80%|███████▉ | 158/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.5.input_layernorm.weight", shape: (1536,), dtype: float16
80%|███████▉ | 158/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.5.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
80%|███████▉ | 158/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:32] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.5.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
80%|███████▉ | 158/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.5.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
80%|███████▉ | 158/198 [00:11<00:01, 28.93it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.5.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
80%|███████▉ | 158/198 [00:11<00:01, 28.93it/s] 83%|████████▎ | 165/198 [00:11<00:01, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.5.post_attention_layernorm.weight", shape: (1536,), dtype: float16
83%|████████▎ | 165/198 [00:11<00:01, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.5.self_attn.c_attn.bias", shape: (2048,), dtype: float16
83%|████████▎ | 165/198 [00:11<00:01, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.5.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
83%|████████▎ | 165/198 [00:11<00:01, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.5.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
83%|████████▎ | 165/198 [00:11<00:01, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.5.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
83%|████████▎ | 165/198 [00:11<00:01, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.5.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
83%|████████▎ | 165/198 [00:11<00:01, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.6.input_layernorm.weight", shape: (1536,), dtype: float16
83%|████████▎ | 165/198 [00:11<00:01, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.6.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
83%|████████▎ | 165/198 [00:11<00:01, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.6.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
83%|████████▎ | 165/198 [00:11<00:01, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.6.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
83%|████████▎ | 165/198 [00:11<00:01, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.6.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
83%|████████▎ | 165/198 [00:11<00:01, 28.96it/s] 87%|████████▋ | 172/198 [00:11<00:00, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.6.post_attention_layernorm.weight", shape: (1536,), dtype: float16
87%|████████▋ | 172/198 [00:11<00:00, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.6.self_attn.c_attn.bias", shape: (2048,), dtype: float16
87%|████████▋ | 172/198 [00:11<00:00, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.6.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
87%|████████▋ | 172/198 [00:11<00:00, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.6.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
87%|████████▋ | 172/198 [00:11<00:00, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.6.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
87%|████████▋ | 172/198 [00:11<00:00, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.6.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
87%|████████▋ | 172/198 [00:11<00:00, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.7.input_layernorm.weight", shape: (1536,), dtype: float16
87%|████████▋ | 172/198 [00:11<00:00, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.7.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
87%|████████▋ | 172/198 [00:11<00:00, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.7.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
87%|████████▋ | 172/198 [00:11<00:00, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.7.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
87%|████████▋ | 172/198 [00:12<00:00, 28.96it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.7.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
87%|████████▋ | 172/198 [00:12<00:00, 28.96it/s] 90%|█████████ | 179/198 [00:12<00:00, 29.04it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.7.post_attention_layernorm.weight", shape: (1536,), dtype: float16
90%|█████████ | 179/198 [00:12<00:00, 29.04it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.7.self_attn.c_attn.bias", shape: (2048,), dtype: float16
90%|█████████ | 179/198 [00:12<00:00, 29.04it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.7.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
90%|█████████ | 179/198 [00:12<00:00, 29.04it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.7.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
90%|█████████ | 179/198 [00:12<00:00, 29.04it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.7.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
90%|█████████ | 179/198 [00:12<00:00, 29.04it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.7.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
90%|█████████ | 179/198 [00:12<00:00, 29.04it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.8.input_layernorm.weight", shape: (1536,), dtype: float16
90%|█████████ | 179/198 [00:12<00:00, 29.04it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.8.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
90%|█████████ | 179/198 [00:12<00:00, 29.04it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.8.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
90%|█████████ | 179/198 [00:12<00:00, 29.04it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.8.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
90%|█████████ | 179/198 [00:12<00:00, 29.04it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.8.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
90%|█████████ | 179/198 [00:12<00:00, 29.04it/s] 94%|█████████▍| 186/198 [00:12<00:00, 29.08it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.8.post_attention_layernorm.weight", shape: (1536,), dtype: float16
94%|█████████▍| 186/198 [00:12<00:00, 29.08it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.8.self_attn.c_attn.bias", shape: (2048,), dtype: float16
94%|█████████▍| 186/198 [00:12<00:00, 29.08it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.8.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
94%|█████████▍| 186/198 [00:12<00:00, 29.08it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.8.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
94%|█████████▍| 186/198 [00:12<00:00, 29.08it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.8.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
94%|█████████▍| 186/198 [00:12<00:00, 29.08it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.8.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
94%|█████████▍| 186/198 [00:12<00:00, 29.08it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.9.input_layernorm.weight", shape: (1536,), dtype: float16
94%|█████████▍| 186/198 [00:12<00:00, 29.08it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.9.mlp.down_proj.q_weight", shape: (1536, 1120), dtype: uint32
94%|█████████▍| 186/198 [00:12<00:00, 29.08it/s] [2024-06-06 22:37:33] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.9.mlp.down_proj.q_scale", shape: (1536, 280), dtype: float16
94%|█████████▍| 186/198 [00:12<00:00, 29.08it/s] [2024-06-06 22:37:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.9.mlp.gate_up_proj.q_weight", shape: (17920, 192), dtype: uint32
94%|█████████▍| 186/198 [00:12<00:00, 29.08it/s] [2024-06-06 22:37:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.9.mlp.gate_up_proj.q_scale", shape: (17920, 48), dtype: float16
94%|█████████▍| 186/198 [00:12<00:00, 29.08it/s] 97%|█████████▋| 193/198 [00:12<00:00, 28.83it/s] [2024-06-06 22:37:34] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.9.post_attention_layernorm.weight", shape: (1536,), dtype: float16
97%|█████████▋| 193/198 [00:12<00:00, 28.83it/s] [2024-06-06 22:37:34] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.layers.9.self_attn.c_attn.bias", shape: (2048,), dtype: float16
97%|█████████▋| 193/198 [00:12<00:00, 28.83it/s] [2024-06-06 22:37:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.9.self_attn.c_attn.q_weight", shape: (2048, 192), dtype: uint32
97%|█████████▋| 193/198 [00:12<00:00, 28.83it/s] [2024-06-06 22:37:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.9.self_attn.c_attn.q_scale", shape: (2048, 48), dtype: float16
97%|█████████▋| 193/198 [00:12<00:00, 28.83it/s] [2024-06-06 22:37:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.9.self_attn.o_proj.q_weight", shape: (1536, 192), dtype: uint32
97%|█████████▋| 193/198 [00:12<00:00, 28.83it/s] [2024-06-06 22:37:34] INFO huggingface_loader.py:167: [Quantized] Parameter: "model.layers.9.self_attn.o_proj.q_scale", shape: (1536, 48), dtype: float16
97%|█████████▋| 193/198 [00:12<00:00, 28.83it/s] [2024-06-06 22:37:34] INFO huggingface_loader.py:175: [Not quantized] Parameter: "model.norm.weight", shape: (1536,), dtype: float16
97%|█████████▋| 193/198 [00:12<00:00, 28.83it/s] 100%|██████████| 198/198 [00:12<00:00, 15.69it/s]
[2024-06-06 22:37:34] INFO huggingface_loader.py:197: Unloading HF weight file: /ssd2/models/Qwen2-1.5B-Instruct/model.safetensors
[2024-06-06 22:37:34] INFO stats.py:77: Time usage: HF loading: 1.909 sec; Pre-quantization mapping: 5.642 sec; Quantization: 2.898 sec
[2024-06-06 22:37:34] INFO stats.py:91: RAM usage: Peak RAM: 5.751 GB. Total bytes loaded from disk: 5.751 GB
[2024-06-06 22:37:34] INFO convert_weight.py:155: Parameter size after quantization: 0.809 GB
[2024-06-06 22:37:34] INFO convert_weight.py:160: Total parameters: 1,543,714,304
[2024-06-06 22:37:34] INFO convert_weight.py:161: Bits per parameter: 4.501
[2024-06-06 22:37:34] INFO convert_weight.py:166: Saved to directory: /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC
All finished, 30 total shards committed, record saved to /models/mlc-delivery/hf/mlc-ai/Qwen2-1.5B-Instruct-q4f16_1-MLC/ndarray-cache.json