luow-amd commited on
Commit
bd24176
·
verified ·
1 Parent(s): 2f1694e

Update README.md (#3)

Browse files

- Update README.md (bdcb7ead83248359087edba314a5125c4689ce45)

Files changed (1) hide show
  1. README.md +24 -1
README.md CHANGED
@@ -30,10 +30,18 @@ python3 quantize_quark.py \
30
  --quant_scheme w_fp8_a_fp8 \
31
  --kv_cache_dtype fp8 \
32
  --num_calib_data 128 \
 
 
 
 
 
 
 
 
33
  <tr>
34
  <td><strong>Benchmark</strong>
35
  </td>
36
- <td><strong>Llama-3.2-90B-Vision-Instruct </strong>
37
  </td>
38
  <td><strong>Llama-3.2-90B-Vision-Instruct-FP8-KV(this model)</strong>
39
  </td>
@@ -46,3 +54,18 @@ python3 quantize_quark.py \
46
  <td>3.8570
47
  </td>
48
  </tr>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
30
  --quant_scheme w_fp8_a_fp8 \
31
  --kv_cache_dtype fp8 \
32
  --num_calib_data 128 \
33
+ ```
34
+ ## Deployment
35
+ Quark has its own export format and allows FP8 quantized models to be efficiently deployed using the vLLM backend(vLLM-compatible).
36
+ ## Evaluation
37
+ Quark currently uses perplexity(PPL) as the evaluation metric for accuracy loss before and after quantization.The specific PPL algorithm can be referenced in the quantize_quark.py.
38
+ The quantization evaluation results are conducted in pseudo-quantization mode, which may slightly differ from the actual quantized inference accuracy. These results are provided for reference only.
39
+ #### Evaluation scores
40
+ <table>
41
  <tr>
42
  <td><strong>Benchmark</strong>
43
  </td>
44
+ <td><strong>Llama-3.2-90B-Vision-Instruct </strong>
45
  </td>
46
  <td><strong>Llama-3.2-90B-Vision-Instruct-FP8-KV(this model)</strong>
47
  </td>
 
54
  <td>3.8570
55
  </td>
56
  </tr>
57
+
58
+ </table>
59
+
60
+ #### License
61
+ Modifications copyright(c) 2024 Advanced Micro Devices,Inc. All rights reserved.
62
+ Licensed under the Apache License, Version 2.0 (the "License");
63
+ you may not use this file except in compliance with the License.
64
+ You may obtain a copy of the License at
65
+ http://www.apache.org/licenses/LICENSE-2.0
66
+ Unless required by applicable law or agreed to in writing, software
67
+ distributed under the License is distributed on an "AS IS" BASIS,
68
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
69
+ See the License for the specific language governing permissions and
70
+ limitations under the License.
71
+