Update README.md (#3)
Browse files- Update README.md (bdcb7ead83248359087edba314a5125c4689ce45)
README.md
CHANGED
@@ -30,10 +30,18 @@ python3 quantize_quark.py \
|
|
30 |
--quant_scheme w_fp8_a_fp8 \
|
31 |
--kv_cache_dtype fp8 \
|
32 |
--num_calib_data 128 \
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
33 |
<tr>
|
34 |
<td><strong>Benchmark</strong>
|
35 |
</td>
|
36 |
-
<td><strong>Llama-3.2-90B-Vision-Instruct
|
37 |
</td>
|
38 |
<td><strong>Llama-3.2-90B-Vision-Instruct-FP8-KV(this model)</strong>
|
39 |
</td>
|
@@ -46,3 +54,18 @@ python3 quantize_quark.py \
|
|
46 |
<td>3.8570
|
47 |
</td>
|
48 |
</tr>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
30 |
--quant_scheme w_fp8_a_fp8 \
|
31 |
--kv_cache_dtype fp8 \
|
32 |
--num_calib_data 128 \
|
33 |
+
```
|
34 |
+
## Deployment
|
35 |
+
Quark has its own export format and allows FP8 quantized models to be efficiently deployed using the vLLM backend(vLLM-compatible).
|
36 |
+
## Evaluation
|
37 |
+
Quark currently uses perplexity(PPL) as the evaluation metric for accuracy loss before and after quantization.The specific PPL algorithm can be referenced in the quantize_quark.py.
|
38 |
+
The quantization evaluation results are conducted in pseudo-quantization mode, which may slightly differ from the actual quantized inference accuracy. These results are provided for reference only.
|
39 |
+
#### Evaluation scores
|
40 |
+
<table>
|
41 |
<tr>
|
42 |
<td><strong>Benchmark</strong>
|
43 |
</td>
|
44 |
+
<td><strong>Llama-3.2-90B-Vision-Instruct </strong>
|
45 |
</td>
|
46 |
<td><strong>Llama-3.2-90B-Vision-Instruct-FP8-KV(this model)</strong>
|
47 |
</td>
|
|
|
54 |
<td>3.8570
|
55 |
</td>
|
56 |
</tr>
|
57 |
+
|
58 |
+
</table>
|
59 |
+
|
60 |
+
#### License
|
61 |
+
Modifications copyright(c) 2024 Advanced Micro Devices,Inc. All rights reserved.
|
62 |
+
Licensed under the Apache License, Version 2.0 (the "License");
|
63 |
+
you may not use this file except in compliance with the License.
|
64 |
+
You may obtain a copy of the License at
|
65 |
+
http://www.apache.org/licenses/LICENSE-2.0
|
66 |
+
Unless required by applicable law or agreed to in writing, software
|
67 |
+
distributed under the License is distributed on an "AS IS" BASIS,
|
68 |
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
69 |
+
See the License for the specific language governing permissions and
|
70 |
+
limitations under the License.
|
71 |
+
|