Update README.md

README.md

@@ -13,6 +13,7 @@ license_link: https://llama.meta.com/llama3/license/
   - **Input:** Text
   - **Output:** Text
 - **Model Optimizations:**
+  - **Activation quantization:** INT8
   - **Weight quantization:** INT8
 - **Intended Use Cases:** Intended for commercial and research use in English. Similarly to [Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct), this models is intended for assistant-like chat.
 - **Out-of-scope:** Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English.
@@ -27,10 +28,14 @@ It achieves an average score of 79.18 on the [OpenLLM](https://huggingface.co/sp
 ### Model Optimizations
 
 This model was obtained by quantizing the weights of [Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) to INT8 data type.
-This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
+This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x).
+Weight quantization also reduces disk size requirements by approximately 50%.
 
-Only the weights of the linear operators within transformers blocks are quantized. Symmetric per-channel quantization is applied, in which a linear scaling per output dimension maps the INT8 and floating point representations of the quantized weights.
-[AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) is used for quantization with 10% damping factor and 128 sequences taken from Neural Magic's [LLM compression calibration dataset](https://huggingface.co/datasets/neuralmagic/LLM_compression_calibration).
+Only weights and activations of the linear operators within transformers blocks are quantized.
+Weights are quantized with a symmetric static per-channel scheme, where a fixed linear scaling factor is applied between INT8 and floating point representations for each output channel dimension.
+Activations are quantized with a symmetric dynamic per-token scheme, computing a linear scaling factor at runtime for each token between INT8 and floating point representations.
+The [GPTQ](https://arxiv.org/abs/2210.17323) algorithm is applied for quantization, as implemented in the [llm-compressor](https://github.com/vllm-project/llm-compressor) library.
+GPTQ used a 10% damping factor and 256 sequences taken from Neural Magic's [LLM compression calibration dataset](https://huggingface.co/datasets/neuralmagic/LLM_compression_calibration).
 
 
 ## Deployment
@@ -43,7 +48,7 @@ This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/
 from vllm import LLM, SamplingParams
 from transformers import AutoTokenizer
 
-model_id = "neuralmagic/Meta-Llama-3-70B-Instruct-quantized.w8a16"
+model_id = "neuralmagic/Meta-Llama-3-70B-Instruct-quantized.w8a8"
 number_gpus = 2
 
 sampling_params = SamplingParams(temperature=0.6, top_p=0.9, max_tokens=256)
@@ -69,13 +74,12 @@ vLLM aslo supports OpenAI-compatible serving. See the [documentation](https://do
 
 ### Use with transformers
 
-This model is supported by Transformers leveraging the integration with the [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) data format.
-The following example contemplates how the model can be used using the `generate()` function.
+The following example contemplates how the model can be deployed in Transformers using the `generate()` function.
 
 ```python
 from transformers import AutoTokenizer, AutoModelForCausalLM
 
-model_id = "neuralmagic/Meta-Llama-3-70B-Instruct-quantized.w8a16"
+model_id = "neuralmagic/Meta-Llama-3-70B-Instruct-quantized.w8a8"
 
 tokenizer = AutoTokenizer.from_pretrained(model_id)
 model = AutoModelForCausalLM.from_pretrained(
@@ -114,17 +118,17 @@ print(tokenizer.decode(response, skip_special_tokens=True))
 
 ## Creation
 
-This model was created by applying the [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) library as presented in the code snipet below.
-Although AutoGPTQ was used for this particular model, Neural Magic is transitioning to using [llm-compressor](https://github.com/vllm-project/llm-compressor) which supports several quantization schemes and models not supported by AutoGPTQ.
+This model was created by using the [llm-compressor](https://github.com/vllm-project/llm-compressor) library as presented in the code snipet below.
 
 ```python
 from transformers import AutoTokenizer
-from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
 from datasets import load_dataset
+from llmcompressor.transformers import SparseAutoModelForCausalLM, oneshot
+from llmcompressor.modifiers.quantization import GPTQModifier
 
-model_id = "meta-llama/Meta-Llama-3-70B-Instruct"
+model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
 
-num_samples = 128
+num_samples = 256
 max_seq_len = 8192
 
 tokenizer = AutoTokenizer.from_pretrained(model_id)
@@ -136,28 +140,31 @@ ds = load_dataset("neuralmagic/LLM_compression_calibration", split="train")
 ds = ds.shuffle().select(range(num_samples))
 ds = ds.map(preprocess_fn)
 
-examples = [tokenizer(example["text"], padding=False, max_length=max_seq_len, truncation=True) for example in ds]
-    
-quantize_config = BaseQuantizeConfig(
-  bits=8,
-  group_size=-1,
-  desc_act=False,
-  model_file_base_name="model",
-  damp_percent=0.1,
+recipe = GPTQModifier(
+  targets="Linear",
+  scheme="W8A8",
+  ignore=["lm_head"],
+  dampening_frac=0.1,
 )
 
-model = AutoGPTQForCausalLM.from_pretrained(
+model = SparseAutoModelForCausalLM.from_pretrained(
   model_id,
-  quantize_config,
   device_map="auto",
+  trust_remote_code=True,
+)
+
+oneshot(
+  model=model,
+  dataset=ds,
+  recipe=recipe,
+  max_seq_length=max_seq_len,
+  num_calibration_samples=num_samples,
 )
 
-model.quantize(examples)
 model.save_pretrained("Meta-Llama-3-70B-Instruct-quantized.w8a8")
 ```
 
 
-
 ## Evaluation
 
 The model was evaluated on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) leaderboard tasks (version 1) with the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/383bbd54bc621086e05aa1b030d8d4d5635b25e6) (commit 383bbd54bc621086e05aa1b030d8d4d5635b25e6) and the [vLLM](https://docs.vllm.ai/en/stable/) engine, using the following command (using 2 GPUs):
@@ -178,7 +185,7 @@ lm_eval \
    </td>
    <td><strong>Meta-Llama-3-70B-Instruct </strong>
    </td>
-   <td><strong>Meta-Llama-3-70B-Instruct-quantized.w8a16 (this model)</strong>
+   <td><strong>Meta-Llama-3-70B-Instruct-quantized.w8a8 (this model)</strong>
    </td>
    <td><strong>Recovery</strong>
    </td>