neuralmagic
/

Meta-Llama-3-8B-Instruct-FP8

Text Generation

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

Lin-K76 commited on Jul 15, 2024

Commit

ac1d815

·

verified ·

1 Parent(s): 3a04ae8

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -90,10 +90,10 @@ quantize_config = BaseQuantizeConfig(
     ignore_patterns=["re:.*lm_head"],
 )
 model = AutoFP8ForCausalLM.from_pretrained(
     pretrained_model_dir, quantize_config=quantize_config
 )
 model.quantize(examples)
 model.save_quantized(quantized_model_dir)
 ```

     ignore_patterns=["re:.*lm_head"],
 )
 model = AutoFP8ForCausalLM.from_pretrained(
     pretrained_model_dir, quantize_config=quantize_config
 )
 model.quantize(examples)
 model.save_quantized(quantized_model_dir)
 ```