michaelfeil
/

ct2fast-flan-alpaca-xl

Inference Endpoints

Model card Files Files and versions Community

michaelfeil commited on May 3, 2023

Commit

891cbdf

·

1 Parent(s): f8dfe8e

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -25,7 +25,7 @@ model = TranslatorCT2fromHfHub(
         # load in int8 on CUDA
         model_name_or_path=model_name,
         device="cuda",
-        compute_type="cpu"
 )
 outputs = model.generate(
     text=["How do you call a fast Flan-ingo?", "Translate to german: How are you doing?"],

         # load in int8 on CUDA
         model_name_or_path=model_name,
         device="cuda",
+        compute_type="int8_float16"
 )
 outputs = model.generate(
     text=["How do you call a fast Flan-ingo?", "Translate to german: How are you doing?"],