Is it fine to do granite-20b model's inference with bfloat16 dtype?

by lkm1 - opened Jun 2

lkm1

Jun 2

I have tried this model's inference with bfloat16 type and have not faced any problems. Is it fine to use bfloat16 dtype for this model's inference, without much effect on score?

mayank-mishra

IBM Granite org Jun 4

yeah, it should be ok @lkm1

mayank-mishra

IBM Granite org Jun 4

we have run scores in bf16

lkm1

Jun 4

ok, thanks @mayank-mishra !

lkm1 changed discussion status to closed Jun 4

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment