microsoft

/

phi-2

like 3.29k

Follow

Microsoft 10.1k

phi-2 is great to run on cpu due to its small size. there aren't that many other options.

can you put the CPU support back? it worked well for me.

with the removal in commit cb2f4533604d8b67de604e7df03bfe6f3ca22869

i get an error:

(.venv) ubuntu@ip-172-31-7-92 ~/t/phi-2 (main)> cat test.py
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

torch.set_default_device("cpu")

model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype=torch.float32, device_map="cpu", trust_remote_cod
e=True)

tokenizer = AutoTokenizer.from_pretrained("microsoft/phi-2", trust_remote_code=True)

inputs = tokenizer('''Instruct: what is LLM?
                   Output:''', return_tensors="pt", return_attention_mask=False)

outputs = model.generate(**inputs, max_length=200)
text = tokenizer.batch_decode(outputs)[0]
print(text)

(.venv) ubuntu@ip-172-31-7-92 ~/t/phi-2 (main)> python test.py
Traceback (most recent call last):
  File "/home/ubuntu/tmp/phi-2/test.py", line 6, in <module>
    model = AutoModelForCausalLM.from_pretrained("microsoft/phi-2", torch_dtype=torch.float32, device_map="cpu", trust_remote
_code=True)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
^^^^^^^^^^^
  File "/home/ubuntu/tmp/phi-2/.venv/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 553, in fro
m_pretrained
    model_class = get_class_from_dynamic_module(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/tmp/phi-2/.venv/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 488, in get_cla
ss_from_dynamic_module
    final_module = get_cached_module_file(
                   ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/tmp/phi-2/.venv/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 315, in get_cac
hed_module_file
    modules_needed = check_imports(resolved_module_file)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/tmp/phi-2/.venv/lib/python3.11/site-packages/transformers/dynamic_module_utils.py", line 180, in check_i
mports
    raise ImportError(
ImportError: This modeling file requires the following packages that were not found in your environment: flash_attn. Run `pip
 install flash_attn`