microsoft

/

Phi-3.5-MoE-instruct

like 516

Follow

Microsoft 4,844

vllm/benchmarks/benchmark_latency.py --model microsoft/Phi-3.5-MoE-instruct --max-model-len 4096 --batch-size 1 -tp 4 --input-len 1024 --output-len 1 --num-iters-warmup 5 --num-iters 5 --trust-remote-code --dtype bfloat16 --distributed-executor-backend "mp"

With the above command facing runtime problem ModuleNotFoundError: No module named 'transformers_modules.microsoft.Phi-3'
running environment shown below
torch 2.4.0a0+git837ece3
transformers 4.45.2
vllm 0.6.3.