Issue with Rope Scaling
#31
by
ritwickchaudhryamazon
- opened
I'm trying to train the Phi-3-small-128k-instruct model and the configuration loading leads to an error in the rope_validation function here because the config has more than 3 hyper-parameters which fails the check.
Is there a fix to this issue?
Way to Reproduce:
from transformers import Phi3ForCausalLM
Phi3ForCausalLM.from_pretrained(<path/to/Phi3_small_128k_instruct>")