 Extensive experiments on our generated probe sets  show that PTLMs perform no better than random guessing (even with fine-tuning) , are heavily impacted by statistical biases, and are not robust to perturbation attacks.
<clarity>  Extensive experiments on our generated probe sets  show that PTLMs perform no better than random guessing (even with fine-tuning) , are heavily impacted by statistical biases, and are not robust to perturbation attacks.
 Extensive experiments on our generated probe sets show that PTLMs perform no better than random guessing on the zero-shot setting , are heavily impacted by statistical biases, and are not robust to perturbation attacks.