 We find that long-tailed zero and few-shot learning markedly benefit from increasing 'dataset-internal' self-supervised pretraining signals, to help reduce the reliance on large external sources .
<meaning-changed>  We find that long-tailed zero and few-shot learning markedly benefit from increasing 'dataset-internal' self-supervised pretraining signals, to help reduce the reliance on large external sources .
 We find our method outperforms RoBERTa, while pretraining and fine-tuning in a 1/5th of RoBERTa's fine-tuning time .