Has Your Pretrained Model Improved? A Multi-head Posterior Based Approach

The emergence of pretrained models has significantly impacted from Natural Language Processing (NLP) and Computer Vision to <PRE_TAG>relational datasets</POST_TAG>. Traditionally, these models are assessed through fine-tuned downstream tasks. However, this raises the question of how to evaluate these models more efficiently and more effectively. In this study, we explore a novel approach where we leverage the meta features associated with each entity as a source of worldly knowledge and employ entity representations from the models. We propose using the consistency between these representations and the meta features as a metric for evaluating pretrained models. Our method's effectiveness is demonstrated across various domains, including models with <PRE_TAG>relational datasets</POST_TAG>, large language models and images models.