 We find that almost all the contextualized word vectors of BERT and RoBERTa have a commonpattern. For BERT, the 557^{th neuron-level method to analyze where these "tails" come from.
<coherence>  We find that almost all the contextualized word vectors of BERT and RoBERTa have a commonpattern. For BERT, the 557^{th neuron-level method to analyze where these "tails" come from.
 We find that almost all the contextualized word vectors derived from pretrained masked language model-based encoders share a common, perhaps undesirable pattern across layers. Namely, we find cases of persistent outlier neurons within BERT and RoBERTa's hidden state vectors that consistently bear the smallest or largest values in said vectors. In an attempt to investigate the source of this information, we introduce a neuron-level method to analyze where these "tails" come from.