---
license: mit
datasets:
- omarmomen/babylm_10M
language:
- en
metrics:
- perplexity
library_name: transformers
pipeline_tag: fill-mask
---
# Model Card for omarmomen/tf_babylm_1

This model is part of the experiments in my master's thesis titled "Linguistic Structure Induction from Language Models" (https://arxiv.org/abs/2403.09714).

"omarmomen/tf_babylm_1" is the vanilla transformer encoder referred to in Chapter 5 (p. 59).

The model is trained on the BabyLM 10M dataset, with a RobertaTokenizer pretrained on the BabyLM 10M dataset with 16K tokens (https://huggingface.co/omarmomen/babylm_bpe_tokenizer_16k).

https://arxiv.org/abs/2403.09714