rttl-ai

/

bert-base-uncased-yelp-reviews

like 2

Follow

rttl labs 2

Hi, for task adaptive pre-training (adapting it for Yelp reviews), you can use:
https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py
You can change the value for mlm_probability to increase the complexity of the prediction task.
for switching to WWM, you just need to use DataCollatorForWholeWordMask or switch to this script:
https://github.com/huggingface/transformers/blob/main/examples/research_projects/mlm_wwm/run_mlm_wwm.py

Hi, for task adaptive pre-training (adapting it for Yelp reviews), you can use:
https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py
You can change the value for mlm_probability to increase the complexity of the prediction task.
for switching to WWM, you just need to use DataCollatorForWholeWordMask or switch to this script:
https://github.com/huggingface/transformers/blob/main/examples/research_projects/mlm_wwm/run_mlm_wwm.py

Thanks for sharing. I'll take a look :)