Hugging Face's logo

DeepPavlov
/
rubert-base-cased

Feature Extraction
Transformers
PyTorch
JAX
Russian
bert
Inference Endpoints
Model card Files Community
3
rubert-base-cased / README.md
Yura Kuratov
upload model with MLM and NSP heads
c89e312
preview code
|
raw
history blame contribute delete
584 Bytes
metadata
language:
  - ru

rubert-base-cased

RuBERT (Russian, cased, 12‑layer, 768‑hidden, 12‑heads, 180M parameters) was trained on the Russian part of Wikipedia and news data. We used this training data to build a vocabulary of Russian subtokens and took a multilingual version of BERT‑base as an initialization for RuBERT[1].

08.11.2021: upload model with MLM and NSP heads

[1]: Kuratov, Y., Arkhipov, M. (2019). Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language. arXiv preprint arXiv:1905.07213.