Edit model card

IndoConvBERT Base Model

IndoConvBERT is a ConvBERT model pretrained on Indo4B.

Pretraining details

We follow a different training procedure: instead of using a two-phase approach, that pre-trains the model for 90% with 128 sequence length and 10% with 512 sequence length, we pre-train the model with 512 sequence length for 1M steps on a v3-8 TPU.

The current version of the model is trained on Indo4B and small Twitter dump.

Acknowledgement

Big thanks to TFRC (TensorFlow Research Cloud) for providing free TPU.

Downloads last month
16
Inference Examples
Inference API (serverless) is not available, repository is disabled.