README.md

---
language: id
license: mit
tags:
- indonesian-roberta-base-indonli
datasets:
- indonli
widget:
- text: Andi tersenyum karena mendapat hasil baik. </s></s> Andi sedih.
model-index:
- name: w11wo/indonesian-roberta-base-indonli
  results:
  - task:
      type: natural-language-inference
      name: Natural Language Inference
    dataset:
      name: indonli
      type: indonli
      config: indonli
      split: test_expert
    metrics:
    - type: accuracy
      value: 0.6072386058981233
      name: Accuracy
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYzY4NDkwNTdlYjI4MzY3ZTk3NmZjYjA1MjE2YWQ5MjJjMGM3NTc1NWVjODQzNTc1ZTYyZWVmYmY5NTI3NWY1ZSIsInZlcnNpb24iOjF9.Aeo_Id90j2JtyApv3LvJHkQtHz-9wO4SNvTdb8O_pp0KFQGfWXnkgX2t2hafIUxSKmZbETIte-FaPbZ9AGZSDA
    - type: precision
      value: 0.6304330508019023
      name: Precision Macro
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiY2I3Y2RkNjU0NzlkYmJiNWYyZjZhNzIzZGE5ODU4NzYxYmQ0NTYxYzZkM2JiNTQwZTdkMmYxOTRmMDlmOGFkMiIsInZlcnNpb24iOjF9.iEt7Mq6a3TubFQfdC3OAxAiZDXp0bPGhN9JPzSfKl89_dxnKzDp0IrVzkt1HNLHR_S22Q75Tevqh3_G8Pp05Dg
    - type: precision
      value: 0.6072386058981233
      name: Precision Micro
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYmI4ZmEwOTY0NTViNTM1ZjcwY2E2ODRmNGJiMTg2ZDJmZTgyNGUxM2UwNjZjYzVmYTcxZGY4OGY3OTI4MzcyMSIsInZlcnNpb24iOjF9.Jn1OPD1ZxkblCqKT1CfeUYOt5Xb6CL6C2ZENLmvfYNzh-p0oHcIBgapfbCHc89oMSR-FhjQk_ME8f8A3eyy6CA
    - type: precision
      value: 0.6320495884503851
      name: Precision Weighted
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiYWU1NmI5MTk3ODY4Y2M5ZjYyNzMwZjRlZTAzNjFmNmUxNzgxYjVhODNjZDAwOTQyNDBlZDJkNDYyNzRlYzBmOSIsInZlcnNpb24iOjF9.ItCi8SouqOtM3P7c0KN5ifRpGOr1090aqo4zX4aVSlVOTq0iQj9_c3z0B_UAzFcr0qW7ObnvuiD8D5d-9EzkBg
    - type: recall
      value: 0.6127303344145852
      name: Recall Macro
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNTZmMGEzMGZjYTI2MzA0OGQ1Nzk5ZDRmNDNhMTRhOGMyM2I4Zjc2NzMzNmM2NjQ0YzVjZDY1Mzk1ZWI5Zjg2NiIsInZlcnNpb24iOjF9.fWCCNatB50nCptCbXopRjwxbWic6BvWIG6frUo_iXJVFXsi3Q_ik91_70fLgZc9NfhIpewpNoe4ETn0Gmps4Cw
    - type: recall
      value: 0.6072386058981233
      name: Recall Micro
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNGMwOWVmOWU4YzM2ZjMyY2NjNjljMDdjZTZmMjU5MjRiNDU0NmVhYWEzZmQ5MzUzMmRhMjdmZjhkNDU4ZTM2ZCIsInZlcnNpb24iOjF9.Sy2c29OhxT-x4UBSr9G7rfwtyqzYOX4KNRe2blonfOdKrqSfSEORY01Y67WweDiKdRvbECzI-DemJUXVtx-QCg
    - type: recall
      value: 0.6072386058981233
      name: Recall Weighted
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiODgwODAwMDNkZWJlMzYyYmQwYzNmMGZiMzhjZjVjNGYyMTg3OWVjNjZmMzFhMDczNGEyMDAyODkwZTZhZWM4NSIsInZlcnNpb24iOjF9.8IxdbjDQHzcNW71RAMtKHzlviweLTQvYVQ4JlrqoZsV-8gyzxpyYOmDjUm3n6uQNfRLRpyvsT-E8ysLHPyMqDg
    - type: f1
      value: 0.6010566054103847
      name: F1 Macro
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZmMwMjg2YjZmYzMzYzlmNTk3OGIyMDc5NGUxNTFlNmNkNmU3MzU2ZWMyZjY0OGFhYmY5YTUyNDkxNjJiODIwNSIsInZlcnNpb24iOjF9.r1ylajrOC-Qu4QNdNnXzisjGlczTF_9tYpNEr8LYdTtdmJtRjNtNmElINneuaWX7XGN9TExdzmg7OWTwutjsAg
    - type: f1
      value: 0.6072386058981233
      name: F1 Micro
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiNDdhMGU1NDNjMzgxN2NjNWYzNTk5OWY5OTZhZWRjOGZkNjI4ZDA1YjI5ZWYxOWNmNDc4NmVhNjllMjUyMTFkMSIsInZlcnNpb24iOjF9.5G1km-a2_ssO_b3WTD8Ools29e6h8X8rjpClFN5Q_I4ADbPxKI2QbCfd5vl89CMHclignQ1_H6vqYbdTL9usDQ
    - type: f1
      value: 0.5995456855334425
      name: F1 Weighted
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiMjYyNTZjOTUzNmMxOTY3MzQzNjMxZGNhYTY3NTQ4Mjg3NWRlMjc2NmY1NjMxOWY0NTFiODlhZjA3ZTEzNGQ3MSIsInZlcnNpb24iOjF9.3iTI9IieFa3WJFr7ovDvO24IPScGB7WQk3Pw_Qxh32zKx5QyOwmZf_p2bgbEG6hBeCkR0KaMDvIiZXnbW6DBDQ
    - type: loss
      value: 1.157181739807129
      name: loss
      verified: true
      verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZmIxNTQyYWRkMjgxMTZkN2JhZTg5NDFiMDRlZGEzOGE5ZDIwYTE5NTU4YmU0NDUxOTE1MDQwMzFlMjQ5MDQ2YSIsInZlcnNpb24iOjF9.M-U6Dp-I-DEXZ3qSGLxYrCdQjgXi6DotHDgz1acjWnHIZWKPApy-n2194FZik1Tpv2AcJVe45tDRLxNSW3zVBg
---

## Indonesian RoBERTa Base IndoNLI

Indonesian RoBERTa Base IndoNLI is a natural language inference (NLI) model based on the [RoBERTa](https://arxiv.org/abs/1907.11692) model. The model was originally the pre-trained [Indonesian RoBERTa Base](https://hf.co/flax-community/indonesian-roberta-base) model, which is then fine-tuned on [`IndoNLI`](https://github.com/ir-nlp-csui/indonli)'s dataset consisting of Indonesian Wikipedia, news, and Web articles [1].

After training, the model achieved an evaluation/dev accuracy of 77.06%. On the benchmark `test_lay` subset, the model achieved an accuracy of 74.24% and on the benchmark `test_expert` subset, the model achieved an accuracy of 61.66%.

Hugging Face's `Trainer` class from the [Transformers](https://huggingface.co/transformers) library was used to train the model. PyTorch was used as the backend framework during training, but the model remains compatible with other frameworks nonetheless.

## Model

| Model                             | #params | Arch.        | Training/Validation data (text) |
| --------------------------------- | ------- | ------------ | ------------------------------- |
| `indonesian-roberta-base-indonli` | 124M    | RoBERTa Base | `IndoNLI`                       |

## Evaluation Results

The model was trained for 5 epochs, with a batch size of 16, a learning rate of 2e-5, a weight decay of 0.1, and a warmup ratio of 0.2, with linear annealing to 0. The best model was loaded at the end.

| Epoch | Training Loss | Validation Loss | Accuracy |
| ----- | ------------- | --------------- | -------- |
| 1     | 0.989200      | 0.691663        | 0.731452 |
| 2     | 0.673000      | 0.621913        | 0.766045 |
| 3     | 0.449900      | 0.662543        | 0.770596 |
| 4     | 0.293600      | 0.777059        | 0.768320 |
| 5     | 0.194200      | 0.948068        | 0.764224 |

## How to Use

### As NLI Classifier

```python
from transformers import pipeline

pretrained_name = "w11wo/indonesian-roberta-base-indonli"

nlp = pipeline(
    "sentiment-analysis",
    model=pretrained_name,
    tokenizer=pretrained_name
)

nlp("Andi tersenyum karena mendapat hasil baik. </s></s> Andi sedih.")
```

## Disclaimer

Do consider the biases which come from both the pre-trained RoBERTa model and the `IndoNLI` dataset that may be carried over into the results of this model.

## References

[1] Mahendra, R., Aji, A. F., Louvan, S., Rahman, F., & Vania, C. (2021, November). [IndoNLI: A Natural Language Inference Dataset for Indonesian](https://arxiv.org/abs/2110.14566). _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.

## Author

Indonesian RoBERTa Base IndoNLI was trained and evaluated by [Wilson Wongso](https://w11wo.github.io/). All computation and development are done on Google Colaboratory using their free GPU access.