Create README.md
Browse filesSabiá-7B is Portuguese language model developed by [Maritaca AI](https://www.maritaca.ai/).
**Input:** The model accepts only text input.
**Output:** The Model generates text only.
**Model Architecture:** Sabiá-7B 2 is an auto-regressive language model that uses the same architecture of LLaMA-1-7B.
**Tokenizer:** It uses the same tokenizer as LLaMA-1-7B.
**Maximum sequence length:** 2048 tokens.
**Pretraining data:** The model was pretrained on 7 billion tokens from the Portuguese subset of ClueWeb22, starting with the weights of LLaMA-1-7B and further trained for an additional 10 billion tokens, approximately 1.4 epochs of the training dataset.
**Data Freshness:** The pretraining data has a cutoff of mid-2022.
**License:** The licensing is the same as LLaMA-1's, restricting the model's use to research purposes only.
**Paper:** For more details, please refer to our paper: [Sabiá: Portuguese Large Language Models](https://arxiv.org/pdf/2304.07880.pdf)
Given that Sabiá-7B was trained solely on a language modeling objective without fine-tuning for instruction following, it is recommended for few-shot tasks rather than zero-shot tasks.
**Results**
Below we show the results on the Poeta benchmark, which consists of 14 Portuguese datasets:
|Model | NPM |
|--|--|
|LLaMA-1-7B| 33.0|
|LLaMA-2-7B| 43.7|
|Sabiá-7B| 48.5|
For more information on the Normalized Preferred Metric (NPM), please check out our paper.
Please use the following bibtex to cite our paper:
```
@InProceedings{10.1007/978-3-031-45392-2_15,
author="Pires, Ramon
and Abonizio, Hugo
and Almeida, Thales Sales
and Nogueira, Rodrigo",
editor="Naldi, Murilo C.
and Bianchi, Reinaldo A. C.",
title="Sabi{\'a}: Portuguese Large Language Models",
booktitle="Intelligent Systems",
year="2023",
publisher="Springer Nature Switzerland",
address="Cham",
pages="226--240",
isbn="978-3-031-45392-2"
}
```