Filled in model card with model info, training and evaluation detail and results; remaining info pending

README.md

@@ -1,12 +1,23 @@
 ---
 library_name: transformers
-tags: []
+tags:
+- nepali
+- roberta
+- nlp
+- language-model
+datasets:
+- IRIISNEPAL/Nepali-Text-Corpus
+language:
+- ne
+metrics:
+- f1
+- accuracy
 ---
 
 # Model Card for Model ID
 
 <!-- Provide a quick summary of what the model is/does. -->
-
+IRIISNEPAL/RoBERTa_Nepali_110M is a RoBERTa-based transformer model developed specifically for the Nepali language. This 110-million-parameter model is intended for tasks in natural language understanding (NLU), such as sentiment analysis, text classification, and named entity recognition in Nepali.
 
 
 ## Model Details
@@ -15,23 +26,15 @@ tags: []
 
 <!-- Provide a longer summary of what this model is. -->
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+- **Developed by:** Institute of Research and Innovation in Intelligent Systems (IRIIS) 
+- **Model type:** RoBERTa-based transformer model specifically trained on Nepali language data
+- **Model Size:** 110 million parameters
+- **Language (NLP):** Nepali
+- **Training Objective:** Masked Language Modeling (MLM) and Next Sentence Prediction (NSP)
 
-### Model Sources [optional]
+The IRIISNEPAL/RoBERTa_Nepali_110M model aims to provide a robust tool for NLP tasks specific to the Nepali language, supporting NLP research and applications within low-resource languages.
 
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+---
 
 ## Uses
 
@@ -40,59 +43,84 @@ This is the model card of a 🤗 transformers model that has been pushed on the
 ### Direct Use
 
 <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+The model provides contextual embeddings for each token in an input sequence (`last_hidden_state`) and a pooled representation of the entire input (`pooler_output`). These outputs can be used for:
 
-[More Information Needed]
+- **Text Classification**: Using `pooler_output` to classify the overall sentiment, intent, or category of a sentence.
+- **Token-Level Tasks**: Leveraging `last_hidden_state` to perform tasks like named entity recognition (NER) or part-of-speech tagging by predicting labels for individual tokens.
+- **Sentence Embeddings**: Using `pooler_output` as an embedding for the entire input text for similarity search or clustering tasks. 
 
-### Downstream Use [optional]
+### Downstream Use
 
 <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+The model was evaluated on the [Nepali Language Understanding Evaluation (Nep-gLUE)](https://nepberta.github.io/nepglue/) benchmark, demonstrating strong performance across various natural language understanding (NLU) tasks:
 
-[More Information Needed]
-
-### Out-of-Scope Use
+- **Named Entity Recognition (NER)**: 93.74
+- **Part-of-Speech (POS) Tagging**: 97.52
+- **Categorical Classification (CC)**: 94.68
+- **Categorical Pair Similarity (CPS)**: 96.49
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+These results indicate the model’s effectiveness in capturing language nuances for multiple NLU tasks in Nepali.
 
-[More Information Needed]
+---
 
 ## Bias, Risks, and Limitations
 
 <!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
+The model may exhibit biases present in its training data, especially regarding social, cultural, and regional aspects of the Nepali language. Users should exercise caution when deploying it in applications that might perpetuate stereotypes or cultural biases.
 
 ### Recommendations
 
 <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+It’s advisable for users to monitor model outputs for fairness and avoid high-stakes applications without thorough testing. Fine-tuning or retraining may be necessary for sensitive applications.
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+---
 
 ## How to Get Started with the Model
 
 Use the code below to get started with the model.
 
-[More Information Needed]
+```python
+# Load model directly
+from transformers import AutoTokenizer, AutoModel
+
+tokenizer = AutoTokenizer.from_pretrained("IRIISNEPAL/RoBERTa_Nepali_110M")
+model = AutoModel.from_pretrained("IRIISNEPAL/RoBERTa_Nepali_110M")
+
+text = "नेपालमा पर्यटनको विकास गर्नुपर्ने आवश्यकता छ।"
+inputs = tokenizer(text, return_tensors="pt")
+outputs = model(**inputs)
+```
+
+---
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+The model was trained on a 27.5 GB Nepali language corpus compiled from 99 Nepali news websites. This dataset represents the largest Nepali language corpus to date, providing a significant expansion in training resources for the language. The preprocessing involved deduplication, translation/removal of non-Nepali content, and noise reduction.
 
-[More Information Needed]
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+You can find detailed information about the dataset in the [dataset card on Hugging Face](https://huggingface.co/datasets/IRIISNEPAL/Nepali-Text-Corpus).
 
 ### Training Procedure
 
 <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
 
+- **Training Regime:** Mixed precision (fp16) on TPU v4-8 hardware
+- **Batch Size:** 256
+- **Learning Rate:** 1e-4 with a warmup over the first 10,000 steps followed by linear decay
+
 #### Preprocessing [optional]
 
 [More Information Needed]
 
-
 #### Training Hyperparameters
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+- **Max Sequence Length:** 512 tokens
+- **Learning Rate Scheduler:** Linear with warmup
+- **Optimizer:** AdamW with β1 = 0.9, β2 = 0.999, and L2 weight decay of 0.01
+- **Dropout Probability:** 0.1 across all layers
+- **Activation Function:** GELU
 
 #### Speeds, Sizes, Times [optional]
 
@@ -100,6 +128,8 @@ Use the code below to get started with the model.
 
 [More Information Needed]
 
+---
+
 ## Evaluation
 
 <!-- This section describes the evaluation protocols and provides the results. -->
@@ -108,35 +138,28 @@ Use the code below to get started with the model.
 
 #### Testing Data
 
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
+The model was evaluated on the [Nepali Language Evaluation Benchmark (Nep-gLUE)](https://nepberta.github.io/nepglue/), which includes tasks like Named Entity Recognition (NER), Part-of-Speech (POS) Tagging, text classification, and categorical pair similarity.
 
 #### Metrics
 
 <!-- These are the evaluation metrics being used, ideally with a description of why. -->
 
-[More Information Needed]
+- Accuracy
+- F1 Score
 
 ### Results
 
-[More Information Needed]
-
-#### Summary
-
+On Nep-gLUE, the model outperformed existing state-of-the-art models with an overall score of 95.60, reflecting its strong language understanding capabilities.
 
+---
 
-## Model Examination [optional]
+## Model Examination
 
 <!-- Relevant interpretability work for the model goes here -->
 
-[More Information Needed]
+Performance analysis indicates robustness in capturing grammatical and syntactical features of Nepali. However, the model may have limited effectiveness in handling dialect-specific content or informal language.
+
+---
 
 ## Environmental Impact
 
@@ -150,50 +173,44 @@ Carbon emissions can be estimated using the [Machine Learning Impact calculator]
 - **Compute Region:** [More Information Needed]
 - **Carbon Emitted:** [More Information Needed]
 
-## Technical Specifications [optional]
+---
+
+## Technical Specifications
 
 ### Model Architecture and Objective
 
-[More Information Needed]
+RoBERTa architecture with 12 transformer layers, hidden size of 768, 12 attention heads, and 110 million parameters. This architecture facilitates strong bidirectional attention for accurate language understanding.
 
 ### Compute Infrastructure
 
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
+- **Hardware:** TPU v4-8 and Nvidia GeForce RTX 3090 GPUs
+- **Software:** Python, PyTorch, Hugging Face Transformers
 
-[More Information Needed]
+---
 
-## Citation [optional]
+## Citation 
 
 <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
 
 **BibTeX:**
 
-[More Information Needed]
+```
+@misc{IRIISNEPAL_RoBERTa_Nepali_110M,
+  title = {Development of Pre-trained Transformer-based Models for the Nepali Language},
+  author = {Thapa, Prajwal and Nyachhyon, Jinu and Sharma, Mridul and Bal, Bal Krishna},
+  year = {2024},
+  note = {Submitted to COLING 2025}
+}
+```
 
 **APA:**
 
-[More Information Needed]
-
-## Glossary [optional]
+```
+Thapa, P., Nyachhyon, J., Sharma, M., & Bal, B. K. (2024). Development of pre-trained transformer-based models for the Nepali language. Manuscript submitted for publication to COLING 2025.
+```
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
+---
 
 ## Model Card Contact
 
-[More Information Needed]
+For questions and support, contact IRIIS Nepal.