README.md

---
library_name: transformers
tags:
- LLMs-Aviation
- AI-Regulatory-Compliance
- RAC-AI-Colombia
license: apache-2.0
datasets:
- somosnlp/Reglamento_aeronautico_Colombiano_FULL_ChatML_format_gemma
language:
- es
widget:
- text: >
    <bos><start_of_turn>system

    You are a helpful AI assistant.

    Responde en formato json.

    Eres un agente experto en la normativa aeronautica Colombiana.<end_of_turn>

    <start_of_turn>user

    ¿Qué sucede con las empresas de servicios aéreos comerciales que no hayan
    actualizado su permiso de operación después del 31 de marzo de
    2024?<end_of_turn>

    <start_of_turn>model
---
# Model Card for Gemma-FULL-RAC-Colombia 🛫

<p align="center">
  <img src="https://cdn-uploads.huggingface.co/production/uploads/6419c2f6b4adb0e101b17b6c/JlVz94BKdUkRN8xU00EgT.png" alt="Model Illustration" width="500">
</p>

🌟 This model card proudly presents Gemma-FULL-RAC-Colombia, a groundbreaking language model fine-tuned with precision on the Colombian Aeronautical Regulations (Reglamento Aeronáutico Colombiano, RAC) dataset. Harnessing the advanced capabilities of Google's original Gemma model, this adaptation focuses keenly on the aviation regulatory sphere within Colombia.

## Table of Contents

- [Model Card for Gemma-FULL-RAC-Colombia 🛫](#model-card-for-gemma-full-rac-colombia-)
  * [Model Details 📈](#model-details-)
    + [Model Description 📘](#model-description-)
  * [Uses 🛠️](#uses-)
    + [Direct Use 🎯](#direct-use-)
    + [Downstream Use [optional] 🔄](#downstream-use-optional-)
    + [Out-of-Scope Use 🚫](#out-of-scope-use-)
  * [Bias, Risks, and Limitations ⚠️](#bias-risks-and-limitations-)
    + [Recommendations 📝](#recommendations-)
  * [How to Get Started with the Model 🚀](#how-to-get-started-with-the-model-)
  * [Training Details 🧠](#training-details-)
    + [Environment and Libraries 🖥️](#environment-and-libraries-)
    + [Model Architecture 🏗️](#model-architecture-)
    + [Fine-tuning with LoRA 🔧](#fine-tuning-with-lora-)
    + [Training Process 🚀](#training-process-)
    + [Training Data 📚](#training-data-)
    + [Training Procedure](#training-procedure)
  * [Environmental Impact](#environmental-impact)



## Model Details 📈

### Model Description 📘

🚀 The Gemma-FULL-RAC-Colombia model marks an innovative leap in deploying language models for niche domains like aviation law and regulation. Developed by Edison Bejarano and Nicolai Potes, it's finely tuned using LoRA (Low-Rank Adaptation), ensuring minimal adjustment to the model's weights while preserving the extensive knowledge and prowess of Google's Gemma. This meticulous approach guarantees not just the retention of broad language comprehension and generation skills but also a profound grasp of the terminologies, concepts, and regulatory frameworks particular to Colombian aviation.

- **Developed by:** [Edison Bejarano](https://huggingface.co/ejbejaranos) and [Nicolai Potes](https://huggingface.co/NickyNicky) ✨
- **Funded by:** Fundación Universitaria Los Libertadores 🏛️
- **Shared by [optional]:** This project is a collective endeavor of passionate NLP and aviation regulation specialists, united by a vision to make aviation laws more accessible and understandable. 👥
- **Model type:** Causal Language Model, specially fine-tuned with LoRA for the distinct area of Colombian Aeronautical Regulations. 📊
- **Language(s) (NLP):** Spanish, tailored for the Colombian legal and regulatory context. 🇨🇴
- **License:** Apache License. This open-source license ensures that the model can be freely used, modified, and distributed. Please check the Hugging Face model page for specific licensing details. 🔓
- **Finetuned from model [optional]:** Derived from Google's Gemma model, utilizing versions such as `gemma-2b-it` for comprehensive training. 🧠

### Loss Function through Epochs 📉
<p align="center">
  <img src="https://cdn-uploads.huggingface.co/production/uploads/6419c2f6b4adb0e101b17b6c/AYLICPk_dvjML0DI2gBVY.png" alt="Loss Function Graph" width="700">
</p>


## Uses 🛠️

The Gemma-FULL-RAC-Colombia model is designed to enhance the understanding and application of the Colombian Aeronautical Regulations (RAC) through natural language processing. It's tailored for professionals and enthusiasts in the aviation industry, regulatory agencies, legal experts, and AI researchers with an interest in domain-specific language model applications.

### Direct Use 🎯

The model can be directly used to generate text, answer questions, and provide insights related to the Colombian Aeronautical Regulations without further fine-tuning. It's ideal for creating educational content, simplifying legal language, and assisting in regulatory compliance efforts.

### Downstream Use [optional] 🔄

When fine-tuned, Gemma-FULL-RAC-Colombia can be integrated into larger systems for automated compliance checks, document summarization, and even training simulators for pilots and air traffic controllers, offering a deeper, contextual understanding of regulations.

### Out-of-Scope Use 🚫

Misuse includes any application that promotes unethical practices, misinterprets aviation law, or uses the model for malicious purposes. The model is not designed for navigational purposes or to replace professional legal advice.

## Bias, Risks, and Limitations ⚠️

The model, while powerful, has limitations inherent to AI, including biases present in the training data. It may not cover all nuances of aviation regulations outside of Colombia or adapt to changes in law without updates.

### Recommendations 📝

Users should verify model outputs against current regulations and consult with professionals for critical applications. Awareness of the model's scope and limitations is crucial for effective use.

## How to Get Started with the Model 🚀

```python
# Assuming Hugging Face's Transformers and Datasets are installed
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "ejbejaranos/gemma-FULL-RAC-Colombia_v2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
# Example usage
input_text = "Explain the regulations for drone flights in Colombia."
input_ids = tokenizer.encode(input_text, return_tensors="pt")
output = model.generate(input_ids, max_length=100)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

## Training Details 🧠

The Gemma-FULL-RAC-Colombia model was specifically fine-tuned on the "Reglamento Aeronáutico Colombiano FULL ChatML format gemma" dataset available through Hugging Face Datasets. This dataset encompasses a comprehensive collection of the Colombian Aeronautical Regulations (RAC), tailored to train the model for understanding and generating responses related to aviation regulations in Colombia.

### Environment and Libraries 🖥️

The training was executed in a Python environment leveraging the following main libraries:

- `transformers` for model loading and fine-tuning facilities.
- `datasets` for handling the dataset efficiently.
- `torch` for the underlying deep learning framework.
- `peft` for applying the Low-Rank Adaptation (LoRA) adjustments to the model.

### Model Architecture 🏗️

The base model is Google's Gemma, an advanced language model with billions of parameters, capable of understanding context and generating human-like text. For this project, specific versions of Gemma (e.g., `gemma-2b-it`) were used as starting points.

### Fine-tuning with LoRA 🔧

LoRA (Low-Rank Adaptation) was employed to adapt the Gemma model to the specific domain of Colombian Aeronautical Regulations without extensively retraining the entire model. This approach allows for the efficient update of the model's weights, focusing on the adaptation layers to integrate the domain-specific knowledge effectively.

- **LoRA Configurations:** r=64, lora_alpha=32, among other settings tailored to balance performance and efficiency.
### Training Process 🚀
Training was carried out considering the following key parameters:
- Batch size: Adapted based on the available computational resources.
- Learning rate: Fine-tuned for optimal convergence without overfitting.
- Epochs: The model was trained for a sufficient number of epochs to ensure that it adequately learned from the dataset, with early stopping criteria to prevent overfitting.
### Evaluation and Performance 📊
'''Pending'''
### Model Merging and Saving 💾
After fine-tuning, the LoRA-adjusted weights were merged back with the base Gemma model to create the final version of Gemma-FULL-RAC-Colombia. The model was then saved and made available through Hugging Face, ensuring easy access for further use and development.
The training process was meticulously documented to ensure reproducibility and facilitate further research and development in the domain of specialized language models for regulatory compliance and legal tech applications.
### Training Data 📚
The training data utilized for fine-tuning the Gemma-FULL-RAC-Colombia model is available on Hugging Face Datasets under the dataset identifier:
[somosnlp/Reglamento_aeronautico_Colombiano_FULL_ChatML_format_gemma](https://huggingface.co/datasets/somosnlp/Reglamento_aeronautico_Colombiano_FULL_ChatML_format_gemma)
This dataset comprises a targeted collection of the Colombian Aeronautical Regulations (RAC), specifically including regulations from RAC 1 to RAC 5. The selection is optimized for chat and language modeling tasks, aiming to train the model to understand and generate accurate responses related to the initial segments of Colombian aviation regulations.
### Training Procedure
The Gemma-FULL-RAC-Colombia model underwent a meticulous training process, leveraging the NVIDIA V100 GPU for its computational needs. The procedure was designed to fine-tune the model on a dataset comprising sections RAC 1 to RAC 5 of the Colombian Aeronautical Regulations.
#### Preprocessing
The dataset was preprocessed to convert the regulatory texts into a format suitable for the model, utilizing the tokenizer associated with the `google/gemma-2b-it` model. This step ensured that the input data was optimized for training efficiency and model comprehension.
#### Training Hyperparameters
The training utilized a mixed precision regime to balance computational efficiency and model accuracy. Specific hyperparameters included:
- **Training regime:** Mixed precision (FP16)
#### Speeds, Sizes, Times
The model was trained over a span of approximately 4.67 hours (runtime: 16,020 seconds), with checkpoints indicating progress at various intervals. The training demonstrated efficient utilization of the V100 GPU, maintaining a steady throughput.
#### Factors
Training focused on the model's ability to understand and generate text related to the specific domain of Colombian Aeronautical Regulations.
#### Metrics
The primary metric for evaluating the model's performance was the training loss, which was observed to decrease steadily over time, indicating effective learning and adaptation by the model.
### Results
The training process resulted in a model capable of accurately interpreting and responding to queries related to the Colombian Aeronautical Regulations, with a notable final training loss of 0.6204 at the last checkpoint.
## Environmental Impact
Given the use of an NVIDIA V100 GPU for approximately 4.67 hours, the carbon emissions can be estimated using the Machine Learning Impact calculator. This tool accounts for the hardware type, runtime, and other factors to provide a comprehensive view of the environmental impact of training large AI models.
- **Hardware Type:** NVIDIA V100 GPU
- **Hours used:** ~4.67
- **Carbon Emitted:** 554.5625
# Constants
power_consumption_kW = 0.25  # 250 watts in kW
runtime_hours = 4.67
carbon_intensity_gCO2eq_per_kWh = 475  # Global average carbon intensity

# Calculate carbon emissions
carbon_emitted_gCO2eq = power_consumption_kW * runtime_hours * carbon_intensity_gCO2eq_per_kWh
carbon_emitted_gCO2eq = 554.5625