Llama-3.2-3B-Instruct LoRA Fine-Tuned Model

Overview

This is a fine-tuned LoRA (Low-Rank Adaptation) model based on the Llama-3.2-3B-Instruct base model. The fine-tuning process was performed for a tipification analysis task, targeting categories of incidents such as "ESTAFA," "ROBO," and their attempted variations.

The model leverages LoRA (Low-Rank Adaptation) for parameter-efficient fine-tuning, optimizing just over 24M trainable parameters while freezing the base model weights.

Key Features

Base Model: unsloth/Llama-3.2-3B-Instruct
Task Type: Causal Language Modeling (CAUSAL_LM)
LoRA Parameters:
- r: 16
- lora_alpha: 16
- lora_dropout: 0.0
Target Modules: gate_proj, up_proj, down_proj, k_proj, q_proj, o_proj, v_proj
Training Loss:
- Started at 0.779
- Finalized at 0.6469 after 93 steps
Number of Trainable Parameters: 24,313,856

Dataset Distribution

The model was fine-tuned on a dataset with the following category distribution:

Category	Count	Percentage
ESTAFA	4610	47.3%
ROBO	2307	23.7%
HURTO	2141	22.0%
TENTATIVA DE ESTAFA	306	3.1%
TENTATIVA DE ROBO	272	2.8%
TENTATIVA DE HURTO	113	1.2%
Total	9749	100%

Training Details

Hardware: Single GPU
Dataset Size: ~10,000 examples
Epochs: 1
Batch Size per Device: 32
Gradient Accumulation Steps: 4
Effective Total Batch Size: 128
Steps: 93

Training Loss

Step	Training Loss
10	0.7790
20	0.6961
30	0.7048
40	0.6847
50	0.6876
60	0.6723
70	0.6443
80	0.6496
90	0.6469

Deployment Instructions

This model is compatible with the Hugging Face Inference API and can be deployed for text generation or classification tasks. Follow the steps below to load and use the model:

Load the Model

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("<your-huggingface-username>/<repo-name>")
model = AutoModelForCausalLM.from_pretrained("<your-huggingface-username>/<repo-name>")

input_text = "TENTATIVA DE ESTAFA:"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Use Cases

This fine-tuned model is particularly useful for:

Classifying text into predefined categories of "ESTAFA", "ROBO", "HURTO", and their "TENTATIVA DE" variations.
Generating text based on the fine-tuned dataset categories.

Fine-Tuning Colab

If you want to fine-tune a similar model, you can refer to the Colab notebook used for this fine-tuning. [Provide the Colab link if applicable]

Limitations

Category Imbalance: Categories such as "TENTATIVA DE HURTO" (1.2% of the dataset) and "TENTATIVA DE ROBO" (2.8% of the dataset) are underrepresented, which may affect the model’s performance on these categories.
Single Epoch Training: Further training may improve performance, especially for smaller categories.

Future Improvements

Augment dataset for better representation of underrepresented categories.
Train for additional epochs to improve classification accuracy and generalization.

Contact

For questions or feedback, feel free to reach out via Hugging Face or email.

Citation

If you use this model, please consider citing:

@misc{unsloth_lora_tipification,
  author = {Your Name},
  title = {Llama-3.2-3B-Instruct LoRA Fine-Tuned Model},
  year = {2024},
  howpublished = {\url{https://huggingface.co/<your-username>/<repo-name>}},
}

PEFT 0.14.0

Petermoyano
/

Llama-3.2-3B-Instruct-LoRA-Tipification-CausalLM-16R-16Alpha

You need to agree to share your contact information to access this model