File size: 1,910 Bytes
df0824d ea338f5 df0824d 15635b1 df0824d 007231c df0824d 15635b1 df0824d 15635b1 df0824d ea338f5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 |
# Llama 3.2 11B-Vision-Instruct Model on Hugging Face
This repository hosts the `Llama 3.2 11B-Vision-Instruct` model, fine-tuned for generating TikZ code from captions and images, suitable for enhancing scientific visualizations.
## Model Description
The `Llama 3.2 11B-Vision-Instruct` is a multimodal model combining the robust textual understanding and generative capabilities of LLaMA 3.2 with a specialized vision encoder, aimed at integrating detailed visual embeddings with textual data for high-quality output.
## Installation
Ensure you have PyTorch and Transformers installed in your environment. If not, you can install them using pip:
```bash
pip install torch transformers
```
```bash
import torch
from datetime import date
from PIL import Image, ImageTk
from transformers import MllamaForConditionalGeneration, AutoProcessor
import tkinter as tk
from tkinter import filedialog, ttk, messagebox
import logging
import json
import os
from peft import LoraConfig, get_peft_model
# Configure logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
# Get today's date
date_string: str = date.today().strftime("%d %b %Y")
model_id = "oe2015/llama-3.2-finetuned-tikzcode"
lora_config = LoraConfig(
r=16, # Rank of the decomposition, typically a small number (e.g., 8, 16)
lora_alpha=32, # Scaling factor for LoRA parameters
target_modules=["q_proj", "v_proj"], # Apply LoRA to specific model layers (e.g., Q, V projections in attention layers)
lora_dropout=0.1, # Dropout for LoRA layers
bias="none" # LoRA doesn’t update biases by default; change to "all" if needed
)
# Load the model and processor
model = MllamaForConditionalGeneration.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
model = get_peft_model(model, lora_config)
processor = AutoProcessor.from_pretrained(model_id)
``` |