|
--- |
|
base_model: |
|
- unsloth/Llama-3.2-1B-Instruct |
|
library_name: transformers |
|
language: |
|
- en |
|
license: cc0-1.0 |
|
tags: |
|
- unsloth |
|
- gguf |
|
--- |
|
### WIP! Results for now are total trash and not worth your time! almost not working! Finnnegan's birth at best, but closer to him being concieved yet. Too far from his wake! No word-play at all! |
|
|
|
PEFT Finnegan-tuned LLaMA 3.2-1B-instruct on part of Finnegans Wake dataset for text generation in the style of James Joyce. |
|
|
|
Space: https://huggingface.co/spaces/genaforvena/huivam_finnegans_spaceship |
|
|
|
## Iteration 3: |
|
Realized that was doing it all wrong and this tie used https://huggingface.co/unsloth/Llama-3.2-1B-Instruct and collab available from there. Only changed dataset. |
|
|
|
My collab is here: https://colab.research.google.com/drive/1JrqcU9idXXR3Wru5mw2e6Uh2TKJWwu7U?usp=sharing |
|
|
|
The only difference: Created dataset like below |
|
``` |
|
from unsloth.chat_templates import get_chat_template |
|
import json |
|
import random |
|
from transformers import AutoTokenizer |
|
from unsloth.chat_templates import get_chat_template # For chat template formatting |
|
from datasets import Dataset, load_dataset |
|
|
|
# Configuration |
|
INPUT_FILE = "finnegans_30.txt" # Path to your Finnegans Wake text file |
|
OUTPUT_FILE = "finnegans_wake_dataset.jsonl" # Local file to save the dataset |
|
CHUNK_SIZE = 24 |
|
|
|
# Apply the chat template |
|
tokenizer = get_chat_template( |
|
tokenizer, |
|
chat_template="llama-3.1", # Use the LLaMA-3.1 chat template |
|
) |
|
|
|
# Load the text |
|
with open(INPUT_FILE, "r", encoding="utf-8") as file: |
|
text = file.read() |
|
|
|
# Tokenize the text |
|
tokens = tokenizer.encode(text, truncation=False, add_special_tokens=False) |
|
|
|
# Split tokens into chunks |
|
chunks = [tokens[i:i + CHUNK_SIZE] for i in range(0, len(tokens), CHUNK_SIZE)] |
|
|
|
# Prepare dataset in conversational format |
|
dataset = [] |
|
for chunk in chunks: |
|
chunk_text = tokenizer.decode(chunk, skip_special_tokens=True) |
|
|
|
# Split the chunk into three parts randomly |
|
split_points = sorted(random.sample(range(len(chunk_text)), 2)) # Two random split points |
|
context = chunk_text[:split_points[0]] |
|
instruction = chunk_text[split_points[0]:split_points[1]] |
|
response = chunk_text[split_points[1]:] |
|
|
|
# Format as a conversation |
|
conversation = [ |
|
{"role": "user", "content": f"### GIVEN THE CONTEXT: {context} ### INSTRUCTION: {instruction}"}, |
|
{"role": "assistant", "content": response}, |
|
] |
|
|
|
# Add to dataset |
|
dataset.append({"conversations": conversation}) |
|
|
|
# Save dataset locally as a .jsonl file |
|
with open(OUTPUT_FILE, "w", encoding="utf-8") as file: |
|
for item in dataset: |
|
json.dump(item, file) |
|
file.write("\n") |
|
|
|
print(f"Dataset saved locally to {OUTPUT_FILE}") |
|
|
|
# Apply the formatting function |
|
def formatting_prompts_func(examples): |
|
convos = examples["conversations"] |
|
texts = [tokenizer.apply_chat_template(convo, tokenize=False, add_generation_prompt=False) for convo in convos] |
|
return {"text": texts} |
|
|
|
# Apply the formatting function using Dataset.from_dict |
|
dataset = Dataset.from_dict({"conversations": [d['conversations'] for d in dataset]}) |
|
|
|
formatted_dataset = dataset.map(formatting_prompts_func, batched=True, remove_columns=['conversations']) |
|
|
|
# Save the formatted dataset |
|
formatted_dataset.to_json("formatted_finnegans_wake_dataset.jsonl") |
|
print("Formatted dataset saved to formatted_finnegans_wake_dataset.jsonl") |
|
|
|
# Load the formatted dataset using load_dataset |
|
dataset = load_dataset("json", data_files="formatted_finnegans_wake_dataset.jsonl", split="train") |
|
dataset = dataset |
|
``` |
|
|
|
## Iteration 2 (Fail): |
|
|
|
Dataset: same (forgot to save config with new dataset). |
|
|
|
finnetune.yaml: |
|
``` |
|
# The ID of the dataset you created |
|
dataset: huivam-finnegans-2 |
|
|
|
# Configuration for text completion fine-tuning |
|
text_completion: |
|
# How the fields of the JSON dataset should be formatted into the input text |
|
input_template: "### GIVEN THE CONTEXT: {context} ### INSTRUCTION: {instruction} ### RESPONSE IS: " |
|
|
|
# How the fields of the JSON dataset should be formatted into the output text |
|
output_template: "ANSWER: {response}" |
|
|
|
# The Fireworks model name of the base model |
|
base_model: accounts/fireworks/models/llama-v3p2-1b-instruct |
|
``` |
|
|
|
Finne-tuning commands used: |
|
``` |
|
./firectl create dataset huivam-finnegans-2 .\finnegans_wake_dataset_2.jsonl |
|
./firectl create fine-tuning-job --settings-file finnetune.yaml --epochs=3 --learning-rate=2e-5 --batch-size=8 |
|
``` |
|
|
|
New params used to finne-tune: |
|
``` |
|
Text Completion: |
|
Input Template: ### GIVEN THE CONTEXT: {context} ### INSTRUCTION: {instruction} ### RESPONSE IS: |
|
Output Template: ANSWER: {response} |
|
Base Model: accounts/fireworks/models/llama-v3p2-1b-instruct |
|
Epochs: 3 |
|
Learning Rate: 2e-05 |
|
Lora Rank: 8 |
|
Batch Size: 8 |
|
Evaluation Split: 0 |
|
``` |
|
|
|
Spent: $0.08 |
|
Time: 5 mins |
|
|
|
## Iteration 1: |
|
|
|
Dataset I prepared like that: |
|
``` |
|
# Load the tokenizer |
|
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME) |
|
|
|
# Load the text |
|
with open(INPUT_FILE, "r", encoding="utf-8") as file: |
|
text = file.read() |
|
|
|
# Tokenize the text |
|
tokens = tokenizer.encode(text, truncation=False, add_special_tokens=False) |
|
|
|
# Split tokens into chunks |
|
chunks = [tokens[i:i + CHUNK_SIZE] for i in range(0, len(tokens), CHUNK_SIZE)] |
|
|
|
# Prepare dataset |
|
dataset = [] |
|
for chunk in chunks: |
|
chunk_text = tokenizer.decode(chunk, skip_special_tokens=True) |
|
|
|
# Split the chunk into three parts randomly |
|
split_points = sorted(random.sample(range(len(chunk_text)), 2)) # Two random split points |
|
context = chunk_text[:split_points[0]] |
|
instruction = chunk_text[split_points[0]:split_points[1]] |
|
response = chunk_text[split_points[1]:] |
|
|
|
# Add to dataset |
|
dataset.append({ |
|
"context": context, |
|
"instruction": instruction, |
|
"response": response, |
|
}) |
|
|
|
# Save dataset locally as a .jsonl file |
|
with open(OUTPUT_FILE, "w", encoding="utf-8") as file: |
|
for item in dataset: |
|
json.dump(item, file) |
|
file.write("\n") |
|
|
|
print(f"Dataset saved locally to {OUTPUT_FILE}") |
|
``` |
|
|
|
Example of dataset entry: |
|
``` |
|
{"context": "riverrun, past Eve and Adam's, from swerve of shore to bend of bay...", "instruction": "Sir Tristram, violer d'amores, fr'over the short sea...", "response": "O here here how hoth sprowled met the duskt the father of fornicationists..."} |
|
``` |
|
|
|
fine-tuned on 1/10th of text on fireworks.ai with params: |
|
``` |
|
dataset: finnegans_wake_dataset |
|
|
|
text_completion: |
|
# How the fields of the JSON dataset should be formatted into the input text |
|
input_template: "### GIVEN THE CONTEXT: {context} ### INSTRUCTION: {instruction} ### RESPONSE IS: " |
|
|
|
# How the fields of the JSON dataset should be formatted into the output text |
|
output_template: "ANSWER: {response}" |
|
|
|
# The Fireworks model name of the base model |
|
base_model: accounts/fireworks/models/llama-v3p2-1b |
|
|
|
# Hyperparameters for fine-tuning (should be passed as args and removed from here) |
|
hyperparameters: |
|
learning_rate: 1e-5 # Learning rate for the optimizer |
|
epochs: 1 # Number of epochs to train |
|
batch_size: 4 # Batch size for training |
|
``` |
|
|
|
Spent: $0.01 |
|
Time: 2 mins |
|
|
|
Result: Seemingly not enough data to affect model output. |
