LoRA
mergekit-extract-lora AuraIndustries/Aura-8B arcee-ai/Llama-3.1-SuperNova-Lite OUTPUT_PATH --no-lazy-unpickle --skip-undecomposable --rank=32 --extend-vocab --model_name=Aura-r32-LoRA --verbose
Does this command extract "all" LoRAs or "all" finetuning from the base model or something else? Sorry ahead of time for such a mundane question, but I have not created LoRAs before and I am very interested in trying to delve into it.
-Also, if it isn't too tedious, are there any environment variables specifically needed or is there a HF Space to help perform the task?
No problem,
The best way I can explain it from my understanding is that when you're pulling a LoRA in mergekit, you're taking the difference between the base model and the finetune with a max rank of x value. Rank will determine how much data you pull with a max of 512 I believe being a 'full finetune'. Though, I've never had much success with a rank that high, so ymmv. The best results I've found come from a rank range from 16-256. Remove --skip-undecomposable if you intend to take as much of the finetune as possible, I have it is since I do a lot of cross archetype merging (L3 to L3.1/3.2/3.3) and the undecompostable embeds negatively affect my results.
To run mergekit's LoRA extraction, you'll need to install bitsandbytes on whatever device you're using. I use Runpod for my merging/lora stuff so I have a Jupyter notebook with all my variables accounted for. Also, for the best results, choose the last known model that was tuned into the finetune in question. To take this repo's LoRA as an example, Aura was finetuned from SuperNova Lite that itself is a finetune of Llama 3.1 Instruct. You can pull Aura off L3.1 Instruct but you wouldn't have the same personality/data as if you were pulling Aura off SuperNova. Again, merging is the wild west of the AI community, so go wild to see what works for you.
Hope that all helps and good luck