grimjim (Jim Lai)

posted an update 7 days ago

Post

1294

I was reading through an abstract and found myself wondering how much LLM performance is being left on the table due to insufficient curation of training datasets: "Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning" by Kaur, Park, Goyal, Arora.
https://arxiv.org/abs/2408.14774
In particular, the observation that "Introducing low quality answers ("shirkers") in 20% of Instruct-SkillMix examples causes performance to plummet..." had me wondering how many ostensibly good datasets out there are in fact populated with a significant number of "shirkers".

2 replies

·

posted an update 19 days ago

Post

2937

I found this paper to be thought-provoking: "Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling" by Bansal, Hosseini, Agarwal, Tran, and Kazemi.
https://arxiv.org/abs/2408.16737
The direct implication is that smaller models could be used to create cost-effective synthetic datasets. And on that note, in the Gemma terms of use, Google explicitly claims no rights on outputs generated from those models, which means one is free to synthgen from the Gemma line. Meta's Llama 3 licence forbids synthetic generation of outputs if used to improve other models. Relevant Mistral, Qwen, and Yi models under the Apache 2.0 license are unrestricted for this purpose.

2 replies

·

posted an update 24 days ago

Post

1613

This merge, this time grounded in Gemma2 9B Instruct fine-tunes, is another demonstration that models without any fine-tuning to support roleplay can still perform the function, maintaining coherence and attention to context. It should be evident that no overt fine-tuning is required for roleplay in text generation; pretraining should provide models with a requisite basic understanding of the world, so all that should be needed is some corrective fine-tuning to address observed defects in portraying the world along with datasets to promote a suitably entertaining writing style. Good Instruct tuning should promote reasoning, coherence, and attention to context.
grimjim/Kitsunebi-v1-Gemma2-8k-9B
grimjim/Kitsunebi-v1-Gemma2-8k-9B-GGUF

I opted not to incorporate the UCLA SPPO fine-tune for Gemma2 9B after observing context confusion occur with some frequency during complex scenarios.

Thanks to Axcxept co., ltd. for fine-tuning HODACHI/EZO-Common-9B-gemma-2-it, and to Princeton NLP Group for fine-tuning princeton-nlp/gemma-2-9b-it-SimPO.
AXCXEPT/EZO-Common-9B-gemma-2-it
princeton-nlp/gemma-2-9b-it-SimPO

posted an update about 2 months ago

Post

2721

I've observed that the layers targeted in various abliteration notebooks (e.g., https://colab.research.google.com/drive/1VYm3hOcvCpbGiqKZb141gJwjdmmCcVpR?usp=sharing ) appear to be arbitrary, reflecting probable brute-force exploration. This doesn't need to be the case.

Taking a cue from the paper "The Unreasonable Ineffectiveness of the Deeper Layers" ( https://arxiv.org/abs/2403.17887 ) and PruneMe (https://github.com/arcee-ai/PruneMe), it seems reasonable to target deeper layers identified as more redundant given measured similarity across layers, as the result should be less damaging to models, reducing the need for subsequent fine-tuning. Intuitively, one should expect the resulting intervention layers to be deep but not final. The only uncertainty is if the redundancy successfully encodes refusals, something which is almost certainly model-dependent. This approach only requires the redundancy to be computed once per model, and the result used as a starting point for which layer range to restrict intervention to.

posted an update about 2 months ago

Post

4163

I've come across theoretical justification for my prior experimentation with extremely low-weight mergers: they amount to flattening a model so its "massive activation" features remain as significant contributors. Extremely low-weight merge weights also effectively sparsify a contributing model with regard to the base model, but in a way which still preserves relationships within the flattened latent space. In the paper "Massive Activations in Large Language Models", the authors observed "very few activations exhibit significantly larger values than others (e.g., 100,000 times larger)", which in turn implies a lower bound in effective application of extremely low weight merging.
https://arxiv.org/abs/2402.17762

1 reply

·

replied to their post about 2 months ago

Not the same model, but a related model.

I created what amounted to an abliteration LoRA by contrasting original L3 Instruct against failspy's abliterated L3 Instruct, then applied and merged the L3-derived LoRA on top of original L3.1 Instruct to obtain the final model.

The result appears to outperform mlabonne's reapplication of the abliteration technique directly to L3.1 Instruct.

replied to their post about 2 months ago

An example use of mergekit to apply the lora is documented at the bottom of the model card as well as in mergekit_config.yaml. Although task_arithmetic was used, a passthrough merge will work as well.

An example of the mergekit LoRA extraction command is at the bottom of the model card for the LoRA:
https://huggingface.co/grimjim/Llama-3-Instruct-abliteration-LoRA-8B

replied to their post about 2 months ago

Select GGUF quants.
https://huggingface.co/grimjim/Llama-3.1-8B-Instruct-abliterated_via_adapter-GGUF

posted an update about 2 months ago

Post

2263

In principle, it's possible to "abliterate" refusals in any Llama 3.1 8B models via application of a LoRA, using only mergekit.

Proof of concept below:
grimjim/Llama-3.1-8B-Instruct-abliterated_via_adapter

5 replies

·

posted an update about 2 months ago

Post

2740

Intelligence is all you need for roleplay.
Roleplay is overlooked as a special case of chain-of-thought, where context must be attended to and inferred state of the world and embodied minds must be persisted and evolved along credible narrative lines. LLMs are also being tasked to function as gamemasters. It's a challenging task which points to potential future benchmarks. The fact that the largest commercial LLMs are adept in generating text for roleplay intuitively implies that model intelligence is sufficient so long as it can generalize properly and pay attention to context without becoming confused.
This recent merge of mine composed using 3 academic fine-tunes, none of which were intended for roleplay, has survived the gauntlet of a Reddit post and appears to be a particularly strong 8B model when it comes to roleplay coherence.
grimjim/llama-3-Nephilim-v3-8B (bf16 weights)
grimjim/llama-3-Nephilim-v3-8B-GGUF (select quants)

1 reply

·

replied to their post 3 months ago

Something odd is happening when merging with the OVA model. It will reduce refusals at medium (0.5-0.6) weight, but at full (1.0) weight against Instruct 8B, the result is incoherent. The LoRA should work, though!

It should also be possible to modify abliteration scripts to instead directly produce a LoRA as output.

posted an update 3 months ago

Post

2250

Below we experiment with negative merger weighting (-1.0!) using task arithmetic. Merge formula on the model card and in the repo itself.

This model is steered to behave opposite to what MopeyMule demonstrated.

Based on the implications of the merge technique, we also propose Orthogonalized Vector Adaptation (OVA). We also extract a LoRA of the counter-refusal abliteration steering vector.

The resulting merger is not a perfect model, but it's a behaviorally interesting model. The model name was inspired by a Philip K. Dick story.
grimjim/Llama-3-Perky-Pat-Instruct-8B

Refusal vector weights ready for use:
grimjim/Llama-3-Instruct-abliteration-OVA-8B
grimjim/Llama-3-Instruct-abliteration-LoRA-8B

3 replies

·

posted an update 3 months ago

Post

2673

Uploaded two basic SLERP merges of princeton-nlp/Llama-3-Instruct-8B-SimPO and UCLA-AGI/Llama-3-Instruct-8B-SPPO-Iter3, alternating the choice of base model, for people to test out and potentially use as merge fuel. (Personally, I am drawn to intelligent and attentive models, hence the experimentation.)

grimjim/Llama-3-Instruct-8B-SPPO-Iter3-SimPO-merge
grimjim/Llama-3-Instruct-8B-SimPO-SPPO-Iter3-merge

posted an update 4 months ago

Post

2180

We explore extremely low-weight merger as an alternative to fine-tuning; e.g., weight 1e-4. Merge formula details here:
grimjim/kukulemon-v3-soul_mix-32k-7B

posted an update 4 months ago

Post

1679

I propose "merge densification", a style of merger which attempts to transfer the benefits of a denser model to a base model. The model weight in this case is 0.02, which is atypically small for mergers, but high compared to the learning rate used during training. In this case, the expectation is more creative text-generation. More details below:
grimjim/kunoichi-lemon-royale-v3-32K-7B

posted an update 4 months ago

Post

1373

I use mergekit regularly, and often enough get acceptable results without performing fine-tuning afterward. My current thinking is that DARE-TIES should be avoided when merging dense models, as the process of thinning inherently punches holes in models.

I've had success using SLERP merges to graft Mistral v0.1 models with Mistral v0.2 models to obtain the context length benefits of the latter, and am looking forward to experimenting with Mistral v0.3, which recently dropped.

1 reply

·

posted an update 4 months ago

Post

1752

I hardly tested this text generation merged model before releasing it a few months ago. Now it's been receiving attention recently in the form of quants.
grimjim/kunoichi-lemon-royale-v2-32K-7B

Judging from download numbers for GGUF quants, people appear to be using it, and at least one person has a merge formula that incorporated the model.
bartowski/kunoichi-lemon-royale-v2-32K-7B-GGUF
bartowski/kunoichi-lemon-royale-v2-32K-7B-exl2
mradermacher/kunoichi-lemon-royale-v2-32K-7B-GGUF

Jim Lai

AI & ML interests

Organizations

grimjim's activity