---
license: cc-by-nc-4.0
base_model:
- kakaocorp/kanana-nano-2.1b-instruct
---
# **Directional Enhancement for Language Models: A Novel Approach to Specialization without Fine-Tuning**

## **Overview**

This model is Made by muzerai aka "AI JOAH" using kakaocorp/kanana-nano-2.1b-instruct. (test purpose)

Subscribe My Youtube Channel [AI JOAH](https://www.youtube.com/@JayLee-gv8tv)

This project presents a methodology for enhancing specific capabilities of language models using the **Directional Enhancement** technique. **This approach does not introduce new knowledge into the model but amplifies its existing latent abilities.** While preserving the general capabilities of the language model, it significantly improves performance in specific domains such as creative writing, education, and technical documentation.

This is a creative writing direction enhancement version of [kakaocorp/kanana-nano-2.1b-instruct](https://huggingface.co/kakaocorp/kanana-nano-2.1b-instruct)

if enhance.txt are changed by the specific domain, this model style can be changed with that domain type. this test only use 95 instruction for that creative domain field.

## **Technical Background**

### **Principle of Directional Enhancement**

This approach identifies a **specialization direction** in the representation space of the language model, associated with a specific capability, and enhances the model’s attention weights in that direction.

1. Compute the difference in representation between **specialized prompts** (domain-specific) and **general prompts** within the model's hidden states.
2. Normalize this difference vector to obtain the **specialization direction**.
3. Enhance the model’s **self-attention output projection weights (`o_proj`)** along this specialized direction.

**This method strengthens the model’s intrinsic abilities rather than introducing completely new knowledge or patterns.** It functions similarly to how a lens amplifies a specific wavelength of light.

### **Computing Specialization Direction**

Unlike conventional fine-tuning, which modifies all weights in the model, this approach **identifies a targeted enhancement direction** by analyzing differences in activations across specialized and general inputs.

- A set of **specialized** prompts (`enhance.txt`) and **general** prompts (`normal.txt`) are fed into the model.
- The activations of a **chosen hidden layer** are extracted for both prompt types.
- The **mean hidden state vector** for specialized prompts is computed and compared to the mean hidden state vector for general prompts.
- Their difference represents the **specialization direction**, which is then **normalized** to create a unit vector.

### **Enhancing Model Weights**

Once the **specialization direction** is computed, it is applied to modify the model’s **self-attention output projection weights (`o_proj`)** in a controlled manner:

1. The specialization direction is **projected** onto the weight matrix of each attention layer.
2. A **scaled enhancement factor** is applied to align the model’s attention outputs more strongly with the specialization direction.
3. This process **amplifies** the model’s responses in the desired direction without altering its fundamental structure.

This targeted adjustment allows the model to **focus more on specific characteristics** (e.g., creativity, technical accuracy, formal tone) while maintaining general competency.

## **Comparison with Existing Methods**

| **Method**                   | **Features** |
|-----------------------------|-------------|
| **Traditional Fine-Tuning** | Updates the entire model’s weights, requiring significant computational resources and extensive training data. Enables learning new knowledge and patterns. |
| **Lightweight Fine-Tuning (LoRA, etc.)** | Adds adaptive low-rank matrices to optimize fine-tuning. More efficient but still requires training. |
| **Directional Enhancement (this method)** | Selectively **amplifies** the model’s intrinsic capabilities by strengthening specialized output directions. Does not introduce new knowledge. |

## **Implementation Details**

### **Data Preparation**

Two types of datasets are used to define the specialization direction:
- **Specialized Dataset (`enhance.txt`)**: Contains prompts focused on the capability to be enhanced.
- **General Dataset (`normal.txt`)**: Contains diverse, neutral prompts to serve as a baseline.

The difference in activations between these two datasets defines the specialization direction, ensuring that the enhancement is aligned with the target capability while preserving the model’s general functionality.

### **Key Parameters**

- **`instructions`**: Number of instruction samples to process (default: 95) -> more data -> can be increased
- **`layer_idx`**: Index of the model layer where specialization direction is computed (default: 60% of total layers)
- **`enhancement_factor`**: Strength of enhancement along the specialization direction (default: 1.5)

### **Core Algorithm**

```python
# Compute specialization direction
specialization_dir = specialized_mean - general_mean
specialization_dir = specialization_dir / specialization_dir.norm()

# Core part of the weight enhancement algorithm
projection_scalars = torch.matmul(attn_output, specialization_dir)
projection = torch.outer(projection_scalars, specialization_dir)
enhanced_weights = attn_output + enhancement_factor * projection
```

### **Improvements in Creative Writing Model**

Experiments with creative writing models demonstrate **significant qualitative improvements**:

- **Enhanced Descriptive Ability**: More vivid and detailed descriptions with richer sensory language.
- **Improved Character Development**: Clearer character traits and more distinct personalities.
- **Enhanced Dialogue Generation**: More natural and engaging conversational exchanges.
- **Stronger Story Structuring**: Improved narrative flow and coherence.
- **Increased Emotional Depth**: Greater emotional nuance and expressiveness.

## **Applications**

This technique can be applied to various specialized models:

- **Creative Writing Models**: Optimized for novel writing, poetry, and storytelling.
- **Educational Content Models**: Tailored for clear, structured, and pedagogical explanations.
- **Technical Documentation Models**: Enhanced for structured and precise documentation.
- **Business Communication Models**: Specialized for professional and formal business writing.
- **Medical/Scientific Models**: Improved for detailed and accurate scientific explanations.

## **Limitations and Future Improvements**

### **Current Limitations**

- **Interpretability of Specialization Directions**: Difficult to precisely determine what specific abilities are being enhanced.
- **Single-Direction Specialization**: Currently enhances only one specific capability at a time.
- **Control Over Enhancement Level**: The optimal enhancement factor is determined empirically.
- **No New Knowledge Acquisition**: Cannot introduce entirely new knowledge beyond what the model already possesses.
- **Dependence on Existing Abilities**: If the model lacks fundamental knowledge in a domain, the enhancement effects are limited.

### **Future Directions**

- **Multi-Directional Enhancement**: Developing techniques to enhance multiple capabilities simultaneously.
- **Automatic Tuning**: Implementing an automated method for optimal enhancement factor selection.
- **Interpretability of Specialization**: Researching better semantic analysis of specialization directions.
- **User-Personalized Specialization**: Customizing specialization directions based on user preferences.
- **Hybrid Approach**: Combining **directional enhancement** with lightweight fine-tuning to enable both ability enhancement and new knowledge learning.

## **Conclusion**

The **Directional Enhancement** technique provides an efficient way to strengthen specific capabilities of language models **without requiring full retraining or additional training data**. While it does not introduce new knowledge, it **amplifies latent abilities** with minimal computational cost. This method offers a practical approach for developing AI models tailored to specialized domains.

## License

The `Kanana` models are licensed under [CC-BY-NC-4.0](https://spdx.org/licenses/CC-BY-NC-4.0).

## **Citation**
```
@misc{DirectionalEnhancement2025,
       title={Directional Enhancement for Language Models: A Novel Approach to Specialization without Fine-Tuning},
       author={AI JOAH},
       year={2025},
       url={https://www.youtube.com/@JayLee-gv8tv},
}
```

## Contact
- AI JOAH : utxopool@gmail.com