---
library_name: peft
datasets:
- cis-lmu/bavarian_to_english
language:
- de
- bar
base_model:
- LSX-UniWue/LLaMmlein_1B
pipeline_tag: text-generation
license: other
---

# LLäMmlein 1B

This is a Bavarian adapter for the German Tinyllama 1B language model which was tuned on a dump of the Bavarian wikipedia, without further optimization. Please don't take it too seriously ;)
Find more details on our [page](https://www.informatik.uni-wuerzburg.de/datascience/projects/nlp/llammlein/) and our [preprint](arxiv.org/abs/2411.11171)!

## Run it
```py
import torch
from peft import PeftConfig, PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# script config
base_model_name = "LSX-UniWue/LLaMmlein_1B"
adapter_name = "LSX-UniWue/Betzerl_1B_wiki_preview"
device = "cuda"  # or mps

# load model
config = PeftConfig.from_pretrained(adapter_name)
base_model = model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.bfloat16,
    device_map=device,
)
base_model.resize_token_embeddings(32001)
model = PeftModel.from_pretrained(base_model, adapter_name)
tokenizer = AutoTokenizer.from_pretrained(adapter_name)
```