Usage for HuggingFace

README.md

@@ -12,13 +12,13 @@ base_model:
 - stabilityai/japanese-stablelm-base-gamma-7b
 ---
 
-# Hinoki-Sak-Sta-slerp-7B
+# 🌲 Hinoki-Sak-Sta-slerp-7B
 
 Hinoki-Sak-Sta-slerp-7B is a merge of the following models using the [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing) of [Maxime Labonne](https://huggingface.co/mlabonne) powered by [MergeKit](https://github.com/arcee-ai/mergekit) of [Arcee AI](https://www.arcee.ai):
 * [SakanaAI/EvoLLM-JP-A-v1-7B](https://huggingface.co/SakanaAI/EvoLLM-JP-A-v1-7B) (Base model)
 * [stabilityai/japanese-stablelm-base-gamma-7b](https://huggingface.co/stabilityai/japanese-stablelm-base-gamma-7b)
 
-## 🧩 Configuration
+## 💻 Configuration
 
 ```yaml
 slices:
@@ -39,40 +39,30 @@ parameters:
 dtype: bfloat16
 ```
 
-## 💻 Usage
+## 🤗 Usage for HuggingFace
 
 ```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
+from transformers import LlamaTokenizerFast, AutoModelForCausalLM
+from transformers import pipeline
+import torch
 
-model_path = "AkimfromParis/Hinoki-Sak-Sta-slerp-7B"
-tokenizer = AutoTokenizer.from_pretrained(model_path)
-model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype="auto", device_map="auto")
-model.eval()
+model_name = "AkimfromParis/Hinoki-Sak-Sta-slerp-7B"
 
-requests = [
-    "大谷翔平選手について教えてください",
-]
+tokenizer = LlamaTokenizerFast.from_pretrained(model_name)
+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.bfloat16)
 
-system_message = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {user_input} ASSISTANT:"
+pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, pad_token_id=tokenizer.eos_token_id)
 
-for req in requests:
-    input_req = system_message.format(user_input=req)
-    input_ids = tokenizer.encode(input_req, return_tensors="pt").to(device=model.device)
-    tokens = model.generate(
-        input_ids,
-        max_new_tokens=1024,
-        do_sample=True,
-        pad_token_id=tokenizer.eos_token_id,
-    )
-    out = tokenizer.decode(tokens[0][len(input_ids[0]):], skip_special_tokens=True)
-    print("USER:\n" + req)
-    print("ASSISTANT:\n" + out)
-    print()
+messages = [
+    {"role": "system", "content": "あなたは誠実で優秀な日本人のアシスタントです。以下のトピックに関する詳細な情報を提供してください。"},
+    {"role": "user", "content": "大谷翔平選手は誰ですか？"},
+    ]
+print(pipe(messages, max_new_tokens=512)[0]['generated_text'][-1])
 ```
 
-# Citation
+# 🔖 Citation
 ```
-@article{goddard2024arcee,
+@misc{goddard2024arcee,
   title={Arcee's MergeKit: A Toolkit for Merging Large Language Models},
   author={Goddard, Charles and Siriwardhana, Shamane and Ehghaghi, Malikeh and Meyers, Luke and Karpukhin, Vlad and Benedict, Brian and McQuade, Mark and Solawetz, Jacob},
   journal={arXiv preprint arXiv:2403.13257},