Create README.md

README.md

@@ -0,0 +1,135 @@
+---
+license: mit
+train: false
+inference: true
+pipeline_tag: text-generation
+base_model:
+- deepseek-ai/DeepSeek-R1-Distill-Llama3-8B
+---
+This is a version of the <a href="https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama3-8B">DeepSeek-R1-Distill-Llama3-8B</a> model re-distilled for better performance.
+
+## Performance
+
+| Models            | <a href="https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama3-8B">DeepSeek-R1-Distill-Llama3-8B</a> | <a href="https://huggingface.co/mobiuslabsgmbh/DeepSeek-R1-ReDistill-Llama3-8B-v1.1">DeepSeek-R1-ReDistill-Llama3-8B-v1.1</a> | 
+|:-------------------:|:--------:|:----------------:|
+| ARC (25-shot)      | 49.32 | <b>50</b> | 
+| HellaSwag (10-shot)| <b>76.75</b>  | 76.2 |
+| MMLU (5-shot)      | 56.87 | <b>58.78</b> | 
+| TruthfulQA-MC2     | 50.53 | <b>51.94</b> | 
+| Winogrande (5-shot)| 68.11 | <b>70.25</b> | 
+| GSM8K (5-shot)     | 61.79 | <b>75.66</b> | 
+| Average            | 60.56 | <b>63.81</b> | 
+
+| Models            | <a href="https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Llama3-8B">DeepSeek-R1-Distill-Llama3-8B</a> | <a href="https://huggingface.co/mobiuslabsgmbh/DeepSeek-R1-ReDistill-Llama3-8B-v1.1">DeepSeek-R1-ReDistill-Llama3-8B-v1.1</a>  | 
+|:-------------------:|:--------:|:----------------:|
+| GPQA (0-shot)     | 29  | <b>33.98</b> | 
+| MMLU PRO (5-shot) | 27.44 | <b>28.4</b> | 
+| MUSR (0-shot)     | 38.29 | <b>41.82</b> | 
+| BBH (3-shot)      | 41.57 | <b>49.59</b> | 
+| IfEval (0-shot) - strict  | <b>42.81</b> | 39.09 | 
+| IfEval (0-shot) - loose   | 31.05 | <b>40.29 </b> | 
+
+## Usage
+```Python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+compute_dtype = torch.bfloat16
+device   = 'cuda'
+model_id = "mobiuslabsgmbh/DeepSeek-R1-ReDistill-Llama3-8B-v1.1"
+
+model     = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=compute_dtype, attn_implementation="sdpa", device_map=device)
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+
+prompt  = "What is 1.5+102.2?"
+chat    = tokenizer.apply_chat_template([{"role":"user", "content":prompt}], tokenize=True, add_generation_prompt=True, return_tensors="pt")
+outputs = model.generate(chat.to(device), max_new_tokens=1024, do_sample=True) 
+print(tokenizer.decode(outputs[0]))
+```
+
+Output:
+```
+<｜begin▁of▁sentence｜><｜User｜>What is 1.5+102.2?<｜Assistant｜><think>
+To solve 1.5 plus 102.2, I'll start by adding the two numbers together.
+
+First, I'll add the whole numbers: 1 plus 102 equals 103.
+
+Then, I'll add the decimal parts: 0.5 plus 0.2 equals 0.7.
+
+Finally, I'll combine the results: 103 plus 0.7 equals 103.7.
+
+Therefore, 1.5 plus 102.2 is 103.7.
+</think>
+
+To find the sum of \(1.5\) and \(102.2\), follow these steps:
+
+1. **Align the decimal points:**
+
+   \[
+   \begin{array}{r}
+     1.5 \\
+   +102.2 \\
+   \hline
+   \end{array}
+   \]
+
+2. **Add the numbers:**
+
+   - Add the whole numbers: \(1 + 102 = 103\)
+   - Add the decimal parts: \(0.5 + 0.2 = 0.7\)
+   - Combine the results: \(103 + 0.7 = 103.7\)
+
+3. **Final Answer:**
+
+   \[
+   \boxed{103.7}
+   \]<｜end▁of▁sentence｜>
+```
+
+## HQQ
+Run ~3.5x faster with <a href="https://github.com/mobiusml/hqq/">HQQ</a>. First, install the dependencies:
+```
+pip install hqq
+```
+
+```Python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+from hqq.models.hf.base import AutoHQQHFModel
+from hqq.core.quantize import *
+
+#Params
+device        = 'cuda:0'
+backend       = "torchao_int4" 
+compute_dtype = torch.bfloat16 if backend=="torchao_int4" else torch.float16
+model_id      = "mobiuslabsgmbh/DeepSeek-R1-ReDistill-Llama3-8B-v1.1"
+
+#Load
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model     = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=compute_dtype, attn_implementation="sdpa")
+
+#Quantize
+quant_config = BaseQuantizeConfig(nbits=4, group_size=64, axis=1)
+AutoHQQHFModel.quantize_model(model, quant_config=quant_config, compute_dtype=compute_dtype, device=device)
+
+#Optimize
+from hqq.utils.patching import prepare_for_inference
+prepare_for_inference(model, backend=backend, verbose=False)
+
+############################################################
+#Generate (streaming)
+from hqq.utils.generation_hf import HFGenerator
+gen = HFGenerator(model, tokenizer, max_new_tokens=8192, do_sample=True, compile='partial').warmup()
+
+prompt = "If A equals B, and C equals B - A, what would be the value of C?" 
+out    = gen.generate(prompt, print_tokens=True)
+
+############################################################
+# #Generate (simple)
+# from hqq.utils.generation_hf import patch_model_for_compiled_runtime
+# patch_model_for_compiled_runtime(model, tokenizer, warmup=True)
+
+# prompt = "If A equals B, and C equals B - A, what would be the value of C?" 
+# chat    = tokenizer.apply_chat_template([{"role":"user", "content":prompt}], tokenize=True, add_generation_prompt=True, return_tensors="pt")
+# outputs = model.generate(chat.to(device), max_new_tokens=8192, do_sample=True) 
+# print(tokenizer.decode(outputs[0]))
+```